JP6493454B2 - Electronic camera - Google Patents

Description

  The present invention relates to an electronic camera.

  Conventionally, an electronic camera as in Patent Document 1 has been proposed in order to enable continuous shooting of high-speed still images. This electronic camera has multiple back engines that perform YCC (YCrCb) conversion processing and JPEG (Joint Photographic Experts Group) compression processing. These back engines share and process a series of images obtained by continuous shooting. To achieve high speed.

  Further, the image processing apparatus further includes a front engine that performs a correction process on each of the series of images described above and distributes the corrected image to a plurality of back engines. The camera is operated by both the front engine and the back engine. Configure the system.

JP 2008-219319 A

  However, the electronic camera disclosed in Patent Document 1 must have both a front engine and a back engine (at least one) even when continuous shooting performance is not so necessary. Had an increasing problem.

  In other words, the above-mentioned electronic camera can be configured by configuring the camera system with only the back engine when continuous shooting performance is not so necessary, and additionally using the front engine when processing with only the back engine is not possible. It wasn't scalable.

  Furthermore, the above-mentioned electronic camera is assumed to be a single lens reflex camera that takes a picture through an optical viewfinder, and transfer between “memory” and “memory” to transfer image data between the front engine and the back engine. Is used. For this reason, in the above-described electronic camera, for example, a delay time (latency) required from when a request such as a data transfer request is issued until the request result is returned is large, and even a single-lens reflex camera has recently become popular. There was also a problem that was not suitable for LiveView.

  Further, although the front engine performs the above-described correction processing, the main function is considered to distribute images to a plurality of back engines, and there was no viewpoint of using this front engine for purposes other than image distribution. . In addition, the above-described electronic camera is merely an example of a single-lens reflex camera capable of continuous shooting at 3 frames / sec. As an example, a full-resolution image exceeding 10M pixels is read out at a speed of 60 fps or more. No mention was made of a camera system that uses an image sensor with an ultra-high pixel rate (hereinafter referred to as “high pixel rate”). Note that the above high pixel rate means a pixel rate that is not able to be received on the back engine side when it is transferred as it is (cannot perform normal image processing).

  Accordingly, the present invention has been made in view of the above-described problems, and can form a minimum camera system that can perform image processing with only the back engine, and yet has a high pixel rate that cannot be received by the back engine. The purpose of the present invention is to provide a scalable electronic camera that uses a front engine in addition when the image sensor is used.

In one aspect of the present invention, an electronic camera includes an imaging unit, a storage unit, a first processing unit and a second processing unit. The imaging unit captures a subject image and outputs image data. The storage unit stores the image data output from the imaging unit. The first processing section, the image data stored in the storage unit Read out to reduce the pixel rate. The second processing section, the image data pixel rate is reduced by the first processing unit for image processing.
In another aspect of the present invention, an electronic camera according to the first invention includes an imaging unit, a front engine, and a back engine. The imaging unit captures a subject image and outputs image data. The front engine has a first input imaging interface, a pixel rate reduction means, and an output imaging interface. The first input imaging interface receives image data by being electrically connected to the imaging unit. The pixel rate reducing means lowers the pixel rate of the image data according to the capacity of the received image data. The output imaging interface outputs image data with a reduced pixel rate. The back engine has a second input imaging interface and image processing means. The second input imaging interface receives image data with a reduced pixel rate by being electrically connected to the output imaging interface. The image processing means performs image processing on the image data with a reduced pixel rate.

  In a second aspect based on the first aspect, the pixel rate reduction means is image reduction means for reducing the size of the image data. The image reduction means reduces the pixel rate by reducing the size of the image data received from the imaging unit.

  In a third aspect based on the first aspect, the pixel rate reduction means is a storage means for temporarily storing the image data received from the imaging unit, and the image data stored in the storage means is slower than at the time of storage. Is read out to reduce the pixel rate.

  In a fourth aspect based on the third aspect, the front engine includes image compression means and image expansion means. The image compression unit compresses the image data received from the imaging unit. The image decompressing means decompresses the compressed data compressed by the image compressing means. The front engine temporarily stores the compressed data compressed by the image compression unit in the storage unit, reads out the compressed data stored in the storage unit at a lower speed than that at the time of storage, and decompresses the decompressed image by the image decompression unit. Data is output from the output imaging interface.

  In a fifth aspect based on any one of the first to fourth aspects, the front engine further includes image data correction means. The image data correction unit performs a correction process on the image data received from the imaging unit.

  In a sixth aspect based on any one of the first to fifth aspects, the front engine has a plurality of pixel rate reduction means, and each pixel rate reduction means has the same image data received from the imaging unit. Or at different pixel rates.

  In a seventh aspect based on the sixth aspect, the front engine includes a first pixel rate means, a second pixel rate means, a low resolution output imaging interface, and a high resolution output imaging interface. The first pixel rate means reduces the image data received from the imaging unit to first image data having a relatively low resolution at the first pixel rate. The second pixel rate means reduces the image data received from the imaging unit to the second image data having a relatively high resolution at the second pixel rate. The low resolution output imaging interface outputs the first image data. The high resolution output imaging interface outputs the second image data.

  The back engine includes a low-resolution input imaging interface, a high-resolution input imaging interface, a first image processing unit, and a second image processing unit. The low resolution input imaging interface receives the first image data. The high resolution input imaging interface receives the second image data. The first image processing means performs image processing on the first image data. The second image processing means performs image processing on the second image data.

  According to the present invention, it is possible to configure a minimum camera system that can perform image processing only with the back engine, and when using an image sensor with a high pixel rate that cannot be received by the back engine, the front engine is used. It is possible to provide a scalable electronic camera that can be used by additionally using.

1 is a block diagram illustrating a configuration example of an electronic camera according to a first embodiment. The figure explaining the internal structure of the front engine 12 and the back engine 13 Diagram showing the flow of image data when shooting still images The block diagram which shows the structural example of the electronic camera of 2nd Embodiment. The figure explaining the example of an internal structure of the front engine 12 and the back engine 13 of the electronic camera 2. The block diagram which shows the structural example of the front engine 12 and the back engine 13 of the electronic camera of 3rd Embodiment. The figure which shows an example of the structure of the software utilized with the conventional electronic camera The figure explaining a virtual imaging circuit The figure which shows an example of the software structure of the electronic camera 1 The figure which shows an example of the block diagram of the electronic camera 50 The block diagram which shows the structural example of the image processing circuit 51a of the back engine 51. The figure which shows the flow of the image data at the time of the live view operation | movement in the electronic camera 50 The figure which shows the flow of the image data at the time of still image photography in the electronic camera 50 The block diagram which shows the other form of the image processing circuit 51a of the back engine 51 The figure explaining an example of the strip dividing process of a still image

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In order to facilitate understanding of the contents of the embodiment of the present invention, the basic configuration of the electronic camera 1 according to the embodiment of the present invention is described, and then, as a comparative example, an electronic camera 50 that does not use a front engine. The configuration and operation of the (conventional electronic camera) will be described. In particular, in the electronic camera 50, basic processing and image data flow in image shooting will be described. Based on these descriptions, the detailed configuration and operation of the electronic camera 1 will be described. Thereby, the difference between the electronic camera 1 and the electronic camera 50 is clarified.

<Configuration of electronic camera 1>
FIG. 1 is a block diagram illustrating a configuration example of the electronic camera according to the first embodiment. The electronic camera 1 includes a photographing optical system 10, an imaging circuit 11, a front engine 12, a back engine 13, a DRAM (Dynamic Random Access Memory) 14, a DRAM 15, and a Flash-ROM (Read Only Memory) 16 (see FIG. , “LCD”), an LCD (Liquid Crystal Display) panel 17, a release button 18, a connector C1, and a connector C2.

  The electronic camera 1 includes peripheral hardware such as an imaging circuit 11 with a front engine 12 and a back engine 13 as the center. The front engine 12 and the back engine 13 are each implemented as one LSI (SoC: System on Chip). Note that “front engine” and “back engine” are names used in Patent Document 1, but these names are also used in the following description. However, even if the same name is used, the internal configurations of the front engine 12 and the back engine 13 are significantly different from those in Patent Document 1. Unlike Patent Document 1, the back engine 13 of this embodiment can constitute a minimum camera system capable of performing image processing, and should be called a main engine functionally.

  The photographing optical system 10 includes a plurality of lens groups including a zoom lens that adjusts the focal length and a focus lens that adjusts the image formation position on the light receiving surface of the image sensor in the image pickup circuit 11. For the sake of simplicity, FIG. 1 shows the photographing optical system 10 as a single lens.

  The imaging circuit 11 includes a TG (Timing Generator) sensor, an AFE (Analog Front End), and the like in addition to an imaging sensor (CMOS (Complementary Metal Oxide Semiconductor) sensor or CCD (Charge Coupled Device) sensor). The imaging circuit 11 may employ a CMOS sensor incorporating TG and AFE.

  When photographing an image, first, an optical image of the subject (subject image) is formed on the image sensor by the photographing optical system 10. Next, this subject image is subjected to photoelectric conversion (exposure) by the image sensor, and finally output as digital image data from the imaging circuit 11. For example, a Bayer array color filter (CFA) is provided on the light receiving surface of the image sensor. In this case, color digital image data having a Bayer array is output from the imaging circuit 11. Control of the imaging circuit 11 such as exposure and output of image data is performed by a small number of control signals and serial communication ports.

  The front engine 12 is an element for the back engine 13 to process image data output from the high pixel rate imaging sensor. The front engine 12 once receives the image data output from the high-pixel-rate image sensor, reduces the pixel rate, and transmits the image data to the back engine 13.

  The back engine 13 is the most important element that bears most of the functions of the electronic camera 1. Here, the part of the arrow heading from the image pickup circuit 11 to the front engine 12 indicates that the image data output from the image pickup circuit 11 flows, and is actually connected by an (ultra) high-speed interface.

  On the other hand, image data with a reduced pixel rate flows in a portion indicated by an arrow from the front engine 12 to the back engine 13. Here, in addition to the front engine 12, an imaging circuit 11 including a general imaging sensor is also included. Connected. An arrow heading from the front engine 12 to the imaging circuit 11 in the figure represents a serial communication port and a control signal for controlling the imaging circuit 11.

  Further, in FIG. 1, there is also a wide white arrow that connects the front engine 12 and the back engine 13, but this white arrow represents a communication interface between the front engine 12 and the back engine 13, for example, A simple serial communication interface may be used. Details of the internal configuration of the front engine 12 and the back engine 13 will be described in detail with reference to FIGS.

  The DRAM 14 is connected to the front engine 12 and temporarily stores, for example, image data output from an image sensor with a high pixel rate. The DRAM 15 is connected to the back engine 13, and temporarily stores image data in various processes of the electronic camera 1, and is used for storing variables and stacks used in the program.

  The Flash-ROM 16 stores, for example, a program (camera firmware) that executes camera functions such as image capturing in the electronic camera 1, image playback, and image transfer to a PC (personal computer). The Flash-ROM 16 also stores various adjustment data of the electronic camera 1 and user setting information.

  The LCD panel 17 is composed of, for example, a liquid crystal display medium. The LCD panel 17 displays images and graphic data (characters, icons, etc.). The release button 18 is a half-press operation (start of shooting preparation operations such as autofocus (AF) and automatic exposure (AE) before shooting) and full-press operation (start of shooting operations to acquire the main image). The instruction input is accepted. The connector C1 is a connection terminal for connecting a detachable USB (Universal Serial Bus) memory U1. The connector C2 is a connection terminal for connecting a removable card type non-volatile memory card M1. FIG. 1 shows the USB memory U1 after being connected to the connector C1 and the memory card M1 after being connected to the connector C2.

<Configuration of electronic camera 50>
Next, a configuration of the electronic camera 50 will be described as a comparative example.

  FIG. 10 is a diagram illustrating an example of a block diagram of the electronic camera 50. The electronic camera 50 includes peripheral hardware such as an imaging circuit 52 with a single back engine 51 as the center. Note that the back engine 51 is mounted as a single LSI, similar to the back engine 13.

  The electronic camera 50 includes a photographing optical system 55, an imaging circuit 52, a back engine 51, a DRAM 53, a Flash-ROM 54, an LCD panel 57, a release button 56, a connector C3, a connector C4, and a bus 51r.

  The photographing optical system 55 has the same configuration as the photographing optical system 10 shown in FIG. The imaging circuit 52 has the same configuration as that of the imaging circuit 11 except that image data at a high pixel rate is not output. The LCD panel 57 and the release button 56 have the same configuration as the LCD panel 17 and the release button 18 shown in FIG. The connector C3 is a connection terminal for connecting the memory card M2, and the connector C4 is a connection terminal for connecting a removable USB memory U2. FIG. 10 shows the memory card M2 after being connected to the connector C3.

  The conventional back engine 51 is the most important element that bears most of the functions of the electronic camera 50, and is an image processing circuit 51 a that performs image processing on the image data output from the imaging circuit 52 (denoted as “image processing” in the figure). The CPU (Central Processing Unit) 51b for controlling the overall operation of the electronic camera, the JPEG codec 51c (indicated as “JPEG” in the figure) for compressing still images, images and graphic data are displayed on the LCD panel 57. LCD controller 51d to be displayed, USB controller 51e (denoted as “USB” in the figure) for transferring image data to a host device such as a PC, and DRAM controller 51f (denoted as “DRAMC” in the figure) for reading / writing data to / from the DRAM 53. Static memory controller 51g (in the figure, “SMC”) for reading / writing data from / to the Flash-ROM 54 SIO (Serial Input / Output) & PIO (Parallel Input / Output) 51h for exchanging data and signals with peripheral devices, CARD controller 51i for reading / writing data to / from memory card M2, etc. (See FIG. 10).

  The image processing circuit 51a, CPU 51b, JPEG codec 51c, LCD controller 51d, USB controller 51e, DRAM controller 51f, Static memory controller 51g, and SIO & PIO 51h are connected to each other via a bus 51r.

  Although not shown, the back engine 51 includes a video codec such as H.264, a DMA controller (DMAC) for transferring image data, an interrupt controller, an A / D converter, a timer, etc. Included (the same applies to the back engine 13). The functions of the electronic camera 50 such as image shooting, image playback, and image transfer to a PC or the like are implemented as a program (camera firmware) executed by the CPU 51b, and the program is stored in the Flash-ROM 54. The Flash-ROM 54 also stores various adjustment data of the electronic camera 50, user setting information, and the like. The DRAM 53 is used not only for temporarily storing image data in various processes of the electronic camera 50 but also for storing and stacking variables used in the program.

  In FIG. 10, these control signals and communication ports are represented by a single arrow from the back engine 51 to the image pickup circuit 52. The back engine 51 uses these to control the image pickup circuit 52 to control the image. Take a picture. The image data output from the imaging circuit 52 enters the image processing circuit 51a of the back engine 51 and undergoes various image processing. Here, this image processing is generally roughly divided into two processes, a pre-process and a post-process. This is because image processing at the time of photographing may be clearly divided into these two processes.

  The preprocess mainly includes a process for correcting variations of the image sensor and corresponds to the correction process performed by the signal processing means of the front engine in Patent Document 1. Specifically, the preprocess performs processing such as defective pixel correction (DPC), OB (Optical Black) clamping, shading correction, sensitivity correction, and the like. The pre-process may include other processes such as noise removal and correction of some lens aberrations.

  Apart from these correction processes, the preprocess also has a 3A detection function for extracting three types of evaluation values. This 3A detection function extracts each evaluation value for AE (Automatic Exposure), AF (Automatic Focusing), and AWB (Automatic White Balance). The electronic camera 50 controls automatic exposure (AE), automatic focus adjustment (AF), and automatic white balance adjustment (AWB) based on each evaluation value.

  On the other hand, the post process mainly includes color processing and corresponds to the YCC conversion / resolution conversion processing performed by the back engine in Patent Document 1. In that sense, it may be referred to as a color process. Specifically, the post process performs processing such as WB adjustment, color interpolation, color space conversion, γ correction, YCC conversion, false color removal, edge enhancement, and resolution conversion.

  Even after the preprocess is performed, the image data is still Bayer array data of 1 color / pixel, but after the post process, the image data is changed to YCbCr data of 3 colors / pixel. This YCbCr data is image data that can be used for display and printing. This YCbCr data is also used for image compression such as JPEG (still image) and H.264 (moving image).

  The image processing circuit 51a includes a preprocess circuit that performs preprocessing and a postprocess circuit (not shown) that performs postprocessing. Here, in the image processing operation, the preprocess circuit and the postprocess circuit are directly connected to each other, one step process for performing two processes integrally, and one frame of preprocessed image data is temporarily stored. There are two types of two-step processing in which the image data is stored in the DRAM 53 and then the image data is read from the DRAM 53 and subjected to a post process. FIG. 11 shows the internal configuration of the image processing circuit 51a that performs these operations and the flow of the image data.

  FIG. 11 is a block diagram illustrating a configuration example of the image processing circuit 51 a of the back engine 51. In order to receive the image data output from the imaging circuit 52, an input imaging interface (SIFi) 51j (denoted as “SIFi” in the drawing) is provided in front of the image processing circuit 51a. “SIFi” is a combination of the characters “Sensor InterFace” and “Input”. There is an output imaging interface (not shown) on the imaging circuit 52 side, and the input imaging interface 51j receives the image data output from the output imaging interface.

  The input imaging interface 51j performs a process for extracting data of the effective pixel area and the OB area from the input signal, and sends these data to the rear image processing circuit 51a. The image processing circuit 51a is preceded by a preprocess circuit 51k (indicated by “PreProc” in the figure), and a selector that selects and outputs one of two input data behind the preprocess circuit 51k. (Sel) 51m (denoted as “Sel” in the figure) is placed, and a post process circuit (PostProc) 51n (denoted as “PostProc” in the figure) is placed behind the selector 51m. ing.

  The preprocess circuit 51k has two outputs. The first output is connected to the first input of the selector 51m, and the second output is connected to the DRAM 53 (more precisely, it is connected to the input port of the DRAMC 51f shown in FIG. 10). ). On the other hand, the second input of the selector 51m is connected to the DRAM 53 (more precisely, to the output port of the DRAMC 51f), the output of the selector 51m is connected to the input of the post-processing circuit 51n, and finally the output of the post-processing circuit 51n. Is connected to the DRAM 53 (more precisely, it is connected to the input port of the DRAMC 51f).

  When the image processing circuit 51a executes one-step processing, the image data output from the first output of the preprocessing circuit 51k is selected by the selector 51m, and the image data is directly input to the post processing circuit 51n. Is set as follows. The image data sent from the input imaging interface 51j enters the preprocess circuit 51k and is first preprocessed. Then, the image data (Bayer) is output from the preprocess circuit 51k and passes through the selector 51m. Further, the post process circuit 51n enters the post process.

  The post-processed image data (YCbCr) is output from the post-processing circuit 51n and stored in a YCbCr buffer 53a provided on the DRAM 53 (see FIG. 11).

  On the other hand, when performing the two-step process, the image data sent from the input imaging interface 51j is set so that the second input data of the selector 51m is selected and enters the post-process circuit 51n. In the case of this setting, the image data read from the DRAM 53 is input to the post process circuit 51n and subjected to the post process.

  The overall operation of the two-step process is as follows. First, image data sent from the input imaging interface 51j is input to the preprocessing circuit 51k, and preprocessing is performed. The preprocessed image data (Bayer) is output from the second output of the preprocess circuit 51k and temporarily stored in the RAW buffer 53d provided on the DRAM 53 (see FIG. 11). This is the first step. In the case of the electronic camera 50 having a function of recording RAW data, it is normal to record Bayer data stored in the RAW buffer 53d.

  Next, the preprocessed Bayer data is read from the RAW buffer 53d, passes through the selector 51m, and further enters the postprocess circuit 51n to be postprocessed. The post-processed YCbCr data is output from the post-process circuit 51n and stored in the YCbCr buffer 53a provided on the DRAM 53 (see FIG. 11). This is the second step.

  The two types of image processing operations are caused by the horizontal size of YCbCr data as the processing result. Roughly speaking, when the horizontal size of YCbCr data is smaller than a preset value, the image processing circuit 51a performs one-step processing, and when the horizontal size is large (greater than a preset value), two-step processing is performed. Since this image processing operation is also closely related to the shooting operation of the electronic camera 50 at the time of shooting, the shooting operation of the electronic camera 50 shown in FIG. 10 will be described next.

<Shooting Operation of Electronic Camera 50>
In the present embodiment, since live view is emphasized, a shooting operation using the live view in the electronic camera 50 will be described as an example.

  At the time of shooting an image (shooting mode), the electronic camera 50 first enters a live view operation. In the live view operation, the imaging circuit 52 is set to the continuous output mode, and imaging (exposure) and output of image data are repeated at a frame rate of 30 fps or 60 fps. In the live view, the photographer decides the composition of the shooting while viewing the live image of the subject on the LCD panel 57, or performs photometric calculation for automatic exposure (AE) and automatic focus adjustment (AF) on the electronic camera 50 side. To do.

  The LCD panel 57 with a built-in electronic camera is often about VGA (640 × 480), and YCbCr data of a live image (through image) displayed on the LCD panel 57 is sufficient. Since the through image has a smaller image size than the recorded image, the live view uses a one-step process. Specifically, in live view, image data output continuously (for example, 30 fps or 60 fps) from the imaging circuit 52 is input to the image processing circuit 51a, and the image processing circuit 51a performs a one-step process to perform through. Generate YCbCr data for the image. The live view data is temporarily stored in the YCbCr buffer 53a on the DRAM 53, and then live view data is read out and sent to the LCD controller 51d and displayed on the LCD panel 57. By repeating this operation every frame, it is possible to view a live image of the subject. The flow of the image data at this time is shown in FIG.

<Flow of image data of electronic camera 50>
FIG. 12 is a diagram illustrating a flow of image data during a live view operation in the electronic camera 50. In FIG. 12, the pre-processing circuit 51k and the post-processing circuit 51n are collectively shown as an image processing circuit 51a, but the inside is as shown in FIG.
During the live view, the preprocess circuit 51k appropriately performs 3A detection and extracts a 3A evaluation value from the 3A buffer 53b.

  In FIG. 12, the 3A evaluation value is stored in the 3A buffer 53 b on the DRAM 53. Since the 3A evaluation value is output from the 3A detection circuit in the preprocess circuit 51k, it is as shown in FIG. The preprocess circuit 51k performs brightness adjustment (AE), focus adjustment (AF), and white balance adjustment (AWB) based on the 3A evaluation value. As a result, the brightness and color of the through image are maintained appropriately.

  In the photometric calculation, the exposure conditions for still image shooting are also calculated. Of the 3A evaluation values, those having a small amount of data are advantageously stored in an SRAM (Static Random Access Memory) in the detection circuit because reading from the CPU 51b is faster. In the preprocess, defective pixel correction is performed, and a defective pixel table 53 c storing the position (address) of the defective pixel is placed on the DRAM 53. A defective pixel correction circuit (not shown) in the preprocess circuit 51k reads information in the defective pixel table 53c, and performs a process of replacing it with a correction value calculated from the surrounding pixels when the pixel at that position arrives.

  The data flow is shown in FIGS. In the one-step processing, since the post-processing is performed all at once, the latency when the YCbCr data of the through image is generated is small. As a result, there is an advantage that the release time lag at the time of shooting is shortened. Further, in the one-step processing, intermediate data for image processing is not stored in the DRAM 53, so that there is little access to the DRAM 53, and there is an advantage that the bandwidth of the DRAM 53 is not so much occupied. The live view operation is not much different in both the still image shooting mode and the moving image shooting mode. When the release button 56 is fully pressed during live view, the operation proceeds to still image shooting or moving image shooting according to the shooting mode at that time.

<Operation of still image shooting>
Next, a still image shooting operation of the electronic camera 50 will be described. When the release button 56 shown in FIG. 10 is fully pressed, a half-press is detected before that, so that focus adjustment (AF) is performed first, and when this is completed, a step moves to a still image shooting operation. When moving to a still image shooting operation, automatic exposure (AE) of the subject is performed based on the photometric calculation performed immediately before. When the target exposure amount is reached, the exposure is terminated, and then image data of the captured still image (hereinafter referred to as “still image data”) is output from the imaging circuit 52.

  FIG. 13 is a diagram illustrating a flow of image data when the electronic camera 50 captures a still image. This image data is received by the input imaging interface 51j of the back engine 51 as shown in FIG. 13, and then enters the image processing circuit 51a. Since the captured still image is generally larger in resolution than a moving image such as a through image, the image processing circuit 51a executes a two-step process. The reason will be described below. The post process includes a number of two-dimensional filter processes. In order to process one image by progressive scanning, each filter circuit must have a line memory corresponding to the number of vertical taps. There was a problem that the circuit scale increased.

  Some filter circuits have a large number of taps, which accelerates the enlargement of the post-process circuit 51n shown in FIG. In order to process one huge image (for example, 50M capacity) at a time in comparison with a through image, a line memory having a storage capacity equivalent to the horizontal size of the image is required. It is difficult to have as many line memories as the number of taps for each filter circuit.

  In order to reduce the circuit scale, methods such as (A) reducing the number of filter circuits and taps, and (B) reducing the storage capacity per line memory can be considered. The method of condition (A) affects the image quality, and the method of condition (B) affects the size of an image that can be processed at one time. If importance is attached to image quality, the storage capacity of the line memory is reduced. In this case, however, how to process a high-resolution image becomes a problem. This point will be described later when the electronic camera 2 of the second embodiment is described.

<Details of still image processing>
Next, the image processing of still images will be described more specifically based on FIG. The image data output from the imaging circuit 52 is received by the input imaging interface 51j of the back engine 51, and then sent to the image processing circuit 51a to be preprocessed and further output from the image processing circuit 51a to be temporarily displayed. It is stored in the middle RAW buffer 53d. At this time, the image processing circuit 51a reads the information of the defective pixel table 53c as in the live view.

  However, when the number of pixels output from the imaging circuit 52 differs between live view and still image shooting, the image processing circuit 51a also prepares two tables for use because the address of the defective pixel also changes. In FIG. 13, an arrow that exits from the imaging circuit 52, passes through the input imaging interface 51j, and then goes to the RAW buffer 53d via the image processing circuit 51a represents the flow of still image data. The arrow toward the circuit 51a represents the flow of defective pixel information. FIG. 11 shows this part in more detail.

  As shown in FIG. 11, still image data flows as “imaging circuit 52 → input imaging interface 51j → preprocess circuit 51k → RAW buffer 53d”, and defective pixel information flows as “defective pixel table 53c → preprocess circuit 51k”. . This is the first step.

  In the subsequent second step, the post process circuit 51n executes the post process based on the strip division. Here, strip division will be described.

  FIG. 15 is a diagram for explaining an example of a strip dividing process for still images. The post process circuit 51n can process a large image by the method shown in FIG. That is, the post-processing circuit 51n divides one large image into small partial images having a small horizontal size as shown in (1), (2), and (3) in the figure. Each image is processed and stitched on the DRAM 53 to complete one image 31.

  Since the strip image has a small horizontal size, the line memory of the filter circuit can be small in capacity. Since the horizontal size of the strip image that can be processed is determined from the storage capacity of one line memory, the post-processing circuit 51n determines how many strip images the original image is divided based on. If the image has a large resolution, the number of strip images to be divided increases. If the horizontal size of one image is less than or equal to the storage capacity of one line memory, the post process circuit 51n can process without dividing the strip. The solid and dotted arrows in the figure indicate the scanning order of the pixel data of each strip image. Since the filter circuit described above also performs filtering in the horizontal (horizontal) direction, the post-processing circuit 51n divides the pixels so that the pixels overlap each other by the total number of horizontal taps of all filter circuits.

  In FIG. 15, bands 32 and 33 indicating the overlapping portions are explicitly shown. When all the filtering processes are completed, the number of pixels is reduced by half of the number of horizontal taps from both strip images at the boundary portion, so that the processed strip images are well connected at the boundary portion. A vertical dotted line in the figure represents a joint of strip images. In this overlapping portion, pixel data is read and read out from the DRAM 53 twice (twice). Therefore, if the horizontal size of the strip image is made too small, the overlapping portion increases, the number of times of data reading increases, and the efficiency decreases. Get out. This strip dividing process is only a post process, and the preprocess usually processes an entire image of one frame at a time.

  That is, the post process circuit 51n divides the preprocessed Bayer data into several strip images on the RAW buffer 53d, reads them one by one and inputs them to the image processing circuit 51a, and performs the post process. The post-processed YCbCr data is stored in the YCbCr buffer 53a. When this processing is executed for all the strip images, the image processing is completed on the YCbCr buffer 53a, and one connected image is completed.

  In FIG. 13, the arrow from the RAW buffer 53d to the YCbCr buffer 53a via the image processing circuit 51a represents the flow of image data at this time. FIG. 11 shows this in more detail.

  As shown in FIG. 11, the Bayer data of each strip image flows as “RAW buffer 53d → selector 51m → post process circuit 51n → YCbCr buffer 53a”. This is the second step.

  In FIG. 13, three YCbCr buffers 53a are provided on the DRAM 53 because three different size images are created by the post process. The first is a large main image that is the main recorded image, the second is a medium size display image (freeze image) for confirming the shooting result, and the third is a small thumbnail for search ( These are all images of YCbCr data. If the post-processing circuit 51n has a function of creating and outputting these three images in parallel, the processing time is shortened because all images are generated in one process.

  If there is no such function, it takes time because it is created sequentially. In the post process, since the strip division is generally performed, the speed is further reduced. Since a still image generally has a resolution higher than that of a through image, it is necessary to execute a post process by dividing into strips, which contributes to a two-step process in which pre-process and post-process are separately performed.

  On the other hand, since the live view has a smaller resolution than the still image, the live view uses a one-step process in which pre-processing and post-processing are integrated, but at that time, auxiliary processing may be required. Let me explain. The image data output from the imaging circuit 52 during live view does not necessarily have a small horizontal size (number of horizontal pixels). When pixel addition or thinning is performed by the image sensor, the number of output pixels also decreases, but even in that case, the horizontal size of the image is often large.

  For this reason, when image data having a large horizontal size is output from the imaging circuit 52 during live view, the storage capacity of the line memory of the filter circuit described above may be exceeded. In the back engine 51, the preprocessing circuit shown in FIG. One-step processing in which the image data output from 51k is directly input to the post-processing circuit 51n and post-processing is performed cannot be performed.

  However, in the case of live view, the problem that the one-step process cannot be performed is avoided because the image size of the finally generated through image YCbCr data is small. The reason for dividing the strip is to perform post-processing with a large image size, but in the case of live view, only a through image with a small image size is generated, so horizontal reduction is performed before post-processing. It doesn't matter.

  In other words, in the back engine 51, a horizontal image reduction circuit (not shown) is provided in front of the post-processing circuit 51n, and the horizontal size is reduced until strip division is no longer necessary. To enter. Since the post-process circuit 51n must generate a through image of a target size without dividing the strip, the internal filter circuit needs to have a line memory sufficient for the size. If the size of the through image is VGA (640 × 480), it is not difficult to have such a line memory.

  In the live view, if image data having more lines than necessary is output from the imaging circuit 52, the back engine 51 uses an image reduction circuit in the vertical (vertical) direction in addition to the image reduction circuit in the horizontal direction. (Not shown) may be provided. Thereby, the back engine 51 does not have to send unnecessary image data to the post-process circuit 51n. This image reduction process is not essential in the pre-process and post-process, and is an auxiliary process provided for performing a one-step process.

  In FIG. 11, in the back engine 51, this image reduction circuit may be provided independently, but it can also be previously placed in an appropriate place in the pre-process circuit 51k or the post-process circuit 51n. For example, it is placed at the end of the pre-process circuit 51k or the head of the post-process circuit 51n. At the time of live view, the post-processing circuit 51n sets this image reduction circuit to “execute” and performs one-step processing. At the time of still image shooting, the post-processing circuit 51n sets the image reduction circuit to “bypass”. Set and perform two-step processing. Thereby, the post-processing circuit 51n can perform image processing suitable for each operation.

  The two-step process is for executing the post process by dividing into strips, but there is another reason for performing the two-step process separately. This is because optimal white balance adjustment (AWB) is performed. WB adjustment is included in the post process, but in order to execute it, it is necessary to set a WB gain (WB: White Balance). The post-process circuit 51n determines the WB gain by executing the WB algorithm based on the AWB evaluation value, but there is a problem of when to perform this. Although the WB gain at the time of live view operation can be used as it is, there is a drawback that it is not based on the actual data of the captured still image. The post-processing circuit 51n needs to determine the WB gain based on the AWB evaluation value extracted from the still image data in order to perform more appropriate AWB adjustment.

  Here, since the 3A detection is performed by the preprocess circuit 51k, an AWB evaluation value is extracted when preprocessing the still image data output from the imaging circuit 52. In FIG. 13, this AWB evaluation value is stored in the AWB buffer 53 f on the DRAM 53. An arrow from the image processing circuit 51a toward the AWB buffer 53f represents the flow of the AWB evaluation value.

  Note that the AE evaluation value and the AF evaluation value need not be extracted because the shooting of the still image has been completed. FIG. 11 shows this part in more detail, and the AWB evaluation value flows as “preprocess circuit 51k → AWB buffer (one of 3A buffers in the figure) 53f”. Based on this AWB evaluation value, the AWB algorithm is executed to determine the WB gain. This WB gain is set in the post-process circuit 51n of FIG. 11, and then the post-process circuit 51n reads the preprocessed Bayer data from the RAW buffer 53d in the figure and performs the post-process.

  That is, since the image processing of the captured still image data is performed in the order of “(preprocess / AWB detection) → AWB algorithm execution → WB gain setting → post process”, two-step processing is necessary. The above is the content of the still image processing.

<Image processing during still image shooting>
Next, further description of image processing during still image shooting will be continued. In the image processing, since three types (three) of images (main image, freeze image, thumbnail) are created, after the still image is captured, the freeze image is first displayed on the LCD panel 57 so that the captured result can be confirmed. .

  Specifically, in response to an instruction from the CPU 51b shown in FIG. 10, a DMA controller (DMAC) (not shown) reads a freeze image from one of the YCbCr buffers 53a in FIG. 13, and sends it to the LCD controller 51d in the figure. It is displayed on the LCD panel 57. In FIG. 13, the arrow that goes out of the YCbCr buffer 53a and goes through the LCD controller 51d toward the LCD panel 57 represents the flow of the freeze image. In the Exif (Exchangeable Image File Format) / DCF standard, a JPEG-compressed main image and thumbnail are recorded, and these two images are JPEG-compressed in parallel with the display of the freeze image. If the freeze image is recorded together, the reproduction is performed at a high speed. Therefore, the JPEG codec 51c determines that the freeze image is also JPEG-compressed.

  In the multi-picture format (MPF) recently standardized by CIPA (Camera & Imaging Products Association), it is possible to record “monitor display images” corresponding to freeze images in addition to main images and thumbnails. good. The DMA controller (DMAC) reads the image data (YCbCr) from the YCbCr buffer 53a, inputs it to the JPEG codec 51c in the figure, and outputs the compressed data output from the JPEG codec 51c to the JPEG buffer 53e on the DRAM 53 (in the figure). “JPEG”). In FIG. 13, an arrow that exits from the YCbCr buffer 53a, passes through the JPEG codec 51c, and goes to the JPEG buffer 53e represents the data flow at this time.

  The DMA controller (DMAC) sequentially executes this process for three images. The DMA controller (DMAC) arranges the JPEG compressed data of the three images together with the tag information on one JPEG buffer 53e according to the format, and forms the file image of the Exif / DCF file or the MPF file. When this file image is completed, the CARD controller 51i then records the data on the memory card M1. That is, the DMA controller (DMAC) reads this file image from the JPEG buffer 53e, inputs it to the CARD controller 51i in the figure, and records it on the memory card M2. In FIG. 13, the arrow that exits the JPEG buffer 53e, passes through the CARD controller 51i, and goes to the memory card M2 represents the data flow at this time. When this is finished, the still image file is saved in the memory card M2 and the shooting is finished.

  The above is the processing for shooting a single still image (single shot), but simply repeating this does not result in high-speed continuous shooting. This is because the imaging circuit 52 does not start the next shooting until the shot still image is recorded in the memory card M2.

  The still image shooting operation includes four processes consisting of (I) exposure / image data output / preprocess, (II) post process, (III) freeze image display / JPEG compression, and (IV) recording to the memory card M2. Although broadly divided, if these are executed in parallel, continuous shooting can be performed at high speed. In the process (I), the imaging circuit 52 and the pre-process circuit 51k are used. In the process (II), the post-process circuit 51n is used. In the process (III), the LCD controller 51d and the LCD panel 57 are used. The JPEG codec 51c is used, and in the process (IV), the CARD controller 51i and the memory card M2 are used, but it can be seen that the resources used in each process do not compete.

  Therefore, the back engine 51 can execute these processes in parallel. There are further conditions (A) and (B) for performing parallel processing. Specifically, (A) pre-processing data (source data) is prepared, (B) post-processing The second is that the buffer memory for storing data is free.

  For example, while the preprocessed Bayer data is read from the RAW buffer 53d and the post process is performed, the RAW buffer 53d is not empty, so a still image of the next frame is captured and the pre process is performed. Bayer data cannot be stored there. Similarly, in the back engine 51, while the main image is read from the YCbCr buffer 53a and JPEG compression is performed, the YCbCr buffer 53a is not empty, so the main image of the next frame cannot be created and stored therein. .

  On the contrary, even if the processing of the previous frame is completed early and the buffer memory is free, when the source data of the next frame is not prepared, the back engine 51 cannot execute the processing. That is, in the back engine 51, while the conditions (A) and (B) are satisfied, the four processes (I) to (IV) are executed in parallel. The operation stops. In order to satisfy these conditions as much as possible, two RAW buffers 53d, YCbCr buffers 53a, and JPEG buffers 53e shown in FIG. 13 may be prepared.

  In other words, the back engine 51 reads out the first frame file image stored in the JPEG buffer 53e and records it in the memory card M2, and at the same time, stores the three images in the second frame stored in the YCbCr buffer 53a ( Main image, freeze image and thumbnail) are JPEG compressed and stored in another JPEG buffer 53e. At the same time, the back engine 51 post-processes the third-bayer Bayer data stored in the RAW buffer 53d and stores it in another YCbCr buffer 53a. Parallel processing is possible in which the process is performed and stored in another RAW buffer 53d.

  The processes (I) and (II) are performed by using the RAW buffer 53d, the processes (II) and (III) are performed by using the YCbCr buffer 53a, and the processes (III) and (IV) are performed by using the JPEG buffer 53e. Since these are shared, when these are executed in parallel, the back engine 51 performs exclusive control so that each process always uses a different buffer. This parallel processing (continuous shooting) can be continued if each processing replaces two buffers each time one frame processing is completed.

  Here, when the four processes (I) to (IV) are executed in parallel, there is no space in the buffer memory, and even if one of the processes ends early, the process of the next frame cannot be started. On the other hand, when the processing times of the four processes (I) to (IV) described above are extremely different, the back engine 51 waits for other processes to finish after the process that has been completed earlier, so that efficient parallel processing cannot be performed.

  Therefore, it is devised so that the processing times are as close as possible. For example, it is possible to devise such as creating three images in parallel in the post process or shortening the time required for code rate control (Bit Rate Control) by having two JPEG codecs 51c. In many cases, two RAW buffers 53d and two YCbCr buffers 53a are provided, and five JPEG buffers 53e are provided, for example. Since the compressed data is stored in the JPEG buffer 53e, the capacity is small, and it is relatively easy to provide a little more.

  Since the memory card M2 is detachable, when a memory card with a slow writing speed is used, the frame speed for continuous shooting is affected. If the number of JPEG buffers 53e is large, the compressed data of the next frame can be stored there without waiting for the previous frame to be recorded. For this reason, even when a low-speed memory card is used, high-speed continuous shooting continues for a certain length of time. For example, when the above-mentioned number of buffer memories are provided, high-speed continuous shooting can be performed for the following number of frames.

Number of frames for high-speed continuous shooting = 2 + 2 + 5 + α = 9 + α ------- (1)
“9” in the equation (1) is the total number of buffer memories, and indicates that high-speed continuous shooting is always possible until these buffer memories are full.

  On the other hand, “α” in the equation (1) represents the number of frames for which writing has been completed before the buffer memory becomes full. This α varies depending on the writing speed of the memory card M2. Even after these buffer memories are full, continuous shooting is continued at the frame speed corresponding to the writing speed of the memory card M2, but it is naturally slower than the frame speed before the buffer memory is full. Some high-speed continuous shootings store image data that has only been preprocessed in the RAW buffer 53d, and are used when image data output from the imaging circuit 52 is extremely fast.

  In this case, a large number of RAW buffers 53d are provided. Since the RAW buffer 53d has a large storage capacity, a DRAM having a large storage capacity is required, leading to an increase in cost. In the electronic camera 50, by increasing the number of buffer memories, high-speed continuous shooting can be performed until they are full. However, since then, shooting is performed while processing image data stored in the buffer memory, the frame speed is high. Become slow. When the continuous shooting frame speed suddenly decreases, it is also useful that the electronic camera 50 once returns to the live view operation and performs focus adjustment (AF), photometry calculation (AE), and composition adjustment again. The live view is not long and needs to be started quickly.

  However, with the configuration of the image processing circuit 51a shown in FIG. 11, it is difficult to start quickly. A post-processing circuit 51n in the figure is used to post-process the Bayer data stored in the RAW buffer 53d. When the number of RAW buffers 53d is large, it takes time until the post-processing circuit 51n becomes free, so it is not possible to immediately enter the live view. Therefore, in order to correspond to the operation of the electronic camera 50, the image processing circuit 51a of FIG. 14 is used.

  FIG. 14 is a block diagram showing another form of the image processing circuit 51 a of the back engine 51. In FIG. 14, a first post-process circuit (PostProc1) 51p (denoted as “PostProc1” in the figure) and a second post-process circuit (PostProc2) 51q (denoted as “PostProc2” in the figure) are included. The post processing circuit 51n of the image processing circuit 51a of FIG. 11 is replaced with a first post processing circuit 51p and a second post processing circuit 51q. The input of the first post-processing circuit 51p is always connected to the first output of the pre-processing circuit 51k, and this first post-processing circuit 51p is used only during the live view operation.

  On the other hand, the second post-processing circuit 51q is used for processing captured still image data. In live view, the image data output from the imaging circuit 52 is first received by the input imaging interface 51j. Next, the image data is sent to the preprocessing circuit 51k and subjected to preprocessing, and further output from the first output of the preprocessing circuit 51k and enters the first postprocessing circuit 51p, where the postprocessing is performed. And converted to YCbCr data of the through image. Finally, the data is output from the first post-processing circuit 51p and stored in the first YCbCr buffer (YCbCr1) 53g on the DRAM 53.

  On the other hand, at the time of still image shooting, the back engine 51 is set so that the second input data of the selector 51m in FIG. 14 is selected and input to the second post-processing circuit 51q and the pre-processing circuit 51k. The first output and the first post-processing circuit 51p are set to a non-operating state. The image data output from the imaging circuit 52 is received by the input imaging interface 51j, and then sent to the preprocessing circuit 51k to be preprocessed. Further, the image data is output from the second output of the preprocess circuit 51 k and temporarily stored in the RAW buffer 53 d on the DRAM 53.

  At this time, the first output of the pre-process circuit 51k and the first post-process circuit 51p are set in a non-operating state, so that there is no wasteful power consumption in the back engine 51. This is the first step. Next, the preprocessed Bayer data is read from the RAW buffer 53d, passes through the selector 51m, and further enters the second postprocess circuit 51q to be postprocessed.

  As shown in FIG. 14, the post-processed YCbCr data is output from the second post-processing circuit 51q and provided on the DRAM 53 as a second YCbCr buffer (YCbCr2) 53h (in the figure, expressed as “YCbCr2”) ). In the case of a large image that requires strip division, the post-process circuit 51n executes post-processing by strip division. This is the second step. In other words, since the resources used are completely different between the live view and the still image shooting post-process, the image processing circuit 51a of FIG. 14 can be used to return to the live view in the middle of continuous shooting and start it immediately. Become. The above is the whole image processing at the time of still image shooting.

<Image processing during movie shooting>
Next, image processing during moving image shooting will be briefly described. The user interface (GUI) for moving image shooting is similar to live view, except that moving image frame compression processing (such as H.264) and recording to a memory card are performed behind it. That is, there are two images, a display image (through image) and a recording image. Recently, HD (High Definition) moving image recording is not uncommon, and the two images are often different. For example, the display (live view) on the LCD panel 57 is performed in the VGA (640 × 480) size and the recording is performed in the HD (720/1080 line) size. In general, the resolution is higher than that of a through image. The image processing circuit of FIG. 14 is useful when creating images of two different sizes.

  That is, in the back engine 51, as shown in FIG. 14, a live view image (through image) is created by the first post-processing circuit 51p, and a recording image is created by the second post-processing circuit 51q. During moving image shooting, the back engine 51 is set so that the first input data of the selector 51m in FIG. 14 is selected and input to the second post-processing circuit 51q. When this setting is made, the Bayer data output from the first output of the preprocess circuit 51k after being preprocessed enters both the first post process circuit 51p and the second post process circuit 51q. Two sizes of images are created in parallel.

  At this time, since the two finally created YCbCr data are only stored in the YCbCr1 buffer 53g and the YCbCr2 buffer 53h, respectively, there is an advantage that the access to the DRAM 53 is small and the bandwidth is not so occupied. This advantage is particularly effective when recording HD video with H.264 compression, since there are many accesses to the DRAM 53. Since the moving image data for recording flows directly from the pre-process circuit 51k to the second post-process circuit 51q, it cannot be divided into strips. The second post-process circuit 51q receives at least full HD from the input image data. It must have a function to create a (1920 × 1080) image (YCbCr) directly.

  This means that the filter circuit included in the second post-processing circuit 51q has a line memory with a large storage capacity, but at present, this is possible because there is an upper limit of full HD on the size of the moving image. ing. Although the cost increases slightly, there is no latency between the live view image (through image) and the recording image, and it is easy to handle. The electronic camera 50 also has a function of playing back recorded still images and moving images and transferring recorded moving images and still image data to a host device such as a PC, which are directly related to the present invention. The explanation is omitted here.

  Here, the basic electronic camera 50 shown in FIG. 10 is configured by using the back engine 51 having the image processing circuit 51a of FIGS. 11 and 14, but the image processing circuit 51a, the JPEG codec 51c, and the CPU 51b are configured. Thus, a high-performance electronic camera can be realized by using only the back engine 51 by increasing the processing capacity of the DRAM 53 and the bandwidth of the DRAM 53.

  However, in recent years, things that cannot be dealt with on their own are starting to occur. Recently, a CMOS sensor has come to be used in place of a CCD sensor. However, a CMOS sensor that reads out a full-size image exceeding 10M pixels at a high frame rate of 60 fps or more can be technically realized. ing. The image data output from the imaging circuit 52 is first received by the 51-input imaging interface 51j of the back engine, and then it is directly input to the preprocessing circuit 51k in the image processing circuit 51a. Becomes a problem. For example, the pixel rate (pixel / sec: pps) of the image data output from the imaging circuit 52 including a high-speed CMOS sensor is as follows.

Pixel rate = 10Mpixel / frame x 60frame / sec = 600Mpixel / sec ----- (2)
That is, image data with a high pixel rate of “600 Mpps” flows through the preprocess circuit 51k, but it is difficult to flow image data with a high pixel rate into the image processing circuit 51a (usually about 200 Mpps). The pixel rate in equation (2) is an average pixel rate that image data is output during the blanking period in the 1V period (Vsync period), and is actually effective for a shorter time excluding the blanking period. Since all data (10M pixels) is output, the pixel rate is higher than 600Mpps. A high pixel rate imaging sensor has a higher-speed output interface than a general imaging sensor in order to output image data at high speed. That is, the interface has a higher transmission clock frequency than usual and also has a large number of data lines because data of a large number of pixels is output in parallel.

  Therefore, there is a problem that an imaging circuit including an ultra-high-speed imaging sensor cannot be directly connected to a back engine that assumes a general imaging sensor, and image data at a high pixel rate cannot be flowed to a preprocessing circuit. . The significance of developing a high-pixel-rate imaging sensor is the technology to reduce the distortion of the moving subject image caused by the rolling shutter in addition to the (ultra) high-speed continuous shooting of full-size images with multiple pixels. There is a good reason. As the electronic shutter of the CMOS sensor, a line reading rolling shutter is used instead of a global shutter in which all pixels are exposed at the same timing. For this reason, in the case of a fast-moving subject, there has been a drawback that a photographed image is distorted.

  Since this distortion is reduced by shortening the readout time of effective pixels, it seems that an image sensor with a high pixel rate was considered. A mechanical shutter can be used to eliminate image distortion due to movement, but this time there is a problem that (ultra) high-speed continuous shooting like 60 fps cannot be performed.

  In view of this, the electronic camera 1 of the present embodiment is configured so that the back engine 13 can process the image data output from the high pixel rate imaging sensor by additionally using the front engine 12. As described above, the front engine 12 is characterized in that it temporarily receives the image data output from the high-pixel-rate image sensor, and transmits the image data to the back engine 13 after reducing the pixel rate. That is, when a general imaging sensor is used, an electronic camera that does not require the front engine 12 is configured, and when a high pixel rate imaging sensor is used, the front engine 12 is additionally used to perform electronic processing. A scalable and flexible camera system is realized so as to constitute a camera. Hereinafter, a detailed configuration of the electronic camera 1 will be described.

<Detailed Configuration of Electronic Camera 1>
FIG. 2 is a diagram illustrating the internal configuration of the front engine 12 and the back engine 13. FIG. 2 shows the flow of image data during a live view operation in the electronic camera 1.

  Here, when the front engine 12 is provided, (A) the existing signal of the back engine 13 is used as much as possible (suppresses an increase in cost), and (B) a live view is created until a through image is created. The latency is as short as possible (reducing the release time lag), (C) the captured still image data is transmitted to the back engine with high resolution, and (D) the functions of the back engine 13 are utilized as much as possible. (E) Considering that the control (sequence) of image capturing is not complicated.

  As shown in FIG. 2, the front engine 12 includes an input imaging interface (SIFi) 12a (denoted as “SIFi” in the figure), an image reduction circuit (Resize1) 12b (denoted as “Resize1” in the figure), and an output. An imaging interface (SIFo) 12c (indicated as “SIFo” in the figure), an SIO 12d, a DRAM C 12e, a CPU 12f, and a bus 12p are provided.

  The input imaging interface 12a, the image reduction circuit 12b, the output imaging interface 12c, the SIO 12d, the DRAMC 12e, and the CPU 12f are connected to each other via the bus 12p.

  The back engine 13 includes an input imaging interface (SIFi) 13a (denoted as “SIFi” in the figure), an image processing circuit 13b, a JPEG codec 13c, an LCD controller 13d, a CPU 13e, a DRAMC 13f, a CARD controller 13g, and a bus 13q. Prepare.

  The input imaging interface 13a, the image processing circuit 13b, the JPEG codec 13c, the LCD controller 13d, the CPU 13e, the DRAMC 13f, and the CARD controller 13g are connected to each other via a bus 13q. Details of the interiors of the front engine 12 and the back engine 13 will be sequentially described.

  As shown in FIG. 1 and FIG. 2, the above (A) is achieved when the input imaging interface 13 a of the back engine 13 receives the image data output from the front engine 12. This is convenient in achieving the above (B). On the other hand, the above (C) is achieved by the front engine 12 including the DRAM 14 (local memory). The above (D) is achieved when the input imaging interface 13a receives image data and performs image processing.

  Since the image data with the reduced pixel rate output from the front engine 12 is received by the input imaging interface 13a of the back engine 13, the front engine 12 has the same output imaging interface as the imaging circuit 11 directly connected to the back engine 13. There must be.

  That is, in terms of interface, it seems that the imaging circuit 11 is connected in front of the back engine 13. This is convenient for achieving (E). The front engine 12 also includes an input imaging interface 12 a for receiving high pixel rate image data output from the imaging circuit 11. These input imaging interfaces 12a and 13a are interfaces of a format in which pixel data in a line sequence starting with an Hsync signal is placed on a transmission clock during one frame period starting with a Vsync signal. That is, in the vertical blanking period in which the first few lines and the last several lines do not include effective pixels, a line including effective pixels is arranged therebetween. Further, in a line including effective pixels, in the horizontal blanking period in which the first several pixels and the last several pixels do not include effective pixels, the effective pixels are arranged without any gap between them and are placed on the transmission clock to obtain pixel data. Transferred.

  Therefore, these input imaging interfaces 12a and 13a read out the image data stored in the DRAM 14 of the front engine 12 and store it in the DRAM 15 of the back engine 13. Therefore, the interface (RAW-I) described in Patent Document 1 is used. / F) is different. Here, since the CCD sensor is generally read at a lower speed than the CMOS sensor, the data output of the imaging circuit has many parallel interfaces in which each bit is divided.

  On the other hand, since a CMOS sensor generally reads faster than a CCD sensor, it is difficult to match so-called skew (distortion) between bits with a parallel interface, and the data output of the imaging circuit transmits each bit serially. There are many serial interfaces.

  In the case of a serial interface, the CMOS sensor is composed of a pair of signals whose data and transmission clock are positive and negative, respectively, and DDR (Double) that transmits each bit of data at both rising and falling edges of the transmission clock. Data Rate) operation. This pair of positive and negative data transmission paths is generally called “Lane”.

  Vsync and Hsync signals are often transmitted as synchronization codes embedded in data. Although the DDR operation is performed, the frequency of the transmission clock needs to be “number of bits / 2” times the pixel frequency. For example, when the data of each pixel is “12 bits”, the frequency of the transmission clock is six times the pixel frequency. When outputting image data at a higher speed, the number of Lanes is increased. In this case, pixels corresponding to the number of Lanes are output in parallel from the imaging circuit. Even if it is not a special high-speed CMOS sensor, in the case of the Bayer array, there are two Lanes, and R and Gr on the R / Gr line, and Gb and B pixels on the Gb / B line are output from different Lanes. There are many in general.

  That is, the two color pixels on one line are color-separated and output from two different Lanes. Therefore, in order to output image data at high speed, the two Lanes are used as the base, and the Lane is increased to 2 times (4-Lane), 3 times (6-Lane), 4 times (8-Lane). To go. An imaging circuit that outputs high pixel rate image data has a large number of Lanes and a high transmission clock frequency.

  The front engine 12 receives such high pixel rate image data. On the other hand, the front engine 12 and the back engine 13 are connected by an output imaging interface 12c and an input imaging interface 13a. This configuration is convenient for live view.

  That is, according to this configuration, the above (B) for creating a through image with low latency is achieved. Specifically, the internal circuit of the front engine 12 and the image processing circuit 13b of the back engine 13 are directly connected to operate as one large pipeline.

  During live view, the image data output from the imaging circuit 11 in FIG. 1 is sent to the back engine 13 through the internal circuit of the front engine 12, and at this time, the image data with a reduced pixel rate is transmitted to the front engine. 12 must be output. Since the image data is output after remaining in the internal circuit of the front engine 12, if the image data input from the image pickup circuit 11 is output at a low speed with the same number of pixels, its own pipeline overflows. .

  Fortunately, during live view, the front engine 12 only creates a through image of about VGA (640 × 480), so the image may be reduced to an appropriate size within the front engine 12. Since the pixel rate of the image data sent from the imaging circuit 11 is constant, if the front engine 12 reduces the image in the front engine 12, the pixel rate is reduced.

  Therefore, the front engine 12 first includes an image reduction circuit 12b as pixel rate reduction means. During live view, the front engine 12 reduces the image data received from the imaging circuit 11 by the image reduction circuit 12b, reduces the pixel rate until it can be received by the back engine 13, and outputs the reduced image data.

  The flow of the image data at this time is shown in FIG. In order to receive the image data output from the imaging circuit 11, the front engine 12 has an input imaging interface 12 a placed at the head in the front engine 12, and then an image reduction circuit 12 b placed behind it. Furthermore, an output imaging interface 12c is placed behind the structure.

  Here, “SIFo” is a combination of the characters “Sensor InterFace” and “Output”. The image data output from the imaging circuit 11 is first received by the input imaging interface 12a of the front engine 12, and then sent to the image reduction circuit 12b for reduction, and further sent to the output imaging interface 12c, from which the back engine is sent. 13 is output.

  The image reduction circuit 12b reduces the pixel rate until it can be received by the back engine 13 by performing reduction in the vertical direction as necessary in addition to reduction in the horizontal direction. The image data output from the output imaging interface 12c of the front engine 12 with the pixel rate reduced is then received by the input imaging interface 13a of the back engine 13, and further enters the image processing circuit 13b for pre-processing and post-processing. Applied. Note that the pre-process and post-process in the back engine 13 are the same as those in the conventional electronic camera 50, and thus description thereof is omitted.

  In FIG. 2, the image pickup circuit 11 passes through the input image pickup interface 12a, the image reduction circuit 12b, and the output image pickup interface 12c in the front engine 12, and further passes through the input image pickup interface 13a and the image processing circuit 13b of the back engine 13. The arrow leading to the YCbCr buffer 15a on the DRAM 15 represents the flow of image data in the live view, and the finally created through image YCbCr data is stored in the YCbCr buffer 15a.

  Next, this YCbCr data is read out, sent to the LCD controller 13d, and displayed on the LCD panel 17.

  On the other hand, during live view, the back engine 13 also performs photometric calculation / brightness adjustment (AE), focus adjustment (AF), and white balance adjustment (AWB), so the image processing circuit 13b in the back engine 13 The 3A evaluation value is extracted as appropriate and stored in the 3A buffer 15b on the DRAM 15. These data flows are also shown in FIG. 2, which are basically the same as those of the conventional electronic camera 50 (see FIG. 12).

  However, FIG. 2 is different from the conventional example in that the defective pixel table 14 a is not placed on the DRAM 15 of the back engine 13. This is because the front engine 12 performs image reduction. In image reduction, LPF (low-pass filter) processing and thinning are generally performed. However, since the image data output from the imaging circuit 11 includes defective pixel data, LPF processing is performed as it is. Is inappropriate.

  Therefore, in the present embodiment, the front engine 12 is also provided with a defective pixel correction circuit (DPC: Defect Pixel Correction). This defective pixel correction circuit needs to be placed in front of the image reduction circuit 12b. In this embodiment, a defective pixel correction circuit (not shown) is provided in the input imaging interface 12a for the convenience of later.

  A DRAM 14 is connected to the front engine 12, and the defective pixel table 14 a is placed on the DRAM 14. During live view, information in the defective pixel table 14a is read out and sent to the input imaging interface 12a, and defective pixel data included in the image data received from the imaging circuit 11 is corrected by an internal defective pixel correction circuit. . This data flow is also shown in FIG.

  Since the image data output from the imaging circuit 11 enters the back engine 13 via the front engine 12, the latency is higher than that of the conventional electronic camera 50, but it is at most a line level latency. Therefore, the frame level latency does not occur as in the electronic camera of Patent Document 1 that performs the transfer of “memory to memory”, and the above-described (B) is achieved in this embodiment.

  The front engine 12 also includes a CPU 12f, and the CPU 12f controls all the processing in the front engine 12 and driving of the imaging circuit 11.

  On the other hand, as can be seen from the fact that the release button 18 is connected to the back engine 13 in FIG. 1, the CPU 13e in the back engine 13 controls the overall operation of the camera, including image shooting.

  Accordingly, the front engine 12 is also placed under the control of the back engine 13, and the CPU 12f of the front engine 12 receives a command necessary for photographing an image from the CPU 12e of the back engine 13, processes the command, and responds to the command. The operation that returns is performed. Since the detailed processing on the front engine 12 side is performed by the CPU 12f, there is an advantage that the load on the CPU 13e of the back engine 13 is reduced correspondingly and the entire control of the electronic camera 1 is smoothly performed.

  In that case, a large number of parameters necessary for processing on the front engine 12 side are stored in advance in the DRAM 14 of the front engine 12, and the CPU 13e of the back engine 13 has a small number of parameters such as exposure time, image size, and processing mode. If only the parameters are sent to the CPU 12f of the front engine 12 together with the command, the communication overhead can be reduced.

  In FIG. 2, a bidirectional arrow is drawn between the CPU 12 f of the front engine 12 and the CPU 13 e of the back engine 13, and this represents a flow of the above-described command, a small number of parameters, and a response. This communication is performed via a communication interface (wide white arrow).

  The CPU 13e of the back engine 13 transmits only a small number of parameters necessary for command execution. This also serves to abstract the other front engine 12 to reduce communication overhead. That is, in this embodiment, it is not necessary to consider the hardware configuration of the front engine 12 and its detailed movement. Therefore, the program executed by the CPU 13e of the back engine 13 can be created without depending on the front engine 12.

  Processing in the front engine 12 and driving of the imaging circuit 11 are controlled by a program executed by the CPU 12f of the front engine 12. A program (application) on the back engine 13 side calls an upper API (Application Program Interface) and uses its function. This increases the portability and reuse of the program, which in turn leads to the common use with the program of the conventional electronic camera. (E) mentioned above is achieved by advancing this.

  The program on the front engine 12 side can be stored in a flash memory (not shown) connected to the front engine 12, or downloaded from the back engine 13 to the DRAM 14 on the front engine 12 via the communication interface. good. As a result, only one flash memory is required on the back engine 13 side, so that an increase in cost can be suppressed.

  In the latter case, first, the CPU 13e of the back engine 13 is activated to initialize the back engine 13 side. Next, the program executed by the CPU 12f of the front engine 12 is read from the flash memory 16 (see FIG. 1), and the program is downloaded to the DRAM 14 of the front engine 12 via the communication interface. Finally, since the CPU 12f of the front engine 12 executes a sequence for executing the program, the startup of the system is slightly delayed.

  When the program is downloaded from the back engine 13 side, the front engine 12 also downloads the above-described many parameters together with the DRAM 14. So far, the two types of engine movement have been described from the viewpoint of software, but the operation of the live view will be described again from the viewpoint of software.

<Operation of live view>
As shown in FIG. 2, first, the CPU 13e of the back engine 13 sets the image processing circuit 13b to the live view processing mode, then sets the input imaging interface 13a to active (Active), and finally sets the live view command. It transmits to CPU12f of front engine 12.

  Parameters necessary for command execution include the exposure time (electronic shutter time) of the imaging circuit 11, the output mode number (number of pixels and frame rate) of the imaging circuit 11, the reduction rate of the image reduction circuit 12b, etc. When the output mode of the circuit 11 is determined, a reduction rate for reducing the pixel rate to a level that can be received by the back engine 13 is automatically determined.

  Therefore, if the combination of the output mode of the imaging circuit 11 and the image reduction ratio is determined and the table is provided on the front engine 12 side, the parameter of the reduction ratio becomes unnecessary. When the image reduction ratio is determined, the filter coefficient of the image reduction circuit is also determined. Therefore, if the table is also included, it is not necessary to send the filter coefficient together with the command. These tables are also downloaded to the DRAM 14 of the front engine 12 together with the defective pixel table 14a at startup.

  Eventually, in this embodiment, it is not necessary to send parameters depending on the hardware of the front engine 12 together with the command, and it can be seen that the above-described abstraction is achieved. The necessary parameters are only those related to the driving of the imaging circuit 11, and when this abstracted API is used, the entire front engine 12 and the imaging circuit 11 can be handled as one virtual imaging circuit (details). Will be described later in the description of the virtual imaging circuit shown in FIG.

  Therefore, in this embodiment, it becomes easy to use those functions from the back engine 13, and it can be shared with the software of the conventional electronic camera. This is preferable from the viewpoint of software reuse. In other words, it is a great advantage that even if it is difficult to completely share with an actual electronic camera, it is possible to obtain a program that can be operated on other cameras with a slight modification.

  The CPU 12f of the front engine 12 operates the internal circuit of the front engine 12 and the image pickup circuit 11 based on the command and parameter described above. Specifically, the CPU 12 f of the front engine 12 calls a device driver to perform live view processing, and transmits live view image data with a reduced pixel rate to the back engine 13.

  In the conventional electronic camera 50, as shown in FIG. 10, since the imaging circuit 52 is directly connected to the back engine 51, an imaging driver is also mounted on the back engine 51. Call the driver.

  On the other hand, in the case of the electronic camera 1 according to the present embodiment having the front engine 12, the API calls a virtual imaging driver (see FIG. 9) described later. Actually, the virtual imaging driver further calls the communication driver and transmits the above-described commands and parameters to the front engine 12, and the CPU 12f of the front engine 12 calls the virtual imaging driver and the communication driver in response to the command. Perform live view operation.

  As described above, in the electronic camera 1 of the present embodiment, (A), (B), and (E) can be achieved in addition to the above-described (C) and (D). When the brightness of the subject changes during live view (for example, when it enters the shade), the CPU 13e of the back engine 13 sends a command for changing the exposure time and a new exposure time (electronic shutter time) to the front engine 12. It transmits to CPU12f.

<Operation of Still Image Shooting in Electronic Camera 1>
Next, the still image shooting operation of the electronic camera 1 will be described with reference to FIGS.

  FIG. 3 is a diagram illustrating a flow of image data during still image shooting. When the release button 18 shown in FIG. 1 is fully pressed during live view, the electronic camera 1 performs focus adjustment (AF) and then proceeds to subject exposure. Since 3A detection is performed by the back engine 13, the focus adjustment (AF) and photometry calculation (AE) are also performed by the CPU 13e of the back engine 13. The processing so far is the same as that of the conventional electronic camera 50, but the subsequent exposure and image data output are different in operation because the front engine 12 exists. The CPU 13e of the back engine 13 first sets the image processing circuit to the still image processing mode, then sets the input imaging interface 12a to active (Active), and finally sends a still image shooting command to the CPU 12f of the front engine 12. Send.

  The parameters transmitted together with the command are the exposure time (electronic shutter time) of the image pickup circuit 11 and the output mode number (full resolution) of the image pickup circuit 11. Since still image shooting is basically a 1-Shot operation, in the case of continuous shooting, these are repeatedly transmitted.

  Actually, on the back engine 13 side, this communication is performed from the above-described API. A virtual imaging driver is called from this API, and a communication driver is further called from the virtual imaging driver. Then, commands and parameters are sent to the CPU 12f of the front engine 12. When the CPU 12f of the front engine 12 receives these pieces of information, the internal circuit of the front engine 12 and the imaging circuit 11 are operated to take a still image.

  The CPU 12f calls those device drivers to first cause the image pickup circuit 11 to perform exposure, and then causes the image pickup circuit 11 to output still image data. Further, the CPU 13e causes the front engine 12 to reduce the pixel rate, and finally, the CPU 12f causes the still image data with the reduced pixel rate to be output from the output imaging interface 12c toward the back engine 13.

  Here, by providing the virtual imaging driver, the same software as that of the conventional electronic camera can be used in the upper layer from the API even in the operation of still image shooting.

  However, there remains a problem of how to reduce the pixel rate of still image data. Since there is a condition (C) for transmitting still image data to the back engine 13 with high resolution, the method of reducing the image as in live view cannot be used.

  Basically, it must be transmitted to the back engine 13 with the resolution (full resolution) at the time of shooting. This is achieved by providing a memory (DRAM) 14 in the front engine 12. That is, the front engine 12 temporarily stores the still image data output from the imaging circuit 11 in the DRAM 14 and reads the still image data with a reduced pixel rate by reading the still image data at a lower speed than the storage time. It can be transmitted to the engine 13.

  That is, the front engine 12 may include a local memory (DRAM) as another pixel rate reduction means. As described above, the DRAM 14 of the front engine 12 is used for storing programs and parameters. This is a secondary usage, and the main use is to temporarily store still image data.

<Flow of image data when shooting still images>
Next, the flow of image data during still image shooting will be described. The photographed still image data is output from the imaging circuit 11 and first received by the input imaging interface 12a of the front engine 12. Then, the still image data is corrected in the defective pixel data by an internal defective pixel correction circuit (not shown), further output from the input imaging interface 12a, and stored in the RAW1 buffer 14b provided on the DRAM 14 of the front engine 12. Once memorized.

  At this time, information of the defective pixel table 14a is read from the same DRAM 14 and input to the defective pixel correction circuit. However, since the front engine 12 may only reduce the pixel rate, the still image data is sent to the back engine 13 without performing defective pixel correction, and is included in the image processing circuit (preprocess circuit) of the back engine 13. However, it may be corrected by a defective pixel correction circuit.

  However, the back engine 13 has a drawback that the control is complicated because it is not possible to perform unified processing for live view and still image shooting. For this reason, defective pixel correction is performed on the front engine 12 side even during still image shooting. Subsequently, on the front engine 12 side, still image data is read from the RAW 1 buffer 14 b, input to the output imaging interface 12 c of the front engine 12, and output from there to the back engine 13.

  At this time, still image data with a reduced pixel rate is output from the output imaging interface 12c. As described above, the still image data stored in the DRAM 14 is read out at “low speed”, which means that it is read out at a low speed that can be read out on the back engine 13 side.

  Here, as shown in FIG. 3, the output imaging interface 12 c of the front engine 12 transmits image data at a transmission speed (transmission clock) that matches the input imaging interface 13 a of the back engine 13. Therefore, when the CPU 12f of the front engine 12 reads still image data from the RAW1 buffer 14b in accordance with the transmission operation of the output imaging interface 12c, as a result, the image data is read at a low speed.

  Transmission from the output imaging interface 12c and reading from the RAW1 buffer 14b are generally asynchronous. Therefore, if the CPU 12f of the front engine 12 transmits the data input from the RAW1 buffer 14b as it is, a gap is generated in the still image data sent to the back engine 13 (becomes a missing tooth shape).

  This is not allowed as a format of the input / output imaging interface 12a. Therefore, the output imaging interface 12c of the front engine 12 must have a line buffer.

  That is, the output imaging interface 12c temporarily stores still image data in this line memory, and when pixel data for one line is accumulated, it is transmitted on the transmission clock without any gap. When at least two line buffers are provided, data for the next line is prepared during transmission, so that data transmission is smooth. The same applies to the live view. The still image data output from the image reduction circuit 12b is temporarily stored in the line buffer and then transmitted to the back engine 13. The still image data output from the output imaging interface 12c of the front engine 12 is then input to the back engine 13 for image processing.

  This still image data is first received by the input imaging interface 13a of the back engine 13, and then sent to the image processing circuit 13b to be preprocessed by an internal preprocess circuit (PreProc). Since the defective pixel correction is performed in advance by the front engine 12, the remaining processing is performed in the preprocess circuit of the back engine 13.

  The preprocessed still image data (Bayer) is output from the preprocess circuit and temporarily stored in the RAW2 buffer 15c provided on the DRAM 15 of the back engine 13. The preprocess circuit in the image processing circuit 13b also performs AWB detection. At this time, the extracted AWB evaluation value is stored in the AWB buffer 15d provided on the same DRAM 15. The steps so far are the first step (preprocess) of the still image processing, and the data flow at this time is shown in FIG.

  The CPU 13e of the back engine 13 executes an AWB algorithm based on the AWB evaluation value to obtain a WB gain, and sets the WB gain in a post process circuit (PostProc) in the image processing circuit 13b. Next, the post process circuit in the image processing circuit 13b executes the second step (post process) of the image processing. Since the still image is generally huge, the post-processing circuit executes the second step by dividing into strips.

  The post-processing circuit divides the preprocessed still image data (Bayer) into several strip images on the RAW2 buffer 15c, reads them one by one and inputs them to the post-processing circuit in the image processing circuit. Post processing is performed, and the post-processed YCbCr data is stored in the YCbCr buffer 15 a on the DRAM 15. When the post-processing circuit executes this processing for all the strip images, the image processing has been completed on the YCbCr buffer 15a, and an image connected to one is completed.

  In the second step, a main image, a display image (freeze image), and a thumbnail are created and stored in each YCbCr buffer 15a. The above is the second step (preprocess), and the data flow at this time is also shown in FIG. Thereafter, a freeze image is first displayed on the LCD panel 17 in order to confirm the photographing result according to an instruction from the CPU 13e. Specifically, in accordance with an instruction from the CPU 13e, the DMA controller reads a freeze image from one of the YCbCr buffers 15a, sends it to the LCD controller 13d, and displays it on the LCD panel 17.

  On the other hand, in accordance with an instruction from the CPU 13e, the JPEG codec 13c performs JPEG compression on the three images described above in parallel. The DMA controller reads the image data (YCbCr) from the YCbCr buffer 15a, inputs it to the JPEG codec 13c, and stores the compressed data output from the JPEG codec 13c in the JPEG buffer 15e on the DRAM 15.

  The JPEG codec 13c compresses three images by sequentially executing the above processing. In response to an instruction from the CPU 13e, the CARD controller 13g arranges JPEG compressed data of three images together with tag information on one JPEG buffer 15e, and forms a file image of an Exif / DCF file or an MPF file. When this file image is completed, the CARD controller 13g records each image on the memory card M1.

  Further, the DMA controller reads this file image from the JPEG buffer 15e, inputs it to the CARD controller 13g, and records it on the memory card. When this recording is completed, the still image file is stored on the memory card 20 and the shooting is completed. The data flow in these processes is also shown in FIG. The DRAM 14 on the front engine 12 side is provided to reduce the pixel rate of still images. However, if only the pixel rate is to be reduced, only one RAW1 buffer 14b is required, so that the storage area of the DRAM 14 is vacant. Is wasted.

  Therefore, this storage area is also used for another purpose. Since this DRAM 14 is also used for program storage and work memory (variable and stack area) of the program, a number of RAW1 buffers 14b are allocated to the storage area excluding them. Thus, high-speed continuous shooting (hereinafter referred to as “RAW buffer continuous shooting”) in which still image data output from the imaging circuit 11 is accumulated in the RAW1 buffer 14b can be performed.

  Concurrently with storing the still image data in the RAW1 buffer 14b, the output imaging interface 12c reads out the previously captured frame data from another RAW1 buffer 14b and sends it to the back engine 13 for processing. be able to. Since the image data output from the imaging circuit 11 has a high pixel rate, the DRAM 14 of the front engine 12 must have a wide band. Further, if a plurality of RAW2 buffers 15c on the back engine 13 side are provided, the continuous shooting can be continued for a long time because the transmission of image data does not stagnate even if the subsequent processing is slow.

  When all the buffer memories (RAW1, RAW2, YCbCr, JPEG) are full or the flow of some image data is stagnant and it reaches the front and the RAW1 buffer 14b is full, high-speed continuous shooting temporarily stops To do.

  Thereafter, since the stored still image data is processed and shooting is performed every time one RAW1 buffer 14b is available, the frame speed of continuous shooting is slowed down. This is the same as with a conventional camera, but if the continuous shooting frame rate is slow, it is also useful to return to the live view and adjust the focus (AF), photometry (AE), and composition again. is there.

  As described above, since the electronic camera 1 according to the present embodiment transfers the image data as it is to the back engine 13 at a low pixel rate, a minimum camera system that can perform image processing with only the back engine 13 is configured. can do. Further, when using an image sensor with a high pixel rate that cannot be received by the back engine 13, the image reduction circuit 12 b of the front engine 12 can reduce the pixel rate until the back engine 13 can receive. That is, it is possible to provide a scalable electronic camera in which the front engine 12 is additionally used for handling.

(Second Embodiment)
Next, a second embodiment will be described. Here, in the case of the electronic camera 50, since the image data output from the imaging circuit 52 is not at a high pixel rate, the resolution is not so high even if a certain frame is recorded as a still image while performing live view. . This is because the number of pixels must be reduced in order to output image data at a high frame rate as in live view.

  Therefore, the electronic camera according to the second embodiment provides means for recording a certain frame as a full-resolution still image while performing live view when high pixel rate image data is received. In the configuration of the electronic camera 1 of the first embodiment (see FIGS. 1 to 3), it is difficult to provide such means. The reason is that the live view image data and the still image data must be transmitted in parallel from the front engine 12 to the back engine 13, but between the two engines shown in FIGS. This is because there is only one transmission path for image data. Therefore, in the electronic camera of the second embodiment, an engine having two image data transmission paths is used.

  Note that, in the first embodiment and the second embodiment, the same components shown in FIG. 1 are denoted by the same reference numerals, and here, differences will be mainly described.

  FIG. 4 is a block diagram illustrating a configuration example of the electronic camera according to the second embodiment. In the figure, a solid line arrow from the front engine 12 to the back engine 13 represents a first image data transmission path, and a similar dotted line arrow represents a second image data transmission path. There is a first output imaging interface 12c of the front engine 12 at the start point of the solid line arrow, and there is a first input imaging interface 13a of the back engine 13 at the end point of the solid line arrow. Becomes the first image data transmission path.

  On the other hand, there is a second output imaging interface 12k of the front engine 12 at the start point of the dotted arrow, and there is a second input imaging interface 13k of the back engine 13 at the end point of the dotted arrow. The signal line becomes the second image data transmission path.

  The first image data transmission path is used in such a manner that relatively low resolution image data flows, and the second image data transmission path uses relatively high resolution image data. Specifically, live view image data is sent to the first image data transmission path, and still image data and recording moving image data are sent to the second image data transmission path.

  This usage can be fixed. The image data flowing through these image data transmission paths must have a reduced pixel rate so that the back engine 13 can receive the image data. The front engine 12 performs this pixel rate reduction. In order to transmit two pieces of image data in parallel, the front engine 12 has a plurality of pixel rate reduction means.

  The back engine 13 receives and processes these two image data. The new front engine 12 and the back engine 13 for transmitting and receiving two pieces of image data in parallel are different in internal configuration from the electronic camera 1 (see FIGS. 2 and 3) and the electronic camera 50 (see FIG. 14). Hereinafter, an internal configuration and a flow of image data of the front engine 12 and the back engine 13 of the second embodiment will be described.

<Internal configuration example of front engine 12>
FIG. 5 is a diagram illustrating an example of the internal configuration of the front engine 12 and the back engine 13 of the electronic camera 2. FIG. 5 shows the flow of image data. Therefore, an internal configuration example of each engine will be described along the flow of image data.

  The high pixel rate image data required to reduce the pixel rate output from the imaging circuit 11 is first received by the input imaging interface 12a of the front engine 12, and then the defective pixel correction circuit (DPC: Defect Pixel Correction). Thus, the defective pixel data is corrected. A defective pixel table storing the positions (addresses) of defective pixels is placed on the DRAM 14 of the front engine 12, and the DPC block 12g reads and corrects the information in this table.

  In FIG. 2 and FIG. 3, the DPC block 12g is included in the input imaging interface 12a for simplification, but in FIG. 5, the two are shown separately for easy understanding. The dotted line frame at the left end inside the front engine 12 in FIG. 5 corresponds to the input imaging interface 12a in FIGS. Image data subjected to defective pixel correction is sent to three pixel rate reduction means. Among them, the first pixel rate reduction means and the second pixel rate reduction means have an image reduction circuit, and the third pixel rate reduction means has a memory.

  The first image reduction circuit 12b in FIG. 5 serves as the first pixel rate reduction means, and the second image reduction circuit (Resize2) 12i (denoted as “Resize2” in the drawing) provides the second pixel rate. The memory (DRAM: RAW1 buffer 14b) corresponds to the third pixel rate reducing means.

  In FIG. 5, a two-input selector (Sel) 12j (indicated as “Sel” in the figure) is placed after the second image reduction circuit 12i, and the second input is input to the first input of the selector 12j. The output of the image reduction circuit 12i is connected, and the output of the RAW1 buffer 14b (output port of the DRAMC) is connected to the second input. Further, the output of the selector 12j is connected to a second output imaging interface (SIFo2) 12k (denoted as “SIFo2” in the drawing).

  Compared with the flow of image data in FIGS. 2 and 3, the dotted line frame at the left end in the front engine 12 in FIG. 5 corresponds to the input imaging interface 12 a in FIGS. 2 and 3. 5 corresponds to the output imaging interface 12c in FIG. 2 or FIG. 3. The dotted frame at the right end inside the front engine 12 in FIG. Further, the second image reduction circuit 12i in FIG. 5 corresponds to the image reduction circuit 12b in FIGS. Further, the blocks in the dotted line frame in FIG. 5 can have the same configuration by using the corresponding blocks in FIG. 2 and FIG.

  On the other hand, the blocks after the first image reduction circuit 12b in FIG. 5 are newly provided. FIG. 2 shows the flow of image data at the time of live view operation. In FIG. 2, the pixel rate is reduced using the same image reduction circuit 12b in both live view and moving image shooting, and image data is output from the same output imaging interface 12c. At the time of moving image shooting, the image data output from the front engine 12 is divided into two in the image processing circuit 13b of the back engine 13, and is described as a first post-processing circuit (PostProc1) 13j ("PostProc1" in the figure). ), A through image (YCbCr) for display is created, and a moving image (YCbCr) for recording is created in the second post-process circuit (PostProc2) 13p (denoted as “PostProc2” in the figure). When there is one transmission path for image data, the flow of image data for live view and moving image shooting is as described above.

  However, when there are two image data transmission paths as shown in FIG. 5, the flow of the live view image data changes. Specifically, the input imaging interface 12a sends live view image data to the first image reduction circuit 12b. A first preprocessing circuit (PreProc1) 12h (denoted as “PreProc1” in the figure) is placed after the first image reduction circuit 12b, and then the first output imaging interface (SIFo1) 12c. (It is written as “SIFo1” in the figure.) This is a newly established data path.

  Live view image data always flows in the order of “first image reduction circuit 12b → first preprocess circuit 12h → first output imaging interface 12c”. This is the same for video shooting. On the other hand, the moving image data for recording flows as “second image reduction circuit 12i → selector 12j → second output imaging interface 12k”. Still image data flows in the order of “RAW1 buffer 14b → selector 12j → second output imaging interface 12k”.

  In other words, the still image data and the moving image data for recording are transmitted to the back engine 13 through the second image data transmission path (dotted arrow), but the live view image data is always the first. Is transmitted to the back engine 13 through the image data transmission path (solid arrow). Since the second image data transmission path is basically the same as that shown in FIGS. 2 and 3, the difference is that the live view image data is transmitted through the newly established first image data transmission path. It is a point.

  When there is one image data transmission path, the back engine 13 uses the moving image data for recording sent at the time of moving image shooting for live view (through image display), but when there are two image data transmission paths, Since the moving image data for live view and the moving image data for recording are sent simultaneously, the back engine 13 processes them separately.

  When recording a movie, it is useless because only one movie data is required for recording. However, the electronic camera 2 can send live view image data through the same transmission path during both still image shooting and movie shooting. There is a merit that processing is unified. That is, since the electronic camera 2 has two transmission paths for image data, the configuration of the back engine 51 shown in FIG. 14 is changed to the configuration of the back engine 13 shown in FIG.

<Internal configuration example of the back engine 13>
Next, a configuration example of the back engine 13 will be described with reference to FIG. In order to receive the image data output from the second output imaging interface 12k of the front engine 12, the back engine 13 receives a second input imaging interface (SIFi2) 13k (denoted as “SIFi2” in the figure). Is provided. A second pre-process circuit (PreProc2) 13m (indicated as “PreProc2” in the figure) is placed after the second input imaging interface 13k, and then a second selector (Sel for two systems) is input. ) 13n (denoted as “Sel” in the figure), and the second post-processing circuit 13p is further placed thereafter.

  As for the output of the second preprocess circuit 13m, the first output is connected to the first input of the second selector 13n, and the second output is the input of the RAW2 buffer 15c provided on the DRAM 15 of the back engine 13 ( DRAMC input port).

  On the other hand, the second input of the second selector 13n is connected to the output of the same RAW2 buffer 15c (DRAMC output port), and the output of the second selector 13n is connected to the input of the second post-processing circuit 13p. Finally, the output of the second post-processing circuit 13p is connected to the input (DRAMC output port) of the YCbCr2 buffer 15g provided on the DRAM 15.

  The circuit from the second input imaging interface 13k to the second post process circuit 13p in FIG. 5 has the same configuration as the circuit from the input imaging interface 51j to the second post process circuit 51q in FIG. This circuit can be configured by using the circuit shown in FIG.

  In the case of still image shooting, the back engine 13 causes the still image data output from the second output of the second preprocess circuit 13m to be sent to the RAW2 buffer 15c and read out from the RAW2 buffer 15c. The still image data is set to be input from the second input of the second selector 13n and sent to the second post-processing circuit 13p.

  The captured still image data is output from the second output imaging interface 12k of the front engine 12 and received by the second input imaging interface 13k of the back engine 13. Next, the still image data is sent to the second preprocess circuit 13m to be preprocessed, and is further output from the second output of the second preprocess circuit 13m and temporarily stored in the RAW2 buffer 15c. .

  In the second preprocess circuit 13m, AWB detection is also performed, and the AWB evaluation value extracted at this time is stored in the AWB buffer 15d of FIG. This is the first step (preprocessing) of still image processing. The CPU 13e (see FIG. 2) of the back engine 13 executes an AWB algorithm based on the AWB evaluation value to obtain a WB gain, and sets the WB gain in the second post process circuit 13p.

  Next, the second post-processing circuit 13p executes the second step (post-processing) of image processing. The process of the second step is generally executed by strip division. The second post-processing circuit 13p first divides the preprocessed still image data (Bayer) into several strip images on the RAW2 buffer 15c. Subsequently, the second post process circuit 13p reads them one by one and performs the post process. Further, the second post-processing circuit 13p stores the post-processed YCbCr data in the YCbCr2 buffer 15g of FIG. Therefore, when the second post-processing circuit 13p executes the process of the second step on all the strip images, the image processing is completed on the YCbCr2 buffer 15g, and an image connected to one is completed.

  In the second step, a main image, a display image (freeze image), and a thumbnail are created and stored in each YCbCr2 buffer 15g. The above is the second step (preprocess) of image processing of still images, but the flow of these still image data is basically the same as in FIG. 3 except for the point passing through the second image data transmission path. . Since the subsequent processing in the still image shooting is as described above, the description is omitted.

  On the other hand, in the case of moving image shooting, recording moving image data is output from the first output of the second preprocess circuit 13m, and the data is input from the first input of the second selector 13n. And set to be sent to the second post-processing circuit 13p.

  The moving image data for recording is output from the second output imaging interface 12k of the front engine 12 and received by the second input imaging interface 13k of the back engine 13. Next, the moving image data for recording is sent to the second preprocess circuit 13m to be preprocessed, and is further output from the first output of the second preprocess circuit 13m to be output to the first of the second selector 13n. Is sent to the second post-processing circuit 13p and subjected to post-processing. The moving image data for recording is finally output from the second post process circuit 13p and temporarily stored in the YCbCr2 buffer 15g of FIG.

  The moving image data (YCbCr) for recording is compressed by a moving image codec (not shown) (H.264, etc.) and then recorded on the memory card M1, but the details are omitted. During movie shooting, the second pre-process circuit 13m generally needs to perform focus adjustment (AF) at all times because the subject moves. The AF detection circuit extracts the AF evaluation value and uses it to adjust the focus (AF). To continue. During video recording, the second pre-processing circuit 13m needs to keep the brightness and white balance of the subject appropriately, and operates the AE detection circuit (AE evaluation value) and the AWB detection circuit (AWB evaluation value) as appropriate. Make those adjustments. The 3A detection will be described in detail in the description of the 3A detection circuit for live view described later.

  Here, the flow of the moving image data for recording is basically the same as that shown in FIG. 2 except for the point passing through the second image data transmission path. In short, the flow of still image data and moving image data for recording is the same in both cases where there are two image data transmission paths, and the front engine 12 and the back engine 13 of FIG. 5 having two image data transmission paths. Indicates that the circuit from the pre-process circuit 51k to the second post-process circuit 51q in FIG. 14 is configured to be used well.

  That is, the two engines in FIG. 5 are also made based on the engines shown in FIG. 2, FIG. 3, and FIG. This is also true for the live view described below.

<Flow of live view image data>
Next, the flow of live view image data will be described. As described above, the live view image data is transmitted from the front engine 12 to the back engine 13 through the newly established first image data transmission path. Just as the front engine 12 includes a new block for live view, the back engine 13 also has a new block for live view.

  These blocks are newly provided to effectively use the first post-processing circuit 51p in FIG. As shown in FIG. 5, a first input imaging interface (SIFi1) 13 a (indicated as “SIFi1” in the figure) is placed at the head of the back engine 13, followed by a first selector for two systems of inputs. (Sel) 13i is placed, and then the first post-processing circuit 13j is placed. Two blocks newly provided in the back engine 13 are the first input imaging interface 13a and the first selector 13i. The output of the first input imaging interface 13a is connected to the first input of the first selector 13i, while the second input of the first selector 13i is the same as the first output of the second preprocess circuit 13m. Further, the output of the selector is connected to the input of the first post-processing circuit 13j.

  When the front engine 12 is used, the first selector 13i is set so that the live view image data output from the first input imaging interface 13a is always sent to the first post-processing circuit 13j.

  On the other hand, when the imaging circuit 11 is directly connected to the second input imaging interface 13k of the back engine 13, the live view image data output from the second pre-processing circuit 13m is sent to the first post-processing circuit 13j. Thus, the first selector 13i is set. This corresponds to the data path from the preprocess circuit 51k to the first post process circuit 51p in FIG. In this case, the first input imaging interface 13a is not used.

  That is, the first selector 13 i in the back engine 13 uses the image data of the live view transmitted from the front engine 12 or the image pickup circuit 11 directly connected to the second input image pickup interface 13 k of the back engine 13. This is for selecting whether to use live view image data output from. This allows a back engine with two input imaging interfaces (SIFi1 and SIFi2) to be used with a conventional electronic camera.

  In the case of live view, the image data output from the first output imaging interface 12 c of the front engine 12 is received by the first input imaging interface 13 a of the back engine 13. Then, the image data is input to the first input of the first selector 13i, sent from there to the first post-processing circuit 13j, subjected to post-processing, and finally output from the first post-processing circuit 13j. And stored in the YCbCr1 buffer 15f of FIG.

  The live view YCbCr data (through image) is read from the YCbCr1 buffer 15f, sent to the LCD controller (not shown), and displayed on the LCD panel 17 (see FIG. 4). In the case of moving image shooting, it has already been described that the moving image data for recording is transmitted from the second image data in parallel with the transmission of the live view image data from the first image data transmission path.

  On the other hand, in the case of still image shooting, the front engine 12 transmits live view image data from the first image data transmission path in parallel with transmitting the captured still image data from the second image data transmission path. Can be sent. Thereby, all the characteristics of the electronic camera described above are realized, and the above-mentioned condition (D) is also satisfied.

  For live view, there are other things to consider. First, live view image data is only preprocessed by the back engine 13. The live view image data must be preprocessed before that.

  The back engine 13 also has a second preprocessing circuit 13m, which is used for processing still image data and moving image data for recording, and cannot be used for preprocessing of live view image data. A separate pre-processing circuit for live view is required. The first preprocess circuit 12h in the front engine 12 is provided for this purpose, and is newly installed together with the first image data transmission path.

  Since the first preprocess circuit 12h is provided in the front engine 12, an increase in the cost of the back engine 13 is suppressed, which is convenient when the back engine 13 is used in a conventional electronic camera.

  The next thing to consider is 3A detection. Since the second preprocess circuit 13m processes still image data and moving image data for recording, a 3A detection circuit (not shown) provided therein cannot be used.

  In the case of moving image shooting, the back engine 13 can also extract a 3A evaluation value from moving image data for recording using a 3A detection circuit built in the second preprocess circuit 13m. However, when processing still image data, the back engine 13 cannot use the 3A detection circuit as it is because the image data is different.

  Accordingly, a 3A detection circuit for live view is separately required. However, if this is provided in the first preprocess circuit 12h of the front engine 12, it is difficult to perform 3A software processing by the back engine 13. Since the evaluation value of 3A exists on the front engine 12 side, if the software on the back engine 13 side uses this, the processing is slowed down. Although 3A processing software can be installed on the front engine 12 side, there is a problem that it is difficult to share software because the system changes significantly from the conventional one.

  Therefore, the electronic camera 2 is provided with a 3A detection circuit for live view on the back engine 13 side. However, when the electronic camera 2 is simply added, the cost of the back engine 13 rises and the conventional electronic camera is used. Disadvantageous. In other words, in the second embodiment, the 3A detection circuit in the second preprocess circuit 13m of the back engine 13 is provided outside so that it can be used even in a live view. There is no particular problem in the case of moving image shooting, but in the case of still image shooting, there is a problem that the 3A detection circuit cannot be used for still image data.

  However, in the still image data processing, the 3A detection circuit is not fully used, and only the AWB detection circuit is required. Therefore, in the second embodiment, the 3A detection circuit is taken out of the second preprocess circuit 13m, and only the AWB detection circuit is added to the second preprocess circuit 13m.

  As a result, the back engine 13 extracts the 3A evaluation value from the live view image data, and the still image data is processed by the second pre-process circuit 13m of the back engine 13 and the AWB evaluation value is extracted therefrom. Can do.

  In FIG. 5, the “3A detection” block above the second preprocess circuit 13m represents the 3A detection circuit 13h. Even when the live view image data transmitted from the front engine 12 is input from the first input imaging interface 13a, the live view image data is directly connected to the second input imaging interface 13k of the back engine 13. Even when image data is input, the 3A detection circuit 13h must be used. Therefore, the output of the first selector 13i in the back engine 13 is connected to the input of the 3A detection circuit 13h as shown in FIG.

  The output of the 3A detection circuit 13h is connected to the input (input port of DRAMC) of the 3A buffer 15b in FIG. 5, and the extracted 3A evaluation value is stored in the 3A buffer 15b.

  On the other hand, the output of the AWB detection circuit added in the second preprocess circuit 13m is connected to the input (input port of DRAMC) of the AWB buffer 15d in FIG. 5, and the extracted AWB evaluation value is the AWB. Stored in the buffer 15d. By using these evaluation values, 3A processing can be performed by software on the back engine 13 side. The second image data transmission path is for relatively high-resolution images because still image data and moving image data for recording flow, and the first image data transmission path is relatively free because live view image data flows. For low-resolution images. The video data for recording in video shooting and the live view image data have the same frame rate (fps), but the former has a higher resolution, so the pixel rate of the image data flowing in the two image data transmission paths is the same. Will make a difference. In general, the pixel rate of the image data flowing through the second image data transmission path is higher.

  Therefore, the cost can be suppressed by providing a difference in the performance of the two image data transmission paths. The transmission clock frequency of the first image data transmission path is lowered or the number of Lanes of the data line is reduced to make the cost lower than that of the second image data transmission path.

  As described above, when the electronic camera 2 of the second embodiment receives high pixel rate image data, the electronic camera 2 records, for example, a certain frame as a full-resolution still image while performing live view. it can. Thereby, in the electronic camera 2, the cost of two engines can be suppressed.

(Third embodiment)
Next, a third embodiment will be described. Here, in the above embodiment, when the front engine 12 is used, it is possible to provide a large number of RAW1 buffers 14b on the DRAM 14 so that continuous shooting of high-speed still images can be continued for a long time. explained. Since still images generally have high resolution, even a single frame RAW1 buffer 14b has a large storage capacity, and it is not easy to provide a large number of them.

  However, if the captured still image data is once compressed and stored in the RAW1 buffer 14b, the amount of data per frame is reduced, so that more RAW1 buffers 14b can be provided. Therefore, in the electronic camera of the third embodiment, the front engine 12 is provided with an image compression circuit. Since the still image data stored in the RAW1 buffer 14b of the front engine 12 is the original data, the above-described image compression needs to be lossless compression with no distortion at all, or Nearly Lossless compression that generates only inconspicuous distortion. is there.

  Note that in the second embodiment and the third embodiment, the same components are denoted by the same reference numerals, and differences will be mainly described here.

  FIG. 6 is a block diagram illustrating a configuration example of the front engine 12 and the back engine 13 of the electronic camera according to the third embodiment. The third embodiment will be described using the electronic camera 2 shown in FIG. Here, the difference in configuration between FIG. 5 and FIG. 6 is expressed as an image compression circuit 12m (denoted as “compression” in the figure) and an image decompression circuit 12n (decompressed in the figure) in the front engine 12. ) And are newly provided.

  According to FIG. 6, the output of the defective pixel correction circuit (DPC) is connected to the input of the image compression circuit 12m, and the output of the image compression circuit 12m is connected to the input of the RAW1 buffer 14b (DRAMC input port) on the DRAM. Yes.

  The still image data output from the defective pixel correction circuit (DPC) is compressed by the image compression circuit 12m, and the compressed data is output from the image compression circuit 12m and temporarily stored in the RAW1 buffer 14b in the figure. . In image compression, since compression is generally performed using correlation between pixels, processing cannot be performed if defective pixel data is included. However, in FIG. 6, there is no problem because defective pixel correction is performed first. Since the image compression circuit 12m must compress the high-pixel-rate still image data output from the image pickup circuit 11 in real time, this real-time compression is achieved by providing a plurality of image compression cores in the interior and processing them in parallel. Is realized.

  The real-time processing is also performed in the first image reduction circuit 12b and the second image reduction circuit 12i in FIG. 6, and the pixel rate is reduced in real time by taking an average of a plurality of pixels.

  On the other hand, the first pre-process circuit (PreProc1) 12h in FIG. 6 processes image data with a reduced pixel rate, and thus may be a normal pipeline circuit. Since the compressed data is stored in the RAW1 buffer 14b, the number of accesses to the DRAM 14 is reduced, resulting in a merit that the bandwidth occupied by the DRAM 14 is reduced. By providing the image compression circuit 12m, many RAW1 buffers 14b can be provided in the same DRAM.

  However, next, the problem of how to transfer the still image data stored therein to the back engine 13 is faced. Even if the compressed still image data is transmitted as it is, the back engine 13 cannot process it.

  Therefore, in the third embodiment, an image decompression circuit 12n is provided in the front engine 12, and the compressed still image data is decompressed and transmitted to the back engine 13.

  That is, in the third embodiment, since the expanded still image data is transmitted, the conventional back engine 13 can be used as it is, and the added image compression circuit 12m and the image expansion circuit 12n are integrated on the front engine 12 side. Therefore, there are two merits that the cost of the back engine 13 can be suppressed.

  As described above, the front engine 12 of FIG. 6 includes the image compression circuit 12m and the image expansion circuit 12n. The output of the RAW1 buffer 14b (DRAMC output port) is connected to the input of the image expansion circuit 12n, and the output of the image expansion circuit 12n is connected to the second input of the selector 12j in the figure. The compressed data of the still image is first read from the RAW1 buffer 14b and sent to the image decompression circuit 12n. Next, the compressed data is decompressed and output from the image decompression circuit 12n, and further passes through the selector 12j to the second data. It is sent to the output imaging interface 12k, and finally transmitted from the second input imaging interface 12k to the back engine 13.

  The decompressed still image data returns to the original amount (increases), but the pixel rate is reduced as planned because the decompression is performed in accordance with the transmission rate of the second image data transmission path.

  5 and 6, the first pre-process circuit 12h in the front engine 12 and the first post-process circuit 13j in the back engine 13 are combined into one image processing circuit (image processing circuit G1). : Refers to the dotted frame). In the figure, it is described as “image processing”.

  In addition, the second pre-process circuit 13m and the second post-process circuit 13p in the back engine 13 together constitute one image processing circuit (image processing circuit G2: refer to the dotted frame). In the figure, it is described as “image processing”.

  In other words, the back engine 13 performs image processing on the first image data received by the first input imaging interface 13a for low resolution by the image processing circuit G1, and the second input imaging interface 13k for high resolution The received second image data is subjected to image processing by the image processing circuit G2. Therefore, the back engine 13 can independently process the first image data and the second image data.

  The above is the entire picture of the flow of image data in the front engine 12 and the back engine 13 having two image data transmission paths.

  Next, supplementary explanation will be given for continuous shooting when a large number of RAW1 buffers 14b are provided on the DRAM 14 of the front engine 12. FIG. When the flow of image data in still image shooting is described with reference to FIG. 3, in the electronic camera 1, after the still image data is stored in the RAW 1 buffer 14 b, the still image data is unconditionally read and transmitted to the back engine 13. However, in practice, transmission is not possible unless the RAW2 buffer 15c on the back engine 13 side is empty.

  On the other hand, when the RAW1 buffer 14b is not empty, still image data cannot be output from the imaging circuit 11. Since the image data cannot be kept in the image pickup circuit 11 (image pickup sensor) for a long time after the exposure is completed, the exposure should not be started when the RAW1 buffer 14b is not empty.

  That is, the front engine 12 must not send the captured still image data to the back engine 13 without permission, and the front engine 12 must not start capturing a still image without permission.

  Since the image capture sequence is controlled by the CPU 13e on the back engine 13 side as the capture sequence control means, the front engine only when the still image data is stored in the RAW1 buffer 14b and the RAW2 buffer 15c is empty. 12 issues a data transmission command to the back engine 13, and transmits a still image shooting command from the front engine 12 to the back engine 13 only when the RAW1 buffer 14b is empty.

  That is, at the time of still image shooting, the front engine 12 issues a command after confirming the state of the RAW1 buffer 14b and the RAW2 buffer 15c. Of these, when confirming the state of the RAW1 buffer 14b, the back engine 13 and the front engine Communication between 12 occurs. In this case, the shooting sequence control means transmits a shooting sequence command to the front engine 12 during shooting. Then, the front engine 12 receives the command and controls at least one of the operation of the imaging circuit 11 and the reading of the image data. Thus, since the shooting sequence is controlled on the back engine 13 side, the front engine 12 side can specialize in communication control with an image sensor having a high pixel rate. That is, a general-purpose back engine can be employed on the back engine 13 side.

  On the other hand, in the case of a conventional electronic camera in which an image pickup circuit is directly connected to the back engine 13, it is assumed that high pixel rate image data is not received. Therefore, if a RAW2 buffer is available, still image shooting starts. There is a difference that you can.

  As described above, according to the electronic camera 2 of the third embodiment, since the expanded still image data is transmitted, the conventional back engine 13 can be used as it is. Further, according to the electronic camera 2, the added image compression circuit 12m and the image expansion circuit 12n are integrated on the front engine 12 side, so that the cost of the back engine 13 can be suppressed. Therefore, according to the electronic camera 2, a highly versatile camera system can be constructed.

(Fourth embodiment)
Next, a fourth embodiment will be described. In the fourth embodiment, taking a still image is considered from the viewpoint of software in order to explain the above-described command in more detail. The purpose is to share and reuse software. First, as a comparative example, software (firmware) of a conventional electronic camera having only a back engine (not requiring a front engine) will be described.

  FIG. 7 is a diagram showing an example of the structure of software used in a conventional electronic camera. In FIG. 7, the software includes an application layer 101, a middleware layer 102, a kernel layer 103, a device driver layer 104, and a hardware layer 105.

  The highest layer is the application layer 101, and is configured by a program that realizes a function specific to the product.

  FIG. 7 shows application programs for realizing the main functions of the electronic camera, and these programs realize a target function by calling a program below it. That is, an API for calling a lower-level program is prepared. As the API, for example, still image shooting 101a, moving image shooting 101b, live view 101c, image playback 101d, image communication 101e, and camera setting 101f are prepared. If these APIs are standardized, they can be used in an electronic camera for each application program. Below the application layer 101 is a middleware layer 102, which includes general-purpose library software. For example, the middleware layer 102 includes a middleware 102a and a library 102b.

The kernel layer 103 includes an RTOS (Real Time Operating System) 103a. The device driver layer 104 includes other peripheral drivers 104a, an image processing driver 104b, and an imaging driver 104c. The hardware layer 105 includes other peripheral hardware 105a, a back engine 105b, and an imaging circuit 105c as hardware control software. Here, some of the library 102b is purchased from a software vendor. The middleware layer 102 is configured by a highly common program that is used by a plurality of application programs. What is important is to make the program independent of the detailed structure of the hardware. For example, resolution conversion (magnification) for changing the size of an image is used for both image shooting and image reproduction. The processing is performed by an image processing circuit (post-processing circuit) 13b in the back engine 13 (see FIG. 3).

  However, the middleware layer 102 program does not directly move the post-processing circuit. For example, the address of the buffer memory in which the image data before scaling is stored, the image size before scaling, and the image before scaling Lower-level program by specifying arguments such as the coordinate value and cut-out size for cutting out the pixels for resolution conversion from, the address of the buffer memory that stores the image data after scaling, and the image size (or magnification) after scaling Call.

  For example, an API that calls a resolution conversion function does not specify what processing of the resolution conversion function is to be performed. That is, the API is configured so as not to depend on hardware for performing resolution conversion (abstraction). It is the program called by this API that actually activates the post-processing circuit.

  Specifically, the image processing driver 104b, which is a program for the device driver layer 104, activates the post-process circuit. Below the middleware layer 102 is a kernel layer 103 on which an RTOS 103a is placed. As described above, in the electronic camera, a plurality of processes are executed in parallel. Parallel processing is realized by using a program for performing these processes as a task of the RTOS 103a. Below the kernel layer 103 is a device driver layer 104, and programs for operating various hardware included in the hardware layer 105 thereunder are placed.

  The hardware layer 105 is connected to an imaging circuit and a back engine in addition to a circuit inside the back engine including an image processing circuit (preprocess circuit, post process circuit), JPEG codec, LCD controller, CARD controller, USB controller, and the like. Peripheral devices are included.

  Since the hardware is abstracted by providing the device driver layer 104, even if the hardware changes, the higher-level program can be moved as it is only by changing the device driver.

  However, an API that calls a lower layer program from an upper layer program needs to be standardized. Based on the above, the software of the electronic camera 1 will be described.

<About software of electronic camera 1>
Although the electronic camera 1 is newly provided with a front engine 12, it has already been described that the front engine 12 is abstracted so as to be shared with software of a conventional electronic camera.

  FIG. 8 is a diagram illustrating the virtual imaging circuit. Specifically, as shown in FIG. 8, the front engine 12 and the imaging circuit connected thereto are combined and abstracted as a virtual imaging circuit. That is, the output imaging interface (SIFo1 and SIFo2) of the front engine 12 is the same as the interface for the imaging circuit to output image data. Thereby, the abstraction of a virtual imaging circuit is realized. By performing this abstraction, the circuit on the left side of FIG. 8 looks like the circuit on the right side. In other words, the left circuit and the right circuit in FIG. 8 can be treated as equivalent circuits. Thereafter, the virtual imaging circuit is connected to the “virtual imaging circuit”, the imaging circuit used in the conventional electronic camera (see FIG. 10) is connected to the “normal imaging circuit”, and the front engine 12 of the electronic camera 1 is connected. The image pickup circuit (left side in FIG. 8) is called a “high-speed image pickup circuit” for distinction.

  The abstraction means that the high-speed imaging circuit 11a and the front engine 12 shown in FIG. 8 are integrated and regarded as a virtual virtual imaging circuit 30, and the virtual imaging circuit 30 is a normal imaging circuit (for example, FIG. 10). It is important to operate in the same manner as the imaging circuit 52) shown in FIG. This is because conventional electronic camera software can be used as it is. This is the purpose of abstraction.

  The normal imaging circuit outputs image data having a pixel rate that can be directly received by the back engine 13. On the other hand, the image data output from the virtual imaging circuit 30 has a reduced pixel rate and can be directly received by the back engine 13, and thus has functional similarity.

  Corresponding to the front engine 12 having two output imaging interfaces (SIFo1 and SIFo2), the virtual imaging circuit 30 also has two interfaces for image data output, but the normal imaging circuit has one interface. This is a big difference.

  However, from the viewpoint of software, the difference in the number of interfaces is not a problem. This difference is because it is shielded by the device driver layer 104 and is not visible to the upper layer program. What is important is the functional difference. The difference is that the virtual imaging circuit 30 can output two different image data simultaneously, and this corresponds to the operation of the front engine 12 already described. Live view image data is output from the solid-line arrow that extends outward from the virtual imaging circuit 30 on the right side of FIG. Still image data and moving image data for recording are output from dotted arrows.

  That is, it is possible to perform parallel operations such as outputting still image data shot while performing live view, or outputting moving image data for recording while performing live view.

  On the other hand, when two different image data are output from the normal imaging circuit, a sequential operation is performed. In the case of a normal imaging circuit, after one image data output mode ends, another image data output mode is shifted to. However, in the case of the virtual imaging circuit 30, another image data output mode is maintained while the one image data output mode is operated. The control sequence is different.

  Although the circuit on the right side of FIG. 8 and the circuit of FIG. 10 are the same in appearance, the electronic camera 1 and the electronic camera 50 have slightly different software because the control sequence is different.

  However, the upper layer software is considerably shared by the abstraction of the virtual imaging circuit 30. That is, almost the same API can be used to move the virtual imaging circuit 30, and an upper layer program can capture an image by calling this API. As described above, although a slight change is necessary, the upper layer program can be appropriately used as the upper layer program because it can be created using substantially the same API.

  In the circuit on the right side of FIG. 8, an arrow from the back engine 13 toward the virtual imaging circuit 30 is a virtual control signal for moving the virtual imaging circuit 30. This virtual control signal is also part of the abstraction and is controlled using the API described above. As a result of the abstraction of the virtual imaging circuit 30, the software on the back engine 13 side is as shown on the left side of FIG.

  FIG. 9 is a diagram illustrating an example of the software structure of the electronic camera 1. The device driver layer 104 on the left side of FIG. 9 has a virtual imaging driver 104d which is a kind of software interface, and this operates the virtual imaging circuit 30 on the right side of FIG. The virtual imaging driver 104d is called from the upper layer program using the above-described API, but since there is no imaging circuit in the hardware layer 105 below, the actual processing is the same as the imaging driver 104c shown in FIG. Is different. That is, the virtual imaging driver 104d further calls the communication driver 104e below the virtual imaging driver 104d, and the communication driver 104e transmits commands and parameters relating to imaging to the front engine 12.

  Since the API described above is called from the upper layer program, only a small number of upper layer information is passed to the argument. For this reason, the parameters sent to the front engine 12 are also reduced, and communication overhead is suppressed.

  The hardware layer 105 on the left side of FIG. 9 includes a communication interface 105d, and this interface is used for communication. Since image data is output from the virtual imaging circuit 30, the image processing circuit 13b in the back engine 13 must receive and process the information sent from the communication interface 105d, and the upper layer program executes virtual imaging. When calling the driver API, it is necessary to call the API of the left image processing driver 104b shown in FIG. The two interfaces of the virtual imaging circuit 30 are not visible from the higher-level API because the image processing driver 104b absorbs the two interfaces.

  Looking at the device driver layer 104 on the left side of FIG. 9, the configuration is the same as that of the device driver layer 104 in FIG. I understand that.

  Next, the software on the front engine 12 side shown on the right side of FIG. 9 will be described. The software on the front engine 12 side is also hierarchized in the same way as the back engine 13 side. The software on the front engine 12 side includes an application layer 201, a middleware layer 202, a kernel layer 203, a device driver layer 204, and a hardware layer 205.

  The application layer 201 includes a still image shooting 201a, a moving image shooting 201b, and a live view 201c. The middleware layer 202 includes middleware 202a and a library 202b. The kernel layer 203 has an RTOS 203a. The device driver layer 204 includes an image processing driver 204a, a communication driver 204b, and an imaging driver 204c. The hardware layer 205 includes a front engine 205a, a communication interface 205b, and an imaging circuit 205c.

  The communication interface 205b of the lowermost hardware layer 205 receives the information sent from the back engine 13. This is a physical layer view, and the logical layer view is that the communication driver 204b in the device driver layer 204 above receives this information.

  When the front engine 12 returns a response, information is transmitted in the reverse direction. The imaging circuit 205c in the hardware layer 205 is the high-speed imaging circuit 11a, and high-speed image data is output from the high-speed imaging circuit 11a.

  The high-speed imaging circuit 11a is driven by the imaging driver 204c of the device driver layer 204 thereon. The remaining one of the hardware layers 205 collectively represents the internal blocks of the front engine 12, and these blocks are all related to the processing of image data output from the high-speed imaging circuit 11a.

  These blocks are driven by an image processing driver 204a in the device driver layer 204 one level above. The software on it has the same configuration as that on the back engine 13 side, and supplementary explanation will be given for the program of the application layer 201 therein.

  On the front engine 12 side, application programs such as still image shooting 201a, moving image shooting 201b, and live view (LiveView) 201c are described. These application programs are only programs that interpret commands corresponding to the application program on the back engine 13 side. The three application programs of still image shooting 201a, moving image shooting 201b, and live view 201c are basic operation modes of the electronic camera. These application programs are determined by the application layer 201, but the information is obtained from the application layer 201. Need to be communicated to the lower layers.

  This is because the processing content of the image data and the data path change depending on the operation mode. Therefore, the command and parameters received from the back engine 13 are first passed to the application program to interpret the command. As a result, when the operation mode is determined, the API described above is called by using the parameter received together as an argument, and the operation according to the mode is performed. As a result, the software on the front engine 12 side also has a hierarchical structure similar to that on the back engine 13 side.

  However, the hierarchical structure on the side of the front engine 12 is an example, and may be a simpler hierarchical structure. Further, since the functions on the front engine 12 side are limited, the use of a general-purpose library may be reduced.

  On the other hand, on the front engine 12 side, still image data is transmitted to the back engine 13 while performing live view, or the image data output from the virtual imaging circuit 30 is stored in the RAW1 buffer 14b in the DRAM 14 while another RAW1 is stored. Parallel processing such as transmitting image data read from a buffer (not shown) to the back engine 13 is performed. Therefore, it is necessary to introduce the RTOS 203a on the front engine 12 side to perform multitask processing. Although the operation of the virtual image pickup circuit 30 is slightly different from that of the normal image pickup circuit due to the two interfaces, the case of still image shooting will be described as another difference.

  First, the operations of the normal imaging circuit and the high-speed imaging circuit 11a are: (i) exposing a subject and accumulating signal charges in an imaging sensor; (ii) converting the accumulated charge signals into digital data, and each imaging circuit. It consists of two outputs:

  In still image shooting by the electronic camera 50, still image data output from the normal imaging circuit (imaging circuit 52 in FIG. 10) under the condition (ii) is received by the back engine 51, and then preprocessed by the preprocessing circuit. Then, the operation of storing in the RAW buffer 53d is performed.

  On the other hand, as shown in FIG. 8, the operation of the virtual imaging circuit 30 also includes (I) generating a digital still image data by exposing a subject, and (II) outputting the generated still image data from the high-speed imaging circuit 11a. Can be divided into two conditions.

  In still image shooting with the electronic camera 1, the virtual imaging circuit 30 sequentially performs the operations of the condition (I) and the condition (II). In the case of the condition (II), the still image data output from the virtual imaging circuit 30. Is received and processed by the back engine 13 as in the conventional electronic camera. Since the operation of the virtual imaging circuit 30 includes the operation of the high-speed imaging circuit 11a and the front engine 12, what operation the conditions (I) and (II) correspond to will be described.

  The operation of the high-speed imaging circuit 11a is composed of the condition (i) and the condition (ii). The operation of the front engine 12 is (a) storing still image data output from the high-speed imaging circuit 11a in the RAW1 buffer 14b. (B) The two conditions are that the still image data stored in the RAW1 buffer 14b is read and transmitted to the back engine 13.

  Among these, since the condition (ii) and the condition (a) are parallel operations, the high-speed imaging circuit 11a and the front engine 12 are interlocked. However, the condition (i) is the single high-speed imaging circuit 11a. It becomes the operation of. The operation of the condition (b) is independent of the operation of the high-speed imaging circuit 11a although the front engine 12 and the back engine 13 are interlocked. At this time, the transmitted still image data is preprocessed. The operation stored in the RAW2 buffer 15c is performed in parallel on the back engine 13 side. It is natural to think that the operation of the condition (b) of the front engine 12 corresponds to the operation of the condition (ii) of the normal imaging circuit and the operation of the condition (II) of the virtual imaging circuit 30.

  Then, the operation of the condition (I) of the virtual imaging circuit 30 is “the operation of the high-speed imaging circuit 11a (condition (i)) + the operation of the high-speed imaging circuit 11a (condition (ii)) + the operation of the front engine 12 (condition (a) )) ”.

  That is, the operation from the exposure of the subject by the high-speed imaging circuit 11a to the storage of the still image data in the RAW1 buffer 14b corresponds to the operation of the condition (I) of the virtual imaging circuit 30, and the still image stored in the RAW1 buffer 14b. It is considered that the operation of reading data and transmitting it to the back engine 13 corresponds to the operation of the condition (II) of the virtual imaging circuit 30. When the two operations of the virtual imaging circuit 30 are made to correspond as described above, the operation of the condition (I) of the virtual imaging circuit 30 and the operation of the condition (i) of the normal imaging circuit (for example, the imaging circuit 52) are the same. Yes, it is examined whether the operation of the condition (II) of the virtual imaging circuit 30 and the operation of the condition (ii) of the normal imaging circuit can be considered the same.

  If these operations are the same, the conventional electronic camera can be used as it is for the still image shooting software. However, there are differences in the operation of the two imaging circuits (normal imaging circuit, virtual imaging circuit). Since several RAW1 buffers 14b are provided on the front engine 12 side, it is possible to perform an operation of storing captured still image data therein (RAW buffer continuous shooting).

  On the other hand, in parallel with this, an operation of reading still image data previously captured from the RAW1 buffer 14b and transmitting it to the back engine 13 is performed.

  The former (RAW buffer continuous shooting) is an operation of the condition (I), which can be considered as an “exposure operation” of the virtual imaging circuit 30. The latter is an operation under the condition (II), which can be considered as “an output operation of still image data” of the virtual imaging circuit 30. The condition (i) that is the exposure operation of the normal imaging circuit and the condition (ii) that is the output operation of the still image data are inseparable operations, but the conditions (I) and (II) of the virtual imaging circuit 30 There are differences in behavior. Indivisible means that the exposure of the next still image cannot be started until the captured still image data is output from the imaging circuit. The high-speed imaging circuit 11a is the same that the operations of the condition (i) and the condition (ii) are inseparable.

  On the other hand, the virtual imaging circuit 30 has a feature that the electronic camera 1 can start exposure of the next still image before outputting the captured still image data. That is, the captured still image data can be stored in the virtual imaging circuit 30. This is a characteristic caused by having several RAW1 buffers 14b. When the exposure of the next still image is started after the captured still image data is output from the virtual imaging circuit 30, there may be one RAW1 buffer 14b, and there is no reason to provide several RAW1 buffers. Therefore, in the electronic camera 1, by providing a number of RAW1 buffers 14b, it is possible to perform high-speed continuous shooting that does not depend on the output operation of still image data.

  However, the next exposure must not be started when there is no more empty RAW1 buffer 14b. When the virtual imaging circuit 30 is abstracted, the RAW1 buffer 14b cannot be seen from the upper-layer program on the back engine 13 side. Therefore, the virtual imaging circuit 30 provides the state “State1”. Thereby, in the virtual imaging circuit 30, when the RAW1 buffer 14b is not empty, State1 is set to BUSY, and when the RAW1 buffer 14b is empty, State1 is set to READY and this information is used.

  The state 1 is transmitted from the virtual imaging driver 104d to the upper layer program. A program for taking a still image calls the above-described API to acquire the state 1 from the virtual imaging driver 104d, and when this is BUSY, the virtual imaging circuit 30 waits until READY and starts exposure when State1 becomes READY. .

  On the other hand, there is a similar situation when the captured still image data is output from the virtual imaging circuit 30. That is, when the RAW1 buffer 14b is empty, the virtual imaging circuit 30 cannot output still image data. Therefore, the virtual imaging circuit 30 provides another “State 2” state. Thus, in the virtual imaging circuit 30, when still image data is stored in the RAW1 buffer 14b, State2 is set to READY, and when the RAW1 buffer 14b is empty, State2 is set to BUSY and this information is used. To do.

  The state 2 is also acquired from the virtual imaging driver 104d. When the RAW2 buffer 15c on the back engine 13 side is not empty, the virtual imaging circuit 30 cannot output still image data, so the state must also be checked. The state 1 and state 2 are acquired from the front engine 12 side by the virtual imaging driver 104d via the communication driver 104e, while the state of the RAW2 buffer 15c is acquired on the back engine 13 side.

  The still image shooting program, which is one of the image shooting software, first checks the state of the RAW2 buffer 15c, and when it is free, calls the above-described API to state the state 2 from the virtual imaging driver 104d. To get. When the state of State2 is BUSY, the still image shooting program waits until it becomes READY, and starts outputting still image data when State2 becomes READY. Note that the interruption of the program is reduced by knowing the timing of the next exposure start and the next still image data output timing by using an interrupt instead of the API (polling) for periodically checking the end of processing. Increases efficiency.

  That is, an interrupt occurs every time the state 1 or state 2 of the virtual imaging circuit 30 becomes READY, and the virtual imaging driver 104d notifies the still image shooting program that the next exposure or still image data output is possible. .

  However, the State2 interrupt is generated not by itself but by AND (logical product) with the state of the RAW2 buffer 15c (see FIG. 3). When an interrupt occurs, the still image shooting program calls the above-described API to cause the virtual imaging circuit 30 to perform exposure and output of still image data. However, as described above, these two are independent operations. It is necessary to prepare the API separately.

  That is, the still image shooting program calls each API in response to the generated interrupt. On the other hand, in the case of a normal imaging circuit, since these two operations are inseparable (integral) operations, a still image shooting program can be called with one API. In the case of the virtual imaging circuit 30, exposure can be performed if State1 is READY, and still image data can be output if State2 is READY and the RAW2 buffer 15c is free. Here, the problem is which of the two methods is faster.

  In the exposure operation of the virtual image pickup circuit 30, the operation until the photographed still image data is stored in the RAW1 buffer 14b is performed. In the output operation of the still image data, the still image data is read from the RAW1 buffer 14b and the back engine 13 is read. The operation to transmit to is performed.

  In the latter still image data output operation, since the pixel rate is reduced, the output operation tends to be delayed. If the processing on the back engine 13 side is slow, this delay is further increased. For this reason, although a plurality of RAW2 buffers 15c are provided to prevent the output of still image data from being delayed, there is a limit because other buffer memories (YCbCr1, YCbCr2, JPEG, etc.) are also provided on the back engine 13 side. is there.

That is, in the case of RAW buffer continuous shooting, a state in which a plurality of still image data that has been shot remains in the virtual imaging circuit 30 occurs. It is necessary to designate a frame number both when performing exposure in this state and when outputting still image data. This is because if the frame number is not specified, there is a risk that the shooting order will be mistaken. Therefore, an argument called a frame number is provided for each of the two APIs described above. In still image shooting, the following API functions that call two functions (operations) of the virtual imaging circuit 30 are considered.
Exposure operation: SensorExposure (n, ...), ----- (3-1)
Still image data output operation: StillImageDataOut (m, ...) ----- (3-2)
“N” in parentheses of the function SensorExposure () is an argument for designating a frame number at the time of exposure (photographing). The other arguments are represented by “...”, But the exposure time, the sensitivity of the imaging circuit, and the like are entered here.

  On the other hand, “m” in parentheses of the function “StillImageDataOut ()” is an argument for designating a frame number when outputting still image data from the virtual imaging circuit 30. If the last exposed (photographed) frame number is “Nmax”, “m ≦ Nmax” is always satisfied. The initial value of the frame number n at the time of exposure is “0”, and the still image shooting program increments n by “1” before starting the exposure, and then calls the API function for the exposure operation.

  Therefore, the frame number n at the time of exposure is simply increased by 1 as “1 → 2 → 3 →. The frame number n at the time of exposure is managed as a set together with still image data sent to the virtual imaging circuit 30 and stored in the virtual imaging circuit 30 after shooting. This management is performed by a program on the front engine 12 side. As a result, it is possible to always know at which frame each of the still image data stored in the virtual imaging circuit 30 is taken. The still image shooting program manages the frame number n at the time of exposure so that the number of frames shot can be known.

  On the other hand, the initial value of the frame number m at the time of outputting the still image data is also “0”, but this person can output any of the still image data remaining in the virtual imaging circuit 30, so the degree of freedom. There is.

  However, since still image data is normally output in the order of shooting, the still image shooting program increments m by “1” before starting output, and then calls an API function for the still image data output operation. Accordingly, the frame number m at the time of outputting the still image data is simply increased by 1 as “1 → 2 → 3 →...”. This frame number m is also sent to the virtual imaging circuit 30 and compared with the frame number n of the still image data remaining inside, and the still image data with the matching frame number is selected and is selected from the virtual imaging circuit 30. Is output.

  The program on the front engine 12 side returns the frame number (m = n) of the still image data output in the response to the still image data output command received from the back engine 13. The frame number of the response is first received by the communication driver 104e on the back engine 13 side, and then transmitted to the still image capturing program via the virtual image capturing driver 104d thereon. By exchanging the frame number with this still image shooting program, it is possible to process still image data shot while correctly grasping the frame number.

  The still image shooting program manages the frame number m at the time of still image data output so that it can know which still image data has been output. The still image shooting program manages both the frame number n at the time of exposure and the frame number m at the time of still image data output, so that it is always possible to determine how many still image data still remain in the virtual imaging circuit 30. I can know.

  By the way, “still image data is output from the virtual imaging circuit 30 in the order of shooting” is described above. However, when an API such as StillImageDataOut () is used, any still image data remaining in the virtual imaging circuit 30 is output. You can also

  That is, an API such as StillImageDataOut () can also output still image data from the virtual imaging circuit 30 in an order different from the imaging order. Depending on the specifications of the product of the electronic camera, this operation can give the electronic camera a new feature.

  Specifically, when the RAW buffer continuous shooting is performed, if the playback button (PLAY button) is suddenly pressed, the setting is made to shift to the playback mode. In this case, it can be said that it is appropriate to play back from the shot still image, and in the still image playback mode, the frame number is played back from the last shot (latest) still image. Is preferred.

  Then, the still image data captured at the end of the interrupted continuous shooting is output and played back first, but if it is an API function such as StillImageDataOut (), the frame number m is specified to specify the still image. Since data can be output, the desired reproduction is realized.

  It should be noted that the arguments of StillImageDataOut () may be appropriately added if necessary except for the frame number m. The electronic camera 1 can perform various operations by preparing APIs for the exposure operation (3-1) and the still image data output operation (3-2).

  By the way, as described above, in the still image shooting of the conventional electronic camera, the exposure operation of the normal imaging circuit and the output operation of the still image data are performed inseparably (integrated), so that it is called by one API.

Therefore, let us examine the operation of the normal imaging circuit during still image shooting. The operation of this normal imaging circuit is classified into the following two conditions, for example.
[1] Still image data output automatically starts after the exposure operation is completed. [2] Different operations are required at the start of the exposure operation and at the start of the still image data output. The type of the condition [1] is 2 One action is literally inseparable and can only be called with one API. On the other hand, the type of condition [2] has a different meaning from inseparable because the two operations are caused by different operations.

  That is, it is inseparable in the system sense that the next exposure is not started until the captured (exposed) still image data is output. From the still image shooting program of the upper layer, these two may appear to be one operation, so that they are called by one API, but in the imaging driver 104c (see FIG. 7) called by the API, 2 It is divided into three sequential operations. That is, the two operations of the condition [2] type are executed separately in the imaging driver 104c, and are only invisible to the still image imaging program.

  However, if it is divided into two operations, they may be called from the API of the still image shooting program. In other words, the API for the exposure operation of (3-1) and the output operation of the still image data of (3-2) can be used, which means that the normal imaging circuit is viewed like the virtual imaging circuit 30. . In such a view, the number of RAW1 buffers 14b in the virtual imaging circuit 30 must be considered as one. If there is one RAW1 buffer 14b, the inseparability that the next exposure cannot be started unless the still image data stored therein is output is reproduced.

  In addition, the two states of State 1 and State 2 of the virtual imaging circuit 30 are also applied to the normal imaging circuit as they are. In other words, when one virtual RAW1 buffer in the normal imaging circuit is free, State1 is READY and State2 is BUSY, and when still image data is stored there, State1 is BUSY and State2 is Become READY.

  If an interrupt is generated according to these states, the timing of exposure of the normal imaging circuit and the output of the still image data can be known, so that continuous shooting can be continued using this interrupt. The states of State1 and State2 are transmitted from the imaging driver 104c in FIG. 7 to the still image shooting program in the upper layer. In the case of a normal imaging circuit, there should be no time from the end of the exposure operation to the start of the still image data output operation (because the charge signal deteriorates).

  Accordingly, the still image shooting program continuously calls the API for the exposure operation (3-1) and the output operation for the still image data (3-2) described above. In that case, the still image shooting program must match the two frame numbers (m = n). This is natural because there is only one RAW1 buffer 14b. If attention is paid to this point, it is possible to shoot a still image using the API of (3) even with a conventional electronic camera. Subsequently, the process proceeds to the normal imaging circuit of the condition [1] type. However, this cannot be divided into two APIs, and only the function (operation) is called by one API.

However, the API that calls the functions (operations) of the virtual imaging circuit 30 and the normal imaging circuit can be matched with the API of the condition [1] type. First, in addition to the frame number n at the time of exposure and the frame number m at the time of still image data output, a total of four arguments of “exp_op” representing the exposure operation and “dout_op” representing the output operation of the still image data are introduced. “Op” in “exp_op” and “dout_op” means “operation”. “Exp_op” and “dout_op” take values of “1” and “0”, but “1” represents an “operating state” and “0” represents a “non-operating state”. Furthermore, only one of the following API functions called from the still image shooting program is prepared.
SendorDrive (exp_op, n, dout_op, m, ...) ----- (4)
In the case of a normal imaging circuit of the condition [1] type, the still image shooting program sets the above-described arguments as follows and calls this API function.

Condition [1] type normal imaging circuit: SendorDrive (1, n, 1, n, ...) ----- (5)
Here, since the exposure operation and the still image data output operation are performed inseparably, one API call is sufficient as in the API function (5). The imaging driver 104c shown in FIG. 7 is called by this API function, in which the exposure operation is first performed, and when it is completed, the still image data output operation is automatically performed.

  Subsequently, the normal imaging circuit of the type of condition [2], the API function of (5) can be used as it is. However, in this case, the exposure operation and the still image data output operation are executed separately in the imaging driver 104c. The imaging driver 104c operates the normal imaging circuit to perform the exposure operation first, and when it is finished, performs another operation to output still image data. Since these appear to be one operation from the still image shooting program, it is necessary to prepare the imaging driver 104c according to the type of the normal imaging circuit.

  However, in the case of the condition [2] type, these two operations can be performed by calling the API function (4) twice from the still image shooting program. In this case, calls are made in the order of the following (6-1) and (6-2).

Conditional imaging circuit of type [2]:
SendorDrive (1, n, 0,0, ...) ----- (6-1)
SendorDrive (0,0,1, n, ...) ----- (6-2)
Here, the API function (6-1) is the first API call, but it can be seen from the value of the argument that it is an exposure operation. The API function (6-2) is the second API call, and it can be seen that the output operation of the still image data is called from the value of the argument. The API function (6-1) appears to be two operations from the still image shooting program, but it is inseparable because the frame number at the time of exposure and the frame number at the time of outputting the still image data are both coincident with “n”. You can see that it is working. That is, by introducing arguments representing two operations, two operations can be called with one API function. Therefore, by looking at the API functions (6-1) and (6-2), it is easy to see how to use this API function to call the two operations of the virtual imaging circuit 30. It looks like this:

Exposure operation of the virtual imaging circuit 30: SendorDrive (1, n, 0, 0, ...)
Still image data output operation of the virtual imaging circuit 30: SendorDrive (0, 0, 1, m, ...)
Since the two operations of the virtual imaging circuit 30 are independent operations, different values are set such that the frame number at the time of exposure is “n” and the frame number at the time of still image data output is “m”. If this setting is made, the API used in the case of still image shooting can be shared by the electronic camera 50 and the electronic cameras 1 and 2 to some extent.

  However, it is assumed that the API of (4) is difficult to use because there are places where it is forcibly matched. Therefore, when it is difficult to use, it is preferable to prepare an easy-to-use API by reviewing arguments and function names. That is, the important point is not to make the APIs coincide. The API is a function that performs a certain amount of processing (macro function), but a large application program function is realized by combining several of these APIs. In other words, it is important to prepare several appropriate APIs.

  In addition, by standardizing APIs, there will be a merit that they can be used on other models (software componentization and commonization). In other words, the abstraction of the virtual imaging circuit 30 is an API introduced from the viewpoint of software.

  Since the virtual imaging circuit 30 performs the intended operation only by calling the API function as described above, the still image shooting program can use the function without knowing the complicated movement of the front engine 12. .

  Although it has been explained so far that a still image shooting program can be easily created by the abstraction of the virtual imaging circuit 30, when this abstraction is performed, it will be easy to see how live view and video shooting operations will be. I will go.

  The live view operates in two image shooting modes, a still image shooting mode and a moving image shooting mode. In the case of image shooting, the camera is activated in the shooting mode set at the time of shooting, and the program for performing live view is first operated to start live view. The image is taken when the release button 18 is fully pressed, and the live view operation continues until then. During the live view, a raw image (through image) of the subject is displayed on the LCD panel 17, and the user determines the composition at the time of shooting and knows the shutter chance by watching this.

  Further, 3A detection is appropriately performed during the live view, and the brightness and color of the through image are adjusted using the AE evaluation value and the AWB evaluation value extracted there. In the still image shooting mode, photometric calculation for AE shooting is also performed. When the release button 18 is fully pressed, each shooting operation starts depending on the shooting mode. However, since the half-press is performed before the full-press, the focus adjustment (AF) on the subject is performed at that time.

  This focus adjustment (AF) is performed using the AF evaluation value. That is, several programs for performing live image data output, live view display, brightness adjustment, WB adjustment (including AWB algorithm), focus adjustment (AF), and the like are called from the live view program.

  These programs are also called from the API (some are called when triggered by an interrupt). Among them, the virtual imaging circuit 30 is related to the output of through image data. The live view program calls an API for outputting live view image data, but this may be a dedicated API.

  The virtual imaging driver 104d and the image processing driver 104b are further called from this API (left side in FIG. 9). Hereinafter, the interface of the virtual imaging circuit 30 corresponding to the first image data transmission path of the front engine 12 is referred to as “first interface”, and the interface corresponding to the second image data transmission path is referred to as “second interface”. Will be referred to as the "interface". The virtual imaging driver 104d sets the virtual imaging circuit 30 so that image data for a through image is output from the first interface.

  On the other hand, the image processing driver 104b sets the image processing circuit so that the image data for the through image is processed by the first post-processing circuit 13j in FIG. In this case, the image processing driver 104b first sets the image processing circuit. As a result, when the image data for the through image is output from the first interface of the virtual imaging circuit 30 (solid line arrow from the first output imaging interface 12c to the second input imaging interface 13a: see FIG. 5). The YCbCr data of the through image created by processing it in the first post-processing circuit 13j is stored in the YCbCr1 buffer 15f of FIG.

  The through image YCbCr data is sent to the LCD controller 13d through the same route as in FIG. There are a plurality of YCbCr1 buffers 15f (in many cases, there are three cases), and while the YCbCr data of the through image output previously is displayed on the LCD panel 17, the next YCbCr data is stored in another YCbCr1 buffer 15f. The display of the through image is continued while switching the YCbCr1 buffer 15f.

  Among the programs described above, the program related to the live view display must switch the YCbCr1 buffer 15f as appropriate. The image data for the through image output from the virtual imaging circuit 30 is also sent to the 3A detection circuit 13h in FIG. 5, and the 3A evaluation value is extracted. Using the AE evaluation value and the AWB evaluation value among them, the brightness adjustment and the WB adjustment of the through image are appropriately performed. The above-described program performs these processes each time it is called.

  Also, photometry calculation for still image shooting is performed from this AE evaluation value. When half-press of the release button 18 is detected, a focus adjustment (AF) program is called to focus on the subject based on the AF evaluation value. If half-pressed but not fully pressed, live view continues after focus adjustment (AF) ends.

  When the half-press is performed and then the full-press is performed, the process proceeds to image shooting. First, the case of the simple moving image shooting mode will be described. In the case of moving image shooting, the moving image data for recording is output from the virtual imaging circuit 30 shown in FIG. 8 while continuing the live view.

  That is, while the image data for the through image is output from the first interface of the virtual imaging circuit 30, the second interface (dotted arrow heading from the second output imaging interface 12k to the second input imaging interface 13k: FIG. 5) to output recording moving image data. This control is easy because only one piece of image data serving as a base is divided inside the virtual imaging circuit 30 and output from two interfaces. After being fully pressed, the moving image shooting program is first called while the live view program is being executed, and an API for outputting moving image data for recording is called in the moving image shooting program.

  The virtual imaging driver 104d and the image processing driver 104b are further called from this API. The virtual imaging driver 104d is set to output moving image data for recording from the second interface of the virtual imaging circuit 30.

  On the other hand, the image processing driver 104b is set so that the data is processed by the second pre-process circuit 13m and the second post-process circuit 13p in FIG. Also in this case, the image processing circuit is set first. When the image data for the through image is output from the second interface of the virtual image pickup circuit 30, it is processed by the second pre-process circuit 13m and the second post-process circuit 13p to create the YCbCr for recording. Data is stored in the YCbCr2 buffer 15g of FIG.

  The YCbCr data for recording is sent to a moving image codec (not shown) (H.264, etc.) and compressed, but the subsequent processing is not related to the virtual imaging circuit 30 and will not be described. In the case of moving image shooting, processing is performed while simultaneously outputting two different image data from the virtual imaging circuit 30. Among image data, moving image data for recording generally has a higher resolution. Note that it is necessary to keep focusing on the subject during moving image shooting, and the focus adjustment (AF) program described above continues the focus adjustment (AF) process based on the AF evaluation value.

  Next, a case where live view and still image shooting operations are performed in parallel will be described. There are two possible cases. In normal still image shooting, when the focus adjustment (AF) is completed, the live view is stopped and the still image shooting operation is started. However, the first operation is to perform still image shooting while continuing the live view. .

  In the first case, a certain frame in the live view is taken out and recorded as a still image. Since the high-speed imaging circuit 11a can also output full-resolution image data at the frame rate of the live view, it can shoot a still image with a high resolution while continuing the live view.

  The operation of the virtual imaging circuit 30 in this case is similar to that for moving image shooting. However, the point that the captured still image data is intermittently output from the virtual imaging circuit 30 is different from the case of moving image shooting. When the release button 18 is fully pressed during live view, the frame immediately after the end of the focus adjustment (AF) is taken out as a still image. The still image data is a virtual imaging circuit as in the case of normal still image shooting. 30 is stored inside. Since this processing corresponds to the exposure operation of the virtual imaging circuit 30, it can be performed using the API function already described. The captured still image data is subsequently output from the virtual imaging circuit, and the output operation of this still image data can also be performed using the API function already described. It can be seen that the operation is the same as that of normal still image shooting except that the live view continues.

  Subsequently, the second case will be described, which has already been described. That is, the live view is resumed before the output of the still image data stored in the virtual imaging circuit 30 is finished when the RAW buffer continuous shooting described above is performed and interrupted or terminated. Live view is not performed while RAW buffer continuous shooting is performed. The photographed still image data is output from the second interface of the virtual imaging circuit, but since the first interface is free, live view image data can be output from there in parallel.

  The live view operation in this case is the same as described above. When the release button 18 is fully pressed during the resumed live view, the operation of still image shooting is started again. This is the same operation as normal still image shooting. If State 1 of the virtual imaging circuit 30 is READY, a still image can be taken, and the already-described API is called to expose the still image. The subsequent steps are the same as those described so far, and will be omitted. In other words, live view or moving image shooting can be processed using the API of the virtual imaging circuit 30.

  As described above, in the fourth embodiment, software can be easily shared and reused, and a highly versatile camera system can be constructed.

1, 2 ... Electronic camera, 11 ... Imaging circuit, 12 ... Front engine, 13 ... Back engine

Claims (9)

An imaging unit that captures a subject image and outputs image data;
A storage unit for storing image data output from the imaging unit;
A first processing unit for reducing the pixel rate Read out the image data stored in the storage unit,
A second processing unit for processing an image data pixel rate is reduced by the first processing unit,
With electronic camera.   The electronic camera according to claim 1,
  The first processing unit is an electronic camera that reads out image data stored in the storage unit at a lower speed than that at the time of storage and reduces a pixel rate. The electronic camera according to claim 1 or 2 ,
A compression unit for compressing the image data output from the imaging unit;
The storage unit stores the image data compressed by the compression unit,
Wherein the first processing unit, an electronic camera for image processing reads the compressed image data stored in the storage unit even slower than the time stored. The electronic camera according to claim 3 .
An image decompression unit for decompressing the image data compressed by the compression unit;
Wherein the first processing unit, an electronic camera that performs image processing on the image data expanded by the image expansion section. The electronic camera according to any one of claims 1 to 4 ,
Wherein the first processing unit, an electronic camera image data output from the imaging unit a smaller amount of data than the data amount per unit are written in the storage unit time, read out per unit time. The electronic camera according to any one of claims 1 to 5,
The first processing unit is an electronic camera that reduces a pixel rate of image data output from the imaging unit based on a data amount of image data output from the imaging unit. The electronic camera according to any one of claims 1 to 6 ,
Said first processing portion includes a first pixel rate portion to reduce the image data output from the imaging unit to the first image data of the first resolution in the first pixel rate, the output from the imaging unit A second pixel rate unit that reduces the image data obtained at a second pixel rate to a second image data having a second resolution different from the first resolution,
The second processing unit, the first and the processing unit, an electronic camera and a second processing unit for the second image processing the image data of the first image processing image data. The electronic camera according to claim 7 ,
The second pixel rate unit is an electronic camera that performs image processing with a higher resolution than the first pixel rate unit. The electronic camera according to any one of claims 1 to 8 ,
Wherein the first processing unit, an electronic camera which performs correction processing on the image data.

Images