JP2004032713A - Imaging apparatus, imaging method, program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To rapidly complete processing using pixel data from a imaging element at photographing. <P>SOLUTION: The imaging apparatus comprises the imaging element 14, where a line in which first and second color filters are arranged to pixels and a line in which first and third color filters are arranged are arranged alternately; an image control section 18 for forming the frame of images by n fields (n: odd number) and reading pixel data in the picture element including all color components in one field interval; a white balance operation section 22 for performing white balance operation, based on image data read from the pixel by the image control section; and a control section 50 for executing the start of the white balance operation by a white balance operation section, before the reading operation of corresponding frame from the pixel is completed by the image control section. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an imaging device, and more particularly, to an imaging device that performs white balance calculation after shooting.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, digital cameras that record and reproduce a still image or a moving image captured by a solid-state imaging device such as a CCD using a memory card having a solid-state memory device as a recording medium are commercially available. In these digital cameras, a photographing process for temporarily capturing output data from the image sensor into a buffer memory was performed, and an image process and a compression process were performed on the captured data, and a development compression process for storing the captured data in the buffer memory again was performed. Thereafter, a process of writing to a removable recording medium is generally performed.
[0003]
Here, various algorithms have been proposed for white balance (WB) control for a recorded image. However, the order of execution is such that after photographing, the signal processing is performed only by outputting from the image pickup device and taking in the memory. Unprocessed raw data (hereinafter referred to as RAW data) is read out, and processing using pixel data like RAW data, for example, a coefficient for each color determined by a gamma correction processing or a color filter of an image sensor is determined by a predetermined algorithm. (Hereinafter, this processing is referred to as WB calculation), white data is corrected by multiplying by a WB coefficient obtained as a result of the calculation, and then development compression processing is performed.
[0004]
By the way, the image pickup device includes a progressive scan type in which all pixels are read out line-sequentially and an interlaced readout type in which two fields are read out every other line. In recent years, the latter two fields have been further increased. As a result, a type of reading out in three fields or more fields has appeared. These multi-field readout type imaging elements have the advantage that the light receiving section can be used effectively and the number of pixels can be increased while keeping the sensor size small.
[0005]
In the conventional multiple-field reading method, the WB calculation is started after waiting for one screen of RAW data to be written to the memory (for example, see Patent Document 1).
[0006]
[Patent Document 1]
JP-A-08-265782 (page 3, paragraph (0021))
[0007]
[Problems to be solved by the invention]
However, in this method, the reading from the image sensor and the WB operation can always be performed only sequentially, so there is no time lag from when the shutter button is pressed until the start of the exposure. It took some time to get out.
[0008]
In continuous shooting, even if the WB coefficient may be the same as that of the first image, the WB calculation is performed for the first image after RAW data for one screen is stored in the buffer memory after shooting. Therefore, there has been a problem that the photographing interval of the first and second images becomes longer than the photographing interval of the second and subsequent images.
[0009]
Therefore, the present invention has been made in view of the above-described problem, and an object of the present invention is to enable a process using pixel data, such as white balance calculation, to be completed quickly at the time of photographing.
[0010]
[Means for Solving the Problems]
In order to solve the above-described problem and achieve the object, an image pickup apparatus according to the present invention is configured such that a line on which first and second color filters are arranged and a first and third color filter are arranged for a pixel. And an image sensor in which n lines (n is an odd number) form an image of one frame, and pixel data of the image sensor so as to include all color components every one field period. Imaging control means for reading out
A white balance calculating means for performing a white balance calculation based on the image data read from the image pickup device by the image pickup control means; and Control means for starting the white balance calculation by the white balance calculation means.
[0011]
Also, in the imaging apparatus according to the present invention, the imaging control means performs interlaced reading for forming an image for one frame by performing reading for every n lines in each field.
[0012]
Further, in the imaging apparatus according to the present invention, the white balance calculation means performs white balance calculation based on image data of a part of image data of a plurality of fields constituting one frame. .
[0013]
The imaging apparatus according to the present invention may further include a display unit for displaying a captured image, wherein the control unit reads the image data for one frame from the image sensor before the image data is completely read out. The display means starts displaying a captured image based on the image data of the field.
[0014]
The imaging apparatus according to the present invention may further include a reduced image forming unit that forms a reduced image having a smaller data amount than the image data for one frame, wherein the control unit controls the image data for one frame from the imaging device. Before the reading of the image is completed, the reduced image forming means forms a reduced image.
[0015]
Further, in the imaging apparatus according to the present invention, the control means executes a start of white balance calculation by the white balance calculation means before the readout operation for one field from the imaging element is completed by the imaging control means. It is characterized by.
[0016]
Further, the imaging method according to the present invention is directed to an imaging device in which a line on which first and second color filters are arranged and a line on which first and third color filters are arranged are alternately arranged for a pixel. An image pickup method using an image pickup device comprising: an image forming apparatus, wherein an image of one frame is formed in n fields (n is an odd number), and pixel data of the image pickup element is included so as to include all color components every one field period. An imaging control step for reading, and a white balance operation step of performing a white balance operation based on the image data read from the imaging element in the imaging control step, wherein one frame from the imaging element is obtained from the imaging element in the imaging control step. The white balance calculation is started before the read operation is completed.
[0017]
Further, in the imaging method according to the present invention, in the imaging control step, reading for every n lines is performed in each field, and interlaced reading for forming an image for one frame is performed.
[0018]
Further, in the imaging method according to the present invention, in the white balance calculation step, white balance calculation is performed based on image data of a part of image data of a plurality of fields constituting one frame. .
[0019]
The imaging method according to the present invention may further include a display step for displaying a captured image, wherein the display step is performed before reading of one frame of image data from the imaging element is completed in the imaging control step. Is characterized by starting display of an image.
[0020]
Further, the imaging method according to the present invention further includes a reduced image forming step of forming a reduced image having a smaller data amount than image data of one frame, and in the temporary storage step, an image of one frame from the image sensor is received from the image sensor. Before the data reading is completed, the reduced image forming step is performed.
[0021]
Further, in the imaging method according to the present invention, the white balance calculation in the white balance calculation step is started before the read operation for one field from the image sensor is completed in the imaging control step. .
[0022]
Further, a program according to the present invention causes a computer to execute the above-described imaging method.
[0023]
Further, a storage medium according to the present invention is characterized in that the above-mentioned program is stored in a computer-readable manner.
[0024]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0025]
(1st Embodiment)
FIG. 1 is a diagram showing a configuration of a digital camera system using an image sensor of a three-field readout system according to a first embodiment of the present invention.
[0026]
In FIG. 1, reference numeral 100 denotes a digital camera device. Reference numeral 10 denotes a photographing lens, and reference numeral 11 denotes an optical finder for confirming a subject image, and the angle of view can be changed according to zoom control. Reference numeral 12 denotes an aperture / shutter having an aperture function. Reference numeral 14 denotes an image pickup device that converts an optical image into an electric signal, and uses a CCD (Charge Coupled Device). Also, a primary color mosaic filter is arranged on the element. Reference numeral 15 denotes a pre-processing circuit including a CDS (correlated double sampling) circuit and an AGC (automatic gain control) circuit for removing output noise of the image sensor 14. Reference numeral 16 denotes an A / D converter for converting an analog signal output from the preprocessing circuit 15 into a digital signal.
[0027]
Reference numeral 18 denotes a timing generation circuit (TG) for supplying a clock signal and a control signal to the image sensor 14 and the A / D converter 16, and their timing is controlled by the signal processing IC 22.
[0028]
The signal processing IC 22 performs predetermined pixel interpolation processing, color conversion processing, enlargement / reduction, and image data format conversion on the data from the A / D converter 16 or the data from the DRAM 30 according to an instruction from the CPU 50. It also has a DMA controller, a D / A converter, and a compression / decompression circuit for compressing / decompressing image data. Further, the signal processing IC 22 performs a predetermined calculation process using the captured image data, and stores the obtained calculation result in the DRAM 30. The CPU 50 performs TTL AWB (auto white balance) processing, AF (auto focus) processing, AE (auto exposure) processing, and EF (strobe pre-flash) processing based on the calculation results.
[0029]
Reference numeral 28 denotes an image display unit composed of a TFT LCD or the like. Display image data written in the DRAM 30 is displayed by the image display unit 28 via a D / A converter (not shown) inside the signal processing IC 22. The image display unit 28 is used for displaying not only images but also various information / mode setting statuses.
[0030]
The electronic viewfinder function can be realized by sequentially displaying captured image data using the image display unit 28.
[0031]
In addition, the image display unit 28 can arbitrarily turn on / off the display in accordance with an instruction from the CPU 50. When the display is turned off, the power consumption of the digital camera device 100 can be significantly reduced.
[0032]
Furthermore, the image display unit 28 is connected to the camera body by a rotatable hinge unit, and can set an arbitrary direction and angle to use the electronic finder function, the reproduction display function, and various display functions. .
[0033]
In addition, the display portion can be stored while protecting the display portion by turning the display portion of the image display portion 28 toward the camera body. In this case, the storage state is detected by the detection SW 98 and the display of the image display portion 28 is performed. Operation can be stopped.
[0034]
The DRAM 30 is a memory used for temporarily storing photographed uncompressed data, holding AF / AE / AWB / EF calculation results, holding a display image on the image display unit 28, holding compressed image data, and the like. The storage capacity is sufficient to store the number of still images and the moving images for a predetermined time.
[0035]
Reference numeral 46 denotes a connector, which is also called an accessory shoe, and includes an electrical contact with the external strobe device 400 and mechanical fixing means.
[0036]
Reference numeral 48 denotes a built-in flash having a TTL light control function.
[0037]
A CPU 50 controls the entire digital camera device 100. In addition to performing AF / AE / AWB / EF control based on the AF / AE / AWB / EF calculation results stored in the DRAM 30 by the signal processing IC 22, data flow control for the signal processing IC 22, various key scan operations, zoom control, It communicates with peripheral modules.
[0038]
Reference numeral 52 denotes a memory for storing variables for operation of the CPU 50 and the like.
[0039]
Reference numeral 54 denotes a display unit such as a liquid crystal display device, a speaker, or the like that displays an operation state, a message, or the like using characters, images, sounds, or the like in accordance with the execution of the program by the CPU 50. One or a plurality of them are installed at easily visible positions, and are configured by a combination of, for example, LCDs, LEDs, sound-generating elements, and the like.
[0040]
The display contents of the display unit 54 include a single shot / continuous shooting display, a self-timer display, a compression ratio display, a recording pixel number display, a recording number display, a remaining image number display, a shutter speed display, an aperture value display, and an exposure correction. Display, strobe display, red-eye reduction display, macro shooting display, buzzer setting display, clock battery remaining amount display, battery remaining amount display, error display, multi-digit information display, recording medium 200 attachment / detachment display, communication I / F operation display, date / time display, and the like.
[0041]
Reference numeral 56 denotes an electrically erasable / recordable flash memory in which a program necessary for operating the CPU 50, camera-specific adjustment data, and the like are written in advance.
[0042]
Hereinafter, the main operation means will be specifically described.
[0043]
Reference numeral 60 denotes a mode dial switch having a two-stage structure, and the lower stage has three states of power-off (OFF), shooting mode, and reproduction mode. At the time of shooting, various shooting modes set by the user according to the purpose and scene at the time of shooting are assigned. Fully automatic shooting mode (AUTO), program shooting mode, shutter speed priority shooting mode, aperture priority shooting mode, manual shooting. Mode, pan focus mode, portrait mode, landscape mode, night view mode, color effect mode, stitch assist mode, and moving image shooting mode.
[0044]
Reference numeral 64 denotes a shutter switch, which is composed of two-stage switches SW1 and SW2. When the shutter button is half-pressed, SW1-ON is performed, and an operation such as an AF (auto focus) process, an AE (auto exposure) process, and an AWB (auto white balance) process is instructed. Further, when the shutter button is fully pressed, SW2-ON is performed, and the signal read from the image sensor 12 is written as image data to the DRAM 30 via the A / D converter 16 and the signal processing IC 30. Next, the signal processing IC 22 reads image data from the DRAM 30 according to an instruction from the CPU 50, performs image processing such as color correction, pixel interpolation, and color conversion, performs compression processing, and writes the data to the recording medium 200.
[0045]
Reference numeral 68 denotes a flash switching switch for switching the flash mode of the flash at the time of photographing to forced flash, non-flash, or automatic flash.
[0046]
Reference numerals 72, 74, and 76 denote a menu key, a set key, and a cross key, respectively, and a combination of these keys allows a user to change various settings or execute a function during shooting or playback while viewing the image display unit 28. Can be. Among them, the cross key is a composite key composed of four direction keys of up, down, left and right.
[0047]
By pressing the menu key 72 once, a menu screen is displayed on the image display unit 28. This menu screen differs depending on the current mode indicated by the mode dial switch 60. Thereafter, the displayed menu item is selected using the cross key 76, and is determined by pressing the set key 74.
[0048]
For example, the menu items when the shooting mode is the program shooting mode include, for example, a recording pixel size, a compression ratio, a recording format, a sensitivity, an AF mode, whether or not shooting is confirmed, and an image quality tuning parameter.
[0049]
Menu items at the time of reproduction include erasing an image (one or all), protecting (protecting) an image, and setting image rotation.
[0050]
The cross key 76 is also used for other than menu setting. For example, when the mode is set to the shutter speed priority mode by the mode dial switch 60, the shutter speed can be changed using the left and right keys of the cross key. Further, in the reproduction mode, left and right direction keys are used to perform image feed.
[0051]
Reference numeral 78 denotes a drive mode switching button for cyclically switching between single shooting, continuous shooting, and a self-timer each time the button is pressed. Reference numeral 70 denotes a group of operation units other than those described above. There are a selection / switch button for setting selection and switching of various functions when executing shooting and playback in a panorama mode or the like, an image display ON / OFF switch for setting ON / OFF of the image display unit 28, and the like. Items that are relatively frequently used are assigned to this operation unit.
[0052]
Reference numeral 80 denotes a power control unit, which includes a battery detection circuit, a DC-DC converter, a switch circuit for switching a block to be energized, and the like. And controls the DC-DC converter based on the instruction of the CPU 50 to supply a necessary voltage to each unit including the recording medium for a necessary period, and display the remaining battery level on the display unit 54 and the image display unit 28 as necessary. Do.
[0053]
Reference numerals 82 and 84 denote connectors, and reference numeral 86 denotes a power supply unit including a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, and an AC adapter.
[0054]
Reference numeral 92 denotes a connector for connecting to a recording medium such as a memory card or a hard disk.
[0055]
In this embodiment, the interface and the connector for attaching the recording medium are described as having one system. Of course, the interface and the connector for attaching the recording medium may have a single or plural number of systems. Further, a configuration may be adopted in which interfaces and connectors of different standards are provided in combination.
[0056]
The interface and the connector may be configured using a standard such as a PCMCIA card or a CF (Compact Flash (R)) card.
[0057]
Reference numeral 98 denotes a detection switch, which includes an image display unit opening / closing detecting means for detecting whether the display portion of the image display unit 28 is stored toward the digital camera device 100 and a battery cover. Battery cover opening / closing detecting means for detecting that the electronic flash has been opened, and a mounting state detecting means for the external strobe 404.
[0058]
Reference numeral 200 denotes a recording medium such as a memory card or a hard disk.
[0059]
The recording medium 200 includes a recording unit 202 including a semiconductor memory or a magnetic disk, an interface 204 with the digital camera device 100, and a connector 206 for connecting to the digital camera device 100.
[0060]
Reference numeral 400 denotes an external strobe device. In the present embodiment, an external strobe can be mounted later in addition to the built-in strobe 48, and TTL flash photography can be performed similarly to the built-in strobe. Reference numeral 402 denotes a connector for connecting to an accessory shoe of the digital camera device 100. Reference numeral 404 denotes an external strobe, which also has an AF auxiliary light projecting function and a strobe light control function.
[0061]
2 and 3 show a flowchart of a main routine of the digital camera device 100 according to the present embodiment.
[0062]
The operation of the digital camera device 100 will be described with reference to FIGS.
[0063]
Upon power-on such as battery replacement, the CPU 50 initializes flags, control variables, and the like (S201), and initializes the image display on the image display unit 28 to the OFF state (S202).
[0064]
The CPU 50 determines the setting position of the mode dial 60, and if the power of the mode dial 60 is set to OFF (S203), changes the display of each display unit to the end state, closes the aperture / shutter 12 and captures an image. Protecting the unit, recording necessary parameters, setting values including flags and control variables, and setting modes in the memory 52, and turning off unnecessary power of each unit of the digital camera device 100 including the image display unit 28 by the power control unit 80. After performing a predetermined end process such as performing (S205), the process returns to S203.
[0065]
If the mode dial 60 has been set to the shooting mode (S203), the process proceeds to S206. If the mode dial 60 has been set to another mode (S203), the CPU 50 executes a process according to the selected mode (S204), and returns to S203 when the process is completed.
[0066]
The CPU 50 uses the power control unit 80 to determine whether or not the remaining capacity and operation status of the power supply 86 composed of a battery or the like has a problem with the operation of the digital camera device 100 (S206). After a predetermined warning is displayed by image or sound using the unit 28 or the display unit 54 (S208), the process returns to S203.
[0067]
If there is no problem in the power supply 86 (S206), the CPU 50 determines whether or not the operation state of the recording medium 200 or 210 has a problem in the operation of the digital camera device 100, particularly in the operation of recording and reproducing image data on the recording medium. (S207) If there is a problem, a predetermined warning is displayed by image or sound using the image display unit 28 or the display unit 54 (S208), and the process returns to S203.
[0068]
If there is no problem in the operation state of the recording medium 200 or 210 (S207), the process proceeds to S208. Thereafter, as a function of a finder, initialization of photographing for displaying a through image on the image display unit 28 is initialized (S208), and when the preparation is completed, display of the through image on the image display unit 28 is started (S209). The through image is an image for confirming a subject image captured by the image sensor 14.
[0069]
Next, the camera operation at the time of operation in the live view image display state will be described with reference to FIG.
[0070]
If the mode dial 60 has been changed (S301), the process jumps to S203 to check the dial state. If it has not been changed, it is next checked whether there is any change in the operation unit and the detection SW (S304), and if there is a change, a process corresponding to the changed portion is performed (S305).
[0071]
If the shutter switch SW1 has not been pressed (S306), a process for continuously displaying a through image is performed. Further, the signal processing IC 22 performs a predetermined photometric calculation on the signal obtained from the image sensor 14 and stores the calculation result in the DRAM 30. The CPU 50 performs AE / AWB processing on the through image based on the calculation result (S307). Then, the process returns to S301.
[0072]
If the shutter switch SW1 is pressed in S306, the CPU 50 performs a distance measurement operation based on the focus information and the photometric information calculated by the signal processing IC 22 and stored in the DRAM 30, and focuses the photographing lens 10 on the subject. Then, the aperture / shutter 12 is controlled to an aperture value obtained in accordance with a program diagram for each shooting mode by performing photometric calculation (S308). Next, the state of SW2 is checked (S309). If SW2 has not been pressed yet, the state of the shutter SW1 is checked again (S310), and if the shutter switch SW1 is released, the flow returns to S307. If the switch SW1 has been pressed, the state of the switch SW2 is monitored again (S309).
[0073]
If the switch SW2 has been pressed in step S309, the settings such as the shutter speed, imaging output gain, and strobe light emission at the time of photographing are finally determined with reference to the current photometric result and the program diagram for each photographing mode. (S311). Further, the display of the through image displayed on the image display unit 28 is stopped (S312).
[0074]
Next, the process proceeds to a shooting sequence. First, a single photographing process for performing a series of photographing operations is performed (S313). In the post-imaging processing DRAM 30, data before signal processing read from the imaging device 14 through the A / D converter 16 (hereinafter, this data is referred to as RAW data) is stored. The signal processing IC 22 reads the RAW data from the DRAM 30 in accordance with an instruction from the CPU 50, and performs color correction based on the color correction information obtained by the photographing process (S314). Next, development / compression processing for converting the image data into image data to be finally recorded is performed, and the processed image data is stored in the DRAM 30 (S315). Next, the processed image data is converted into a display image and stored in the DRAM 30. The signal processing IC 22 reads out the display image data at a predetermined rate, performs D / A conversion internally, and outputs the data to the image display unit 28, thereby displaying a confirmation image (S316). In S315, the compressed image data stored in the DRAM 30 is written to the recording medium 200 (S317), and the process returns to S307.
[0075]
FIG. 4 shows the color filter arrangement and the electrode arrangement of the pixels of the image sensor 14 in this embodiment for two horizontal pixels and 12 vertical pixels. This is repeated in the horizontal and vertical directions. Reference numeral 21 denotes a pixel to which a color filter is attached. Horizontal pixel lines repeating red (R) and green (G) and horizontal pixel lines repeating green (G) and blue (B) are alternately arranged in the vertical direction. ing. Reference numeral 22 denotes a transfer electrode of the vertical CCD, and one pixel corresponds to every two transfer electrodes of the vertical CCD.
[0076]
The transfer electrodes V1, V2A and V2B, V3, V4A and V4B, V5 and V6, which are represented by the same numerals, have six-phase drive pulses φV1, φV2A and φV2B, φV3, φV4A and φV4B, φV5 and φV6, respectively. , The data can be transferred in six phases. Among the six-phase driving pulses, the same pulse φV2A and φV2B, φV4A and φV4B relating to the charge transfer of the vertical CCD are used.
[0077]
Further, the read pulse of the first field is applied to the transfer electrodes V2A and V2B also serving as the read electrodes, and charges the first horizontal pixel line, the fourth horizontal pixel line, the seventh horizontal pixel line, and the tenth horizontal pixel line. It can be read out to a vertical CCD.
[0078]
The read pulse of the second field is applied to the transfer electrodes V4A and V4B also serving as read electrodes, and the charges of the second horizontal pixel line, the fifth horizontal pixel line, the eighth horizontal pixel line, and the eleventh horizontal pixel line are transferred to the vertical CCD. Can be read out.
[0079]
The readout pulse of the third field is applied to the transfer electrode V6 also serving as a readout electrode, and reads out charges of the third horizontal pixel line, the sixth horizontal pixel line, the ninth horizontal pixel line, and the twelfth horizontal pixel line to the vertical CCD. It is configured to be able to.
[0080]
Further, the transfer electrode V2A reads out the charges of the first horizontal pixel line and the tenth horizontal pixel line to the vertical CCD, and the transfer electrode V2B reads out the charges of the fourth horizontal pixel line and the seventh horizontal pixel line to the vertical CCD. The transfer electrode V4A can read the charges of the fifth horizontal pixel line and the eighth horizontal pixel line to the vertical CCD, and the transfer electrode V4B can read the charges of the second horizontal pixel line and the eleventh horizontal pixel line to the vertical CCD. It is configured as follows.
[0081]
With reference to the timing chart of FIG. 5 and the pixel arrangement of FIG. 6, CCD reading at the time of shooting will be described. The imaging mode is a mode in which charges of all pixels are read out in three fields, and the number of vertical pixels is assumed to be 1440 pixels. VD is a vertical synchronizing signal, and a mechanical shutter controls exposure by opening and closing, and an electronic shutter controls exposure by applying a pulse to the substrate potential of the solid-state imaging device and extracting charges of pixels in the direction of the substrate. The exposure time is from time t11 when the electronic shutter pulse ends to time t12 when the mechanical shutter closes.
[0082]
The drive pulses φV2AR, φV2BR, φV4AR, φV4BR, and φV6R indicate only read pulses applied to φV2A, φV2B, φV4A, φV4B, and φV6, respectively. The CCD output indicates the output of the image sensor 14. At time t13, since the read pulse is applied to φV2AR and φV2BR, the first horizontal pixel line (RG line), the fourth horizontal pixel line (GB line), and the seventh horizontal pixel line (RG line) Then, the electric charges of the tenth horizontal pixel line (GB line) are read out to the vertical CCD. This is illustrated in FIG. 6A, and the output of this charge is the first field of the CCD output. As the signal, 1/3 of the number of vertical pixels is output. Next, at time t14, since a read pulse is applied to φV4AR and φV4BR, the second horizontal pixel line (GB line), the fifth horizontal pixel line (RG line), and the eighth horizontal pixel line (G- The charges on the B line) and the eleventh horizontal pixel line (RG line) are read out to the vertical CCD. This is illustrated in FIG. 6B, and the output of this charge is the second field of the CCD output. As the signal, 1/3 of the number of vertical pixels is output. Further, since a read pulse is applied to φV6R at time t15, the third horizontal pixel line (RG line), the sixth horizontal pixel line (GB line), the ninth horizontal pixel line (RG line) and The charge of the twelfth horizontal pixel line (GB line) is read out to the vertical CCD. This is illustrated in FIG. 6 (c), and the output of this charge is the third field of the CCD output. As the signal, 1/3 of the number of vertical pixels is output. The output signals of the three fields are temporarily stored in the DRAM 30 in the same manner as the pixel array, then subjected to image processing by the signal processing circuit, and stored again in the DRAM 30.
[0083]
In a conventional interlaced read-out type primary color CCD, pixels obtained in one field are either R and G or G and B, and all information cannot be obtained only in one field, and white balance calculation such as luminance information is performed. Information needed to do this was lacking. On the other hand, in the three-field readout method, as described above, the pixels read out in each field include both the horizontal pixel line composed of R and G and the horizontal pixel line composed of G and B. Even if the reading is not completed, the information of R, G, and B is prepared before the reading of one field is completed, so that the white balance can be corrected for the captured image.
[0084]
FIG. 7 is a flowchart showing details of the single-shot shooting process, which performs a series of operations such as exposure, shutter closing, sensor reading, and white balance calculation.
[0085]
First, the CPU 50 performs a DRAM write setting for temporarily storing RAW data in the DRAM 30 (S701). At this time, the setting for writing to the DRAM 30 is not performed continuously on the memory, but is performed on the DMA controller in the signal processing IC 22 so that the data is written every two lines in accordance with a three-field reading method. Next, an electronic shutter stop command is issued to the timing generation circuit 18 to start exposure, and the shutter closing timing of the aperture / shutter 12 is set in a CPU built-in timer according to the determined shutter time (S702). The CPU built-in timer is designed to output a pulse for fully closing the aperture / mechanical shutter 12 when a specified time has elapsed. Next, the control waits for the mechanical shutter to close (S703). When the mechanical shutter closes, a write start command for the first field is first issued in the DRAM area set in S701 in accordance with the write setting performed in S701, and the image sensor 14 outputs the first command. An instruction is issued to the timing generator 18 to read out the imaging data of one field (S704). Next, the process waits for the X-th line of the first field (the X-th line indicates a second or more line in which all the color information of R, G, and B are aligned) (S705). The signal processing IC 22 sequentially reads out the image data written in the DRAM 30 up to the middle of the first field and starts a WB (white balance) operation (S706). At this time, control is performed so that the color correction information obtained as a result of the WB operation is stored in another area in the same DRAM 30. Next, it waits for the completion of the RAW data write operation of the first field to the DRAM 30 (S707). Upon completion, a write start command for the second field is issued in accordance with the write setting made in S701, and an instruction is issued to the timing generator 18 to read out the image data of the second field from the image sensor 14 (S708). ).
[0086]
Next, after the completion of the RAW data write operation of the second field to the DRAM 30 (S709), a write start command for the third field is issued, and the timing generator 18 is caused to read the third field of image data from the image sensor 14. Then, an instruction is issued (S710). Next, the process waits for completion of the third field RAW data write operation to the DRAM 30 (S711).
[0087]
Upon completion, post-photographing processing for opening the mechanical shutter, restarting the electronic shutter operation, and the like is performed so that the next photographing can be performed promptly (S712). Further, the process waits for the completion of the WB operation started in S706 (S713), and when completed, ends the photographing process.
[0088]
As for the method of determining the X-th line in S705, it is necessary to guarantee that the completion of the DRAM writing to the first field ends before the completion of the DRAM reading in the WB operation.
[0089]
FIG. 8 is a timing chart showing pixels read to the DRAM, the start of WB calculation, and the creation of various images when X = 2.
[0090]
When the readout pulses φV2AR and φV2BR are output from the timing generator 18 at the timing of time t801 in the figure, the pixels shown in FIG. The data is output to the DRAM 30 through the image sensor 14, the pre-processing circuit 15, the A / D converter 16, and the signal processing IC 22. Time t802 is the time when the RAW data up to the second line of the first field is read. At this timing, the reading of the memory for the WB operation is started, and the reading from the memory is completed at time t804. Then, the WB calculation is completed at time t806.
[0091]
On the other hand, reading of the first field from the image sensor 14 is completed at time t803, and one third of all pixels are completely stored in the DRAM 30. When the second field readout pulses φV4AR and φV4BR are output from the timing generator 18 at time t805, the pixels shown in FIG. The data is output to the DRAM 30 through the data processing circuit 15, the A / D converter 16, and the signal processing IC 22, and at time t807, the reading of the RAW data in the second field is completed. Similarly, when the third field readout pulse φV6R is output from the timing generator 18 at time t808, the pixels shown in FIG. The data is output to the DRAM 30 through the preprocessing circuit 15, the A / D converter 16, and the signal processing IC 22, and the reading of the RAW data of the third field is completed at time t809.
[0092]
In the three-field readout type imaging device of the present embodiment,
(Driving rate of image sensor)> (DRAM read rate at WB operation)
Is satisfied, the transfer operation of the first field imaging data to the DRAM 30 is not overtaken by the DRAM read operation of the WB operation. Therefore, X should be set to about several lines with a margin. become.
[0093]
In the case where the relationship of the above equation does not hold, in FIG. 8, when Tr is the time required for memory reading at the time of WB calculation,
t803-t802 <Tr
It is necessary to determine the WB calculation start time t802 in a range where the condition is satisfied. If the above equation does not hold, the DRAM reading for the WB operation is completed before the RAW data for one field read from the CCD is completely stored in the DRAM 30, and the indefinite data before the storage is stored. This is because it will be read. Of course, as is clear from FIG. 8, the closer the time t802 is to the time t801, the sooner the WB calculation can be completed.
[0094]
As described above, in the present embodiment, the WB calculation operation is started from the middle of the first field, and the WB calculation is already completed within the image signal readout period. There is almost no need to provide a special time required for the WB operation after reading, which contributes to a reduction in the total post-imaging image processing time.
[0095]
In the above-described embodiment, an example of single-shot shooting has been described, but the present invention is of course effective in continuous shooting. Usually, during continuous shooting, the WB calculation is performed only for the first image, and the WB coefficients for the second and subsequent images are often processed using the same WB coefficients as those for the first image. In this case, conventionally, the continuous shooting interval between the first and second frames is longer than the continuous shooting interval after the second frame, making it difficult to take a timing. However, according to the present embodiment, imaging of the first field is performed. By performing the WB calculation in parallel with the signal readout, the WB calculation is completed when the readout of all fields is completed. Therefore, according to the present embodiment, continuous shooting can be performed at substantially equal intervals from the first image. In addition, in the case where the WB calculation is performed not only for the first image but also for each of the second and subsequent images in the continuous shooting, the time for the WB calculation can be included in the readout time of the image pickup signal, so that the continuous shooting speed is reduced. It is possible to implement without.
[0096]
(Second embodiment)
In the above-described embodiment, the WB calculation is started using the imaging signal obtained from the first field, and the WB calculation ends in the middle of the second field. If the remaining image signal reading period is used, an image confirmation display image (rec review image) is created using the image data read in the first field, which is smaller than the image data amount for one frame, and a simplified reduced image (thumbnail) is used. Image) can be created. This can be realized by performing image conversion by the signal processing IC 22 based on the imaging signal of the first field and the result obtained by the WB operation, and storing the converted image in another area in the DRAM 30. These images, including the WB operation, are designed so that they can basically be operated by interrupt driving, and can be operated in parallel with the reading of the imaging signal.
[0097]
FIG. 9 is a timing chart showing the state of the pixels read out to the DRAM, the start of the WB calculation, and the creation of various images in this embodiment. When the read pulses φV2AR and φV2BR are output from the timing generator 18 at the timing of time t901 in the figure, the pixels shown in FIG. When the RAW data is output to the DRAM 30 through the image sensor 14, the preprocessing circuit 15, the A / D converter 16, and the signal processing IC 22, and the RAW data up to the second line of the first field is read at the time t902, Reading of the memory for the WB operation is started, and the reading from the memory is completed at time t904. Then, the WB calculation is completed at time t906.
[0098]
On the other hand, reading of the first field from the image sensor 14 is completed at time t903, and one third of all pixels are completely stored in the DRAM 30. When the second field readout pulses φV4AR and φV4BR are output from the timing generator 18 at a time t905, the pixels shown in FIG. The data is output to the DRAM 30 through the placement processing circuit 15, the A / D converter 16, and the signal processing IC 22, and at time t908, the reading of the RAW data of the second field is completed. Similarly, when the third field readout pulse φV6R is output from the timing generator 18 at the time t911, the pixels shown in FIG. The data is output to the DRAM 30 through the preprocessing circuit 15, the A / D converter 16, and the signal processing IC 22, and the reading of the RAW data in the third field is completed at time t912.
[0099]
In the present embodiment, utilizing the fact that the WB calculation is completed at the time t906, the remaining time, that is, while the image data is being transferred from the image sensor to the DRAM 30, the rec review display image and the thumbnail image are displayed. We are making. At time t906, an instruction to start creation of a rec review display image is issued from the CPU to the signal processing IC 22, and the signal processing IC 22 outputs the RAW data of the first field from the DRAM 30 and the result of the WB operation from the RAW data. The obtained color correction information is read, converted into data suitable for display, adjusted in image size, and stored in the DRAM 30 (complete at time t907). By repeatedly reading out the display image data and outputting it to the image display unit 28, a rec review display can be realized.
[0100]
Also, for the creation of a thumbnail image (reduced image) for displaying a list, the RAW data of the first field and the color correction information obtained as a result of the WB operation from the RAW data can be used, as in the rec review display. . At time t907, when an instruction to start creating a thumbnail image is issued from the CPU to the signal processing IC 22, the signal processing IC 22 reads the image of the first field from the DRAM 30 and converts it into data suitable for the thumbnail image. Then, the image size is adjusted and stored in the DRAM 30 (complete at time t910).
[0101]
At time t910, all the RAW data of one frame is stored in the DRAM 30. Therefore, the color correction information obtained as a result of the WB operation is read out from the RAW data and the RAW data of the first field, and the development for creating a recording image is performed. The compression processing is performed, and the compressed image data is stored in the DRAM 30.
[0102]
As described above, in the present embodiment, not only WB calculation but also creation of a rec review display image and a thumbnail image are performed while RAW data is read from the image sensor and transferred to the DRAM. There is an advantage that a high-speed processing can be achieved, and a rec review display after shooting can be quickly performed.
[0103]
(Third embodiment)
In the first and second embodiments, the WB calculation is performed only in the first field, and the color correction information obtained as a result of the calculation is also used for creating a recorded image. This means that the white balance was corrected using the image that was thinned out. However, when one frame reads out image data in more odd fields (for example, five fields) than three fields, accurate white balance correction may not be performed with only some fields. In the present embodiment, for the purpose of creating a rec review display image or a thumbnail image, color correction is performed using the WB operation result using the image data of the first field, and the main image (which has a larger data amount than a simple image such as a thumbnail image) It is characterized in that at the time of creation of an image for recording), WB calculation is performed using all the field image data to perform color correction.
[0104]
FIG. 10 is a timing chart showing the state of the pixels read to the DRAM, the start of the WB calculation, and the creation of various images in the present embodiment. The reading of the RAW data from the image sensor is the same as in the first and second embodiments, and therefore, a part thereof is omitted here.
[0105]
After generating the readout pulses φV2AR and φ2V2BR of the first field of the first field at t1001, from t1002, while reading the raw data of the first field stored in the DRAM up to halfway, the WB calculation for the rec review display and thumbnail image is performed. To start. Then, the WB calculation ends at t1006. Thereafter, in parallel with the second field readout, a rec review image is created from t1006 to t1007, and a thumbnail image is created from t1007 to t1008. Next, after generating a readout pulse φV6R of the third field at t1010, the WB calculation for the recorded image (main image) is performed at t1011. Here, the WB operation is performed by reading the RAW data of the frame instead of the RAW data of some fields. At t1012, all the RAW data of the third field from the image sensor is stored in the DRAM. Then, the WB calculation for the recorded image is completed at t1014, the creation of the recorded image is started using the color correction information obtained as a result, and the process ends at t1015.
[0106]
In this embodiment, a white balance calculation is performed after shooting using RAW data including a part of fields for creating a rec review display image and a thumbnail image, so that the time between shooting and the next shooting is reduced. Can be shortened. On the other hand, for a recording image, since white balance calculation is performed after shooting using the RAW data of the frame, there is an advantage that more accurate color correction can be performed.
[0107]
(Fourth embodiment)
In the first to third embodiments, the image sensor of the three-field readout method is used. However, the present invention can be applied to an image sensor having an even larger number of odd fields.
[0108]
Further, the WB operation is performed using the RAW data of the first field as the RAW data obtained from some fields, but the RAW data of another field or the added field may be used.
[0109]
As described above, according to the above-described first to fourth embodiments, the post-imaging white balance calculation is performed in parallel with the operation of reading out the image pickup signal from the image pickup device, and furthermore, one of the plurality of fields is performed. Since the white balance calculation is performed after photographing using only the field of the section, the time from photographing to the next photographing preparation state can be reduced.
[0110]
In addition, the REC review display image can be created in parallel with the image signal reading operation by using the WB operation result, so that it is possible to reduce the time from photographing until the REC review image is displayed. it can.
[0111]
Further, in continuous shooting, the WB coefficient of the conventional WB coefficient is used for both the case where the WB coefficient obtained from the WB calculation of the first image is used and the case where the WB operation is performed for each photographing for the second and subsequent images. Since the time required for the calculation can be eliminated, the photographing interval between the first and second images does not become longer than the photographing interval for the second and subsequent images. There is an effect that continuous shooting can be performed at a higher speed than in the related art.
[0112]
(Other embodiments)
The present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), but can be applied to a device including one device (for example, a copier, a facsimile machine, etc.). May be applied.
[0113]
Further, although the WB calculation has been described as the calculation process (image processing) for photographing performed using only some of the plurality of fields, other image processing may be performed. For example, instead of performing the WB calculation, a thumbnail image is created, a rec-review image is created, a through image is displayed on a viewfinder at the time of imaging, AE processing (luminance signal generation), AF processing, gamma correction processing, area extraction processing, and the like are performed. May be. At this time, color temperature information obtained by providing an external color temperature sensor or a WB coefficient from a viewfinder display immediately before shooting may be used. Thereby, the image processing time at the time of imaging is reduced.
[0114]
It is also possible to control the file size by reading out some fields, performing compression processing in advance, and reflecting the result at the time of compression after reading out one frame.
[0115]
Further, an object of the present invention is to supply a storage medium (or a recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and a computer (or a CPU or a CPU) of the system or the apparatus. Needless to say, the present invention can also be achieved by an MPU) reading and executing a program code stored in a storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. When the computer executes the readout program code, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a part or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.
[0116]
Further, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is executed based on the instruction of the program code. It goes without saying that the CPU included in the expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.
[0117]
When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.
[0118]
【The invention's effect】
As described above, according to the present invention, it is possible to quickly complete processing using pixel data from an image sensor at the time of shooting.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of a digital camera system according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating an operation of the digital camera system according to the embodiment of the present invention.
FIG. 3 is a flowchart illustrating an operation of the digital camera system according to the embodiment of the present invention.
FIG. 4 is a schematic diagram showing a color filter arrangement and an electrode arrangement of pixels of an image sensor in a digital camera system according to an embodiment of the present invention.
FIG. 5 is a timing chart at the time of reading image data from a three-field readout type imaging device according to an embodiment of the present invention.
FIG. 6 is a schematic diagram of image data readout from a three-field readout type imaging device according to an embodiment of the present invention.
FIG. 7 is a timing chart of a single shooting process in the digital camera system according to the embodiment of the present invention.
FIG. 8 is a timing chart at the time of reading image data from an imaging element in the imaging device according to the first embodiment of the present invention.
FIG. 9 is a timing chart at the time of reading image data from an imaging element in an imaging device according to a second embodiment of the present invention.
FIG. 10 is a timing chart at the time of reading image data from an imaging device in an imaging device according to a third embodiment of the present invention.
[Explanation of symbols]
10 Shooting lens
11 Optical viewfinder
12 Aperture shutter
14 Image sensor
15 Preprocessing circuit
16 A / D converter
18 Timing generator
22 Signal Processing IC
28 Image display section
30 DRAM
48 Built-in flash
50 CPU
52 memory
54 Display
56 flash memory
60 Mode dial
64 shutter switch
68 Flash switch SW
70 Operation unit
72 Menu key
74 Set key
76 Four-way controller
78 Drive mode switching button
80 Power supply control circuit
82 Connector
84 Connector
86 power supply
100 Digital camera device
200 recording medium
202 Recorder
204 interface
206 Connector
400 External strobe device
402 Connector
404 External strobe

Claims (16)

An image sensor that outputs first, second, and third color components of a color filter for one pixel in one field;
imaging control means for forming an image of one frame in n fields (n is an odd number of 3 or more) and reading out pixel data of the image sensor so as to include all color components every field period;
Processing means for performing processing based on pixel data read from the imaging element by the imaging control means;
An image pickup apparatus, wherein the processing means executes the start of the processing before the reading operation for one frame from the image sensor is completed by the image pickup control means. The processing means includes:
The processing means performs a white balance calculation based on the pixel data read from the imaging device by the imaging control means,
2. The image pickup apparatus according to claim 1, wherein a white balance calculation is started before a read operation for one frame from the image pickup device is completed by the image pickup control means. The imaging apparatus according to claim 1, wherein the imaging control unit performs interlace reading for forming an image for one frame by performing reading for every n lines in each field. 3. The imaging apparatus according to claim 2, wherein the processing unit performs a white balance operation based on image data of at least one field among image data of a plurality of fields forming one frame. A display unit for displaying a captured image, wherein the processing unit performs the display based on the image data of the field that has been already read before reading of one frame of image data from the image sensor is completed. 2. The image pickup apparatus according to claim 1, wherein said means starts displaying a photographed image. The image processing apparatus further includes a reduced image forming unit that forms a reduced image having a smaller data amount than the image data for one frame, wherein the processing unit performs the reduction before the reading of the image data for one frame from the imaging device is completed. The imaging apparatus according to claim 1, wherein the image forming unit forms a reduced image. 3. The image pickup apparatus according to claim 2, wherein the processing unit executes a start of white balance calculation by the processing unit before the reading operation for one field from the image sensor is completed by the imaging control unit. An imaging method using an imaging device including an imaging element that outputs first, second, and third color components of a color filter to a pixel in one field,
an imaging control step of forming an image of one frame in n fields (n is an odd number of 3 or more) and reading pixel data of the imaging element so as to include all color components every field period;
A processing step of performing processing based on the pixel data read from the imaging element in the imaging control step,
In the imaging control step, the start of the processing is executed before the reading operation for one frame from the imaging element is completed. The processing step performs a white balance calculation based on the pixel data read from the imaging device in the imaging control step,
An image pickup method, wherein the white balance calculation is started before a read operation for one frame from the image pickup device is completed in the image pickup control step. 9. The image pickup method according to claim 8, wherein in the image pickup control step, reading for every n lines is performed in each field, and interlaced reading for forming an image for one frame is performed. The imaging method according to claim 9, wherein in the processing step, a white balance operation is performed based on image data of at least one field among image data of a plurality of fields forming one frame. A display step for displaying a captured image, wherein display of an image by the display step is started before reading of one frame of image data from the image sensor is completed in the imaging control step. The imaging method according to claim 8, wherein The method further includes a reduced image forming step of forming a reduced image having a smaller data amount than the image data of one frame, and before the reading of the image data of one frame from the imaging device is completed in the temporary storage step, 9. The imaging method according to claim 8, wherein a reduced image forming step is performed. 10. The imaging method according to claim 9, wherein the white balance calculation in the processing step is started before the reading operation for one field from the imaging element is completed in the imaging control step. A program for causing a computer to execute the imaging method according to claim 8. A storage medium storing the program according to claim 15 in a computer-readable manner.

Images