JP5381142B2 - Imaging device and imaging apparatus - Google Patents

Description

  The present invention relates to an imaging element and an imaging apparatus.

  There is known an imaging apparatus in which imaging pixels and focus detection pixels are developed on a plane, and focus detection of a photographing lens is performed by a phase difference detection method based on a distance measurement signal output from the focus detection pixels ( For example, see Patent Document 1).

JP 2000-156823 A

In the conventional imaging apparatus described above, a still image shooting mode for reading out signals of all lines on the image sensor to obtain a high-resolution still image, and an EVF (Electronic View Finder) display (through image display) And a thinning mode for thinning out lines on the image sensor for video shooting and reading out signals, and a ranging mode for reading out signals of lines including focus detection pixels for focus detection of the photographing optical system, When shooting a still image, first display the EVF in the thinning mode to determine the shooting composition, then switch to the ranging mode, perform AF (Auto Focus) of the shooting optical system, and finally shoot the still image The mode is switched to the mode. On the other hand, at the time of moving image shooting, EVF display is performed in the thinning mode and the image signal read out is compressed and recorded in the memory, and the AF of the photographing optical system is performed by switching to the distance measuring mode every predetermined time.
As described above, the conventional imaging apparatus has a problem that the control sequence becomes complicated because the three modes of the still image shooting mode, the thinning mode, and the distance measuring mode must be switched during shooting.

(1) In the first aspect of the invention, a plurality of signals for focus detection are output to a part of an imaging device in which a plurality of imaging pixels that output a signal for generating an image are two-dimensionally arranged. The focus detection pixels are arrayed, and the image pickup pixels and the focus detection pixels are expanded in a two-dimensional shape having an X-axis address and a Y-axis address, respectively, and specify the X and Y addresses. A readout circuit that reads out signals for each pixel, and the readout circuit thins out all pixels in the pixel readout mode and all-pixel readout mode in which signals are read out from all imaging pixels and all focus detection pixels. has a thinning readout mode for reading the signal, the Te, the thinning readout mode, in addition to the signal of the part of the pixels of the image pickup pixel, the signals of all the focus detection pixels, in the read period of one frame Reading With out, at the time of execution of the thinning readout mode, the readout signal of all the focus detection pixels in the vertical blanking period in one frame, and a part of the pixels of the image pickup pixels after the vertical blanking period in one frame Read out the signal .
(2) The focus detection pixel of the image pickup device according to claim 2 includes a photoelectric conversion unit that receives one or both of a pair of light beams that have passed through different regions of the pupil of the optical system.
(3) The invention of claim 3 separates the image sensor according to claim 1 or 2 and the signal output from the imaging pixel and the signal output from the focus detection pixel when executing the thinning readout mode. The signal separation means, the image generation means for generating an image from the signal of the imaging pixel separated by the signal separation means, and the focus adjustment state of the optical system is detected by the signal of the focus detection pixel separated by the signal separation means And a focus adjusting means for adjusting the focus of the optical system based on the focus adjustment state.
(4) The imaging device according to claim 4 is an interpolation unit that obtains a signal for generating an image at the position of the focus detection pixel by interpolation based on a signal of the imaging pixel around the focus detection pixel; White balance evaluation value calculating means for calculating an evaluation value used for white balance adjustment based on the image pickup pixel signal, and the focus detection pixel position obtained by the image pickup pixel signal and interpolation means when executing the all-pixel read mode. And an evaluation value correcting means for correcting an evaluation value used for white balance adjustment based on the above signal.
(5) In the imaging device according to the fifth aspect , it is possible to set whether or not the signal of the focus detection pixel is separated by the signal separation unit, and when the separation setting is made, the image pickup device is repeatedly performed in the thinning readout mode. One of the mode for separating the signal for the focus detection pixel only once in the thinning readout and the mode for separating the signal for the focus detection pixel for every predetermined number of times of thinning readout can be selected.
(6) In the imaging apparatus according to the sixth aspect , the focus adjusting unit includes a CPU and a memory capable of reading and writing at high speed, and when executing the thinning readout mode, the focus detection pixel signal separated by the signal separating unit is stored in the memory. The CPU directly executes focus detection calculation and focus adjustment control based on the signal stored in the memory.
(7) In the image pickup apparatus according to claim 7 , the focus adjusting unit further includes a coprocessor dedicated to calculation, and when executing the thinning readout mode, the signal of the focus detection pixel separated by the signal separating unit is directly stored in the memory. The coprocessor executes a focus detection calculation based on the signal stored in the memory, and the CPU executes the focus adjustment control based on the focus detection calculation result.
(8) In the imaging apparatus according to the eighth aspect , the interpolation unit and the evaluation value correction unit execute the interpolation calculation and the evaluation value correction calculation using a coprocessor and a high-speed readable / writable memory.

  According to the present invention, when AF control is performed in the thinning readout mode, it is not necessary to switch the mode from the thinning readout mode to another mode, and the control sequence can be simplified.

The figure which shows the structure of the digital camera of one embodiment The figure which shows the detailed structure of an image processing circuit Diagram showing typical RGB primary color Bayer array of image sensor The figure which shows an example of the signal reading in thinning mode The figure which shows an example of the signal reading in still image shooting mode The figure which shows the image processing method in still picture photographing mode The figure which shows the example of the setting of the window for detecting AE / AWB evaluation value FIG. 4 is an enlarged view of a line including the sensor cell S1 and a line including the sensor cell S2. Diagram showing signal readout timing in thinning mode Diagram showing an example of AF sensor array arrangement The figure which shows the method of calculating | requiring the image signal of the focus detection pixel position by interpolation Diagram showing the readout order of signals from the AF sensor array The figure which shows one Example of the phase difference calculating circuit which attached importance to cost The figure which shows the other Example of the phase difference arithmetic circuit which attached importance to performance and flexibility

  FIG. 1 is a diagram showing a configuration of a digital camera according to an embodiment, and FIG. 2 is a diagram showing a detailed configuration of an image processing circuit.

  A digital camera according to an embodiment is mainly configured by a DSC engine (system LSI) 6 that controls most functions and operations of the camera. The DSC engine 6 includes a CPU 30, an image processing circuit 22, a JPEG codec 24, a display engine 25, an SDRAM controller (SDRAMC) 32, a Flash memory IF 31, a memory card IF 28, a USB controller 29, a clock generator 33, and a serial port (SIO) 26. A parallel port (PIO) 27 and the like are provided, and various information is exchanged via the image bus 34 and the CPU bus 35. The DSC engine 6 is provided with a timer, an interrupt controller, a DMA controller (DMAC), a general-purpose AD converter, etc., not shown.

  Various external components are connected to the DSC engine 6. The main external components are SDRAM 12, flash memory 9, LCD panel 11, memory card 7, crystal oscillator (X'tal) 10, solid-state image sensor (CCD, CMOS, etc.) 2, AFE (Analog Front End) 3 , TG (Timing Generator) 5, photographic lens 1, motor driver IC (MDIC) 4, etc. The photographing lens 1 is configured as a lens barrel unit, and is provided with a zooming lens, a focusing lens, a mechanical aperture, a mechanical shutter, and the like (not shown) and a motor for driving them.

  In this embodiment, an example in which a CCD is used for the solid-state imaging device 2 is shown, and the color filter of the CCD has a typical RGB primary color Bayer array shown in FIG. In addition to these, the DSC engine 6 is connected to a speaker, a microphone, an operation button, and the like (not shown). In the case of a DSC engine that does not incorporate an audio codec (not shown), this is also externally attached. These external components also operate under the control of the DSC engine 6.

  The SDRAM 12 is used as an image memory and a work memory for the CPU 30, and a high-speed memory device is used because the access frequency is very high. Recently, DDR-SDRAM is often used. The flash memory 9 stores a program (firmware) of the CPU 30, various adjustment values different for each camera, camera setting information by a user, a file of captured image data, and the like. On the other hand, the memory card 7 is a detachable recording medium, and when the storage area is free, the photographed image is preferentially recorded on this side. The LCD panel 11 is a display device built in the camera, and is used to display various camera information such as characters and icons in addition to displaying images such as through images and playback images.

  Here, the general operation of a digital camera using a normal CCD as the solid-state image sensor 2 will be described with reference to FIG. The subject image is formed on the solid-state image sensor 2 by the photographing lens 1. At the time of shooting, the zooming lens, the focusing lens, the mechanical aperture, the mechanical shutter, etc. of the shooting lens 1 are driven by the respective motors to perform shooting. These motors are driven by a motor driver IC 4 in the figure, and the control is performed by the DSC engine 6 via the SIO 26 and the PIO 27.

  The solid-state image pickup device 2 is driven by a pulse of TG5, and the exposure of the subject (accumulation of the image signal charge) and various readout operations of the image signal are controlled thereby. The image signal read from the solid-state imaging device 2 is sampled by the AFE 3 and converted into digital data for each pixel. TG 5 and AFE 3 are controlled by the DSC engine 6 via the SIO 26. This digital image data (RAW data) is input to the DSC engine 6 and processed by the image processing circuit 22 to become YCbCr data. The YCbCr data output from the image processing circuit 22 is stored in the SDRAM 12 via the image bus 34. When YCbCr data is read from the SDRAM 12 and input to the display engine 25 from the image bus 34, the image is displayed on the LCD panel 11.

  Similarly, when YCbCr data is read from the SDRAM 12 and input to the JPEG codec 24 via the image bus 34, the YCbCr data is compressed. The compressed data output from the JPEG codec 24 is stored in the SDRAM 12 via the image bus 34. When this compressed data is read from the SDRAM 12 and sent from the CPU bus 35 to the memory card IF 28, the compressed data is recorded in the memory card 7, and a series of photographing operations is completed. On the other hand, in order to reproduce an image, compressed data is read from the memory card 7 and is temporarily stored in the SDRAM 12 via the CPU bus 35. When this compressed data is read from the SDRAM 12 and input from the image bus 34 to the JPEG codec 24, the compressed data is expanded.

  The decompressed image data output from the JPEG codec 24 is stored in the SDRAM 12 again via the image bus 34. This image data is read from the SDRAM 12 and input from the image bus 34 to the image processing circuit 22 to be converted into an appropriate size. The image data after resolution conversion output from the image processing circuit 22 is stored in the SDRAM 12 via the image bus 34. When this image data is read from the SDRAM 12 and input from the image bus 34 to the display engine 25, a reproduced image is displayed on the LCD panel 11. If no resolution conversion is required, the image data after JPEG decompression is displayed as it is.

  In order to save the image data (JPEG file etc.) recorded on the memory card 7 in the host PC 8, the camera and the host PC 8 are connected by a USB cable. First, an image file is read from the memory card 7 and is temporarily stored in the SDRAM 12 via the CPU bus 35. When this image file is read from the SDRAM 12 and sent to the USB controller 29 via the CPU bus 35, it is stored in the host PC 8. The file on the host PC 8 can be recorded on the memory card 7 of the camera in the reverse flow.

  Next, signal readout from the solid-state imaging device 2 at the time of shooting and the operation of the image processing circuit 22 will be described. First, the camera has two different shooting operations, one is an EVF display (through image display) operation, and the other is a full-resolution still image shooting operation. The difference between these two operations is the difference in the signal readout mode from the solid-state imaging device 2. That is, in the EVF display operation, signals are read from the solid-state imaging device 2 in the thinning mode, but in the still image shooting operation, signals of all pixels are read from the solid-state imaging device 2 in the still image shooting mode.

  FIG. 4 shows an example of signal readout in the thinning mode. In the figure, “n” and “n + 1” represent line numbers in the thinning mode. In the thinning mode, only a small number of pixel line signals are read out, so the vertical resolution is lowered, but since the signal reading time for one frame is shortened, a high frame rate such as about 30 frames / sec is realized. This is suitable for observing a subject image (through image) in real time, AF (Auto Focusing) control, and AE (Auto Exposure) photometry. In the conventional imaging apparatus, signals are read out only for the imaging pixels R and G lines and the imaging pixels G and B in the figure, and the lines including the AF sensor cells (focus detection pixels) S1 and S2 are read. Although no signal is read out, in this embodiment, the signal of the line including the AF sensor cells S1 and S2 is read out in the thinning mode. Details of the signal readout in the thinning-out mode of this embodiment will be described later.

  FIG. 5 shows an example of signal readout in the still image shooting mode. In the still image shooting mode, since the signals of all the pixel lines are read out, it takes a long time, and since the pixel lines of the image sensor 2 are read out in a jump, the image signal of one frame is divided into a plurality of field signals ( Interlace reading). On the other hand, in the above-described thinning mode, the pixel lines of the image sensor 2 are read in order from top to bottom, so that they are output as frame image signals from the beginning (progressive reading). As described above, since the image signal reading operation is different between the still image shooting mode and the thinning mode, the procedure of each image processing is greatly different correspondingly.

  In FIG. 2 showing the detailed configuration of the image processing circuit, the control blocks indicated by shading, that is, the AF signal interpolation circuit 61 and the AWB correction value circuit 62 are new control blocks that were not found in the conventional digital camera. Details of these new control blocks 61 and 62 will be described later. First, control blocks of a conventional digital camera will be described.

  Image data (RAW data) input to the image processing circuit 22 is first subjected to defect pixel correction by the defect correction circuit 41. The address of the defective pixel is registered in the correction table 46, and the pixel data matching this address is replaced with a correction value calculated from the surrounding pixels. Subsequently, an OB clamp process for determining the black level of the image is performed by the OB clamp circuit 42. The CCD of the solid-state imaging device 2 is provided with an OB (Optical Black) region, and the black level is determined by detecting the average value of the signal in the OB region and subtracting it from the value of each pixel.

  Next, the image data is subjected to different gain adjustment (sensitivity ratio adjustment) for each color by the sensitivity ratio adjustment circuit 43. Since the pixels of the image sensor 2 have different sensitivities for each color, even when a white (gray) subject is photographed, the signal levels of the R, G, and B colors are different and are not white (gray) as they are. Therefore, it is necessary to match the sensitivity (signal level) of each color of R, G, and B under a standard light source (for example, D65: daylight). Normally, the gain of G is fixed to 1 and the gain adjustment (multiplication) is performed so that the sensitivity of R and B matches G.

  The image processing so far is the processing by the pre-processing unit (Pre-Process) 22a. In the still image shooting mode, the image data is temporarily transferred to the SDRAM 12 here. Therefore, an output buffer 44 (FIFO or the like) is provided for the image bus at the final stage of preprocessing. On the other hand, in the thinning mode, the processing by the subsequent post-processing unit 22b is continuously executed. Since the thinning-out mode is an image signal for progressive readout, preprocessing and post-processing can be performed together. On the other hand, since the still image shooting mode is a field image signal for interlace readout, post-processing cannot be performed as it is.

  Therefore, the post-processing operations in the two modes are divided below, and first, post-processing in the thinning mode will be described. In the post-processing, horizontal thinning processing 51 is first performed by the horizontal thinning circuit 51 to reduce the number of horizontal pixels of the image data. In the post-processing, two-dimensional digital filter processing is performed several times, so a line memory is provided. In the thinning mode, the number of lines is reduced, but the number of horizontal pixels is still large, so that the line memory overflows as it is. Since the size of the through image is at most about VGA (640 × 480), the number of horizontal pixels may be reduced to the number necessary to create it (for example, over 700 pixels). This reduces the capacity of the line memory.

  Next, the WB adjustment circuit 52 performs WB (White Balance) adjustment on the image data. Since the spectral distribution of the illumination light varies depending on the shooting environment, even if a white (gray) subject is shot, it is not white (gray) as it is. The signal levels of the three colors are adjusted by applying different gains for each of R, G, and B colors. Normally, the gain level of G is fixed to 1, and the signal levels of R and B are adjusted to G. How to determine the gain of R and B will be described later. After the WB adjustment, the γ correction circuit 53 performs γ correction on the image data. In γ correction, gradation conversion suitable for an output device such as a display device is performed. For a well-known sRGB color space display, “γ = 2.2”.

  Up to this point, the Bayer array signal of one color per pixel has been converted into a normal color image signal composed of three colors per pixel by the color interpolation processing by the color interpolation circuit 54. Thereafter, three color signals flow through the image processing circuit 22 in parallel. The subsequent color conversion / color correction circuit 55 performs conversion to an output color space signal and correction for realizing a preferable color. Typical color spaces are sRGB suitable for display and AdodeRGB suitable for printing workflow. In color correction, memory colors such as skin color, sky and sea color are expressed as preferable colors. In color conversion / color correction, color matrix processing is mainly used, but when performing advanced color correction, LUT (Look Up Table) is also used. The image data after this color conversion / color correction is normally YCbCr4: 4: 4.

  The YCbCr data undergoes resolution conversion by the resolution conversion circuit 56 and is converted to a target size. For a through image, QVGA (320 × 240) and VGA are common. Next, the spatial filter circuit 57 performs spatial filter processing on the image data. Here, edge enhancement of the Y signal and LPF processing of the Cb / Cr signal are performed. Finally, the Cb / Cr signal is thinned by the CbCr thinning circuit 58 to be converted into YCbCr4: 2: 2 data and stored in the SDRAM 12 via the image bus 34, thereby completing the post-thinning mode post-processing. Therefore, an output buffer (FIFO or the like) 63 is provided toward the image bus 34 at the end of the post-processing. This data is sent to the display engine 25 and displayed on the LCD panel 11 as a through image.

  Next, processing by the post-processing unit 22c in the still image shooting mode will be described. In the still image shooting mode, as shown in FIG. 5, since the image signal is read out in the order of skipped lines (interlace), there is no adjacent line. Therefore, color interpolation processing cannot be directly performed on such a field image signal. Therefore, the field image signals that have undergone preprocessing are once discharged from the image processing circuit 22 and arranged so that they are arranged in the order of adjacent lines (progressive) on the SDRAM 12. If this image signal is read from the SDRAM 12 and input to the image processing circuit 22 from the image bus 34, post-processing can be performed. Therefore, an input buffer (such as a FIFO) 59 is provided from the image bus 34 toward the post-processing circuit 22.

  However, since a high-resolution image is created in the still image shooting mode, horizontal thinning in the figure is skipped. If this is the case, image data having a large number of horizontal pixels is sent to the subsequent processing. Therefore, in the case of a processing block having a line memory, the line memory overflows and cannot be processed. Therefore, in the still image shooting mode, a large still image is divided into strip-like blocks having a short width as shown in FIG. 6, and post-processing is performed for each block. In post-processing, some digital filter processing (convolution) is performed, but after that processing, peripheral pixels are cut off and the image size is slightly reduced. Since the processed images must be smoothly connected at the block boundaries shown in FIG. 6, at least the pixels to be cut off are processed by overlapping the blocks. The post-processing of each block is performed in the same flow as the thinning mode, but the image processing parameters are generally different.

  In the still image shooting mode, in addition to such a high-resolution main image, a display image for confirmation of shooting (also referred to as a freeze image) and a search thumbnail are often created. Since the latter two images are created from a large original image with a small size, if post-processing is performed in the same flow as the main image, block division is required as shown in FIG. Therefore, when creating a freeze image or a thumbnail, processing is performed in advance by reducing the number of horizontal pixels to near the target size using horizontal thinning. Since post-processing is performed on the original image data of a large size as it is, a small image is created in one pass. In this case, the post-processing is exactly the same flow as the thinning mode.

  In the post-processing, after color conversion / color correction, it is used not only for shooting but also for playback, so a data path from the input buffer (FIFO, etc.) in the figure to color conversion / color correction is also prepared. ing. This path is used for color conversion “YCbCr ← → RGB”, color correction at the time of image reproduction, or resolution conversion for displaying an image of a size suitable for the LCD panel 11. Image processing in the still image shooting mode is divided into pre-processing and post-processing. The image signal is divided into a plurality of fields, and it must be divided into strip-shaped blocks as shown in FIG. This is due to two reasons.

  Next, control by the detection circuit 47 will be described. Here, the targets of detection are AE and AWB (Auto White Blannce). In FIG. 2, a detection circuit 47 is provided, in which AE and AWB evaluation values 45 are extracted from the image data after the sensitivity ratio adjustment. The AE and AWB evaluation values are information necessary for performing each control of AE and AWB.

  A pupil division type phase difference detection method is used for AF control. This is because a pair of luminous fluxes that have passed through different regions on the exit pupil plane of the photographic lens 1 are received by the pair of AF sensor cells S1 and S2 of the image sensor 2, and a signal obtained from the array of the AF sensor cells S1 and an AF sensor. Based on the phase difference from the signal obtained from the array of the cells S2, the focus adjustment state of the taking lens 1, that is, the defocus amount is detected.

  The evaluation values of AE and AWB are extracted by setting a window, and these are often extracted using the same window setting. FIG. 7B shows a window setting example of the AE / AWB evaluation value. The pixel values contained in these windows are integrated for each color to obtain an AE / AWB evaluation value for each window. In the case of exposure by AE, exposure is performed by determining an exposure amount (aperture value and shutter time) from the AE evaluation value. On the other hand, the AWB evaluation value represents a deviation from the correct color. If the color of a window is white (gray), if the same number of R, G, and B values in that window are added for each color, they should be approximately equal. Since the spectral distribution changes, the integrated values do not match.

  Therefore, if the ratio of the gains of the three colors by the WB adjustment circuit 52 shown in FIG. 2 is adjusted so as to match the inverse ratio of the three integrated values, the color should be white (gray). Actually, as described above, the gain of G is fixed to 1, and the gains of R and B are adjusted to coincide with the inverse ratio of the three integrated values. This is the principle of WB (White Balance) adjustment. AWB means that the camera automatically performs WB adjustment based on the AWB evaluation value. In the case of the Bayer arrangement color filter shown in FIG. 3, there are twice as many G pixels as R and B. Therefore, a value obtained by multiplying the integrated value of G in each window by “½” is used as the AWB evaluation value. Therefore, the size of each window should be set to a multiple of “2” for both horizontal and vertical.

  In general, since the center of the image is the most important, the number of AE and AWB evaluation value windows tends to be set to odd numbers for both horizontal and vertical as shown in FIG. Note that since the WB adjustment is performed in the post-processing, the sensitivity ratio adjustment in the pre-processing is not essential. When the sensitivity ratio adjustment is not performed, the AE / AWB evaluation value includes the error. Therefore, it is necessary to correct by multiplying the integrated value of R and B by a gain corresponding to the sensitivity ratio adjustment. Therefore, it is desirable to adjust the sensitivity ratio. In the thinning mode, AE and AWB evaluation values are extracted at appropriate frame intervals while displaying a through image, and AE and AWB are controlled. Since the mechanical shutter remains open, TG's electronic shutter is used for AE control.

  AWB control is performed by adjusting the gains of R and B as described above. Since the brightness and color temperature of the subject (spectral distribution of illumination light) rarely change rapidly, the control cycle of AE / AWB may be long. A pupil division type phase difference detection method is used for AF control. Except when tracking the subject, AF control is often performed once each when the live view display starts and when the release button is pressed halfway. When the release button is pressed halfway, in preparation for taking a still image, the exposure amount (aperture value and shutter time) is calculated based on the AE evaluation value in addition to AF control.

  When the release button is fully pressed, the operation proceeds to a still image exposure operation. The shooting lens is focused on the subject by AF control with the previous half-press, and the mechanical aperture value and shutter time are determined by the calculation of the exposure amount performed at the same time, so the subject as a still image is based on it Are exposed. The exposure operation is terminated by closing the mechanical shutter. Next, an image signal is read from the CCD of the image sensor 2 in the still image shooting mode, and image processing of the still image is performed according to the above-described procedure. In this case, only the AWB evaluation value is extracted from the 3A evaluation values. Since the exposure is over, there is no need to extract AE and AF evaluation values.

  Since the AWB evaluation value of the still image is extracted for each field, in order to obtain the AWB evaluation value of one whole image (frame), the evaluation values for each field are integrated. If the AWB detection circuit does not have this integration function, the evaluation value of each field is integrated by the CPU software. R and B gains are determined from the accumulated AWB evaluation values and set in the WB adjustment circuit before post-processing. As described above, since the AWB evaluation value is extracted from the captured still image data and the AWB is adjusted, the accuracy is high. This is why the WB adjustment is included in the post-processing.

  On the other hand, the thinning mode (through image display) is different in that the extracted AWB evaluation value is used for AWB adjustment of the subsequent frames. Note that the same thinning mode as the viewfinder operation (through image display) is used in the moving image shooting operation, and the through image data is directly JPEG compressed and recorded on the memory card.

  By the way, the imaging device 2 of the embodiment is a type of imaging device incorporating a phase difference AF type sensor. The imaging device 2 is obtained by replacing a part of pixels (photoelectric conversion cells) forming an image signal with sensor cells (focus detection pixels) for the phase difference AF method as shown in FIG. FIG. 8 is an enlarged view of a line including the sensor cell S1 and a line including the sensor cell S2 shown in FIG. This AF sensor consists of a one-dimensional array of sensor cells S1 and S2, and is used in a set of two different sensor arrays. Each sensor cell of this sensor array has openings S1a and S2a that are biased to one side from the center, and are arranged one-dimensionally along the direction of the bias. In each sensor cell of the two sensor arrays, the openings S1a and S2a are biased in opposite directions, and the bias distance is the same. The two sensor arrays are placed side by side at positions that are considered substantially identical.

  As shown in FIGS. 4 and 8, in the image pickup device 2, AF sensor cells S1 and S2 having openings S1a and S2a eccentric in opposite directions are placed in place of the G pixels in the RGB primary color Bayer array. Yes. As shown in FIGS. 4 and 8, the AF sensor cells S1 and S2 may be arranged on adjacent lines, or may be arranged alternately on the same line. These AF sensor cells S1 and S2 realize a pupil division phase difference AF method. That is, if two partial light beams that are symmetric with respect to the optical axis of the photographing lens 1 are received by the sensor array of S1 and the sensor array of S2, respectively, among the light beams passing through the exit pupil, the two sensor arrays From the phase difference of the output signal, the direction of focus shift (moving direction of the focusing lens) and the amount of focus shift (moving amount of the focusing lens) are known. This enables quick focusing.

  Since the position of the subject always moves in the shooting screen, it is desirable to provide this AF sensor array in several places in the shooting screen. In each of these locations, the sensor arrays S1 and S2 are arranged side by side. Note that the shapes of the openings S1a and S2a of the sensor cells S1 and S2 are not limited to the rectangle shown in FIG. 8, and may be, for example, a semicircle.

  Incidentally, since the AF sensor cells S1 and S2 are embedded in the imaging region, there is a problem that the image signal of that portion is lost. In this embodiment, the image sensor 2 is used while solving the problem. First, in the still image shooting mode, signals from all photoelectric conversion cells are read out, so that a signal including both an image signal and an AF signal is output. If this is the case, the image signal from the AF sensor section is missing, so a still image with satisfactory image quality cannot be obtained.

  Therefore, the AF signal is replaced with a signal interpolated from surrounding image signals. Since it is difficult to perform this interpolation processing in real time (On the fly), all signals are temporarily stored in the SDRAM 12, and interpolation is performed later taking time. Since the AF sensor array is provided only in several places on the shooting screen, even a still image obtained by performing such interpolation is sufficient in image quality.

  On the other hand, an EVF operation (through image display) is also performed in the digital camera. In this case, since signals are continuously read from the image sensor 2 at a high rate of about 30 fps, interpolation processing is performed later taking time. I can't do that. In this case, since the thinning mode is used, the interpolation process is eliminated so that the line in which the AF sensor is embedded is thinned out. AF sensor arrays are provided at several places on the shooting screen, and even if they are thinned out, sufficient image signals are secured to create a through image.

  However, since the AF signal cannot be obtained in the thinning mode, there arises a problem that AF cannot be controlled this time. This is a problem because digital cameras perform AF control during EVF operation (through image display). A conventional imaging apparatus is provided with a distance measuring mode, and when performing AF control, the mode is changed from the thinning mode to the distance measuring mode. In the distance measurement mode, only a small number of lines in which the AF sensor array is embedded are read out, so the period in which the distance measurement mode is entered is small. If the release button is operated from halfway down to full-press at once, the mode switches to the distance measurement mode when half-pressing is detected.However, since full-pressing is detected continuously, thinning is performed after AF control is completed. Without returning to the mode (through image display), the still image exposure operation starts.

  When performing AF control, photometry for AE is also performed. Photometry is performed in the same manner as a conventional camera, using the AE evaluation value in the thinning mode immediately before shifting to the distance measuring mode. When the release button is fully pressed, a still image exposure operation is started, and exposure by AE is performed according to the exposure amount (mechanical aperture value and shutter time) obtained by photometry when half-pressed.

  When the exposure is completed, the image signal and the AF signal are read out by moving to the above-described still image shooting mode, and then the AF signal is replaced with the image signal by the interpolation process described above. When the interpolation processing is completed, the above-described image processing (post-processing) is performed to obtain YCbCr data. When this is JPEG compressed and recorded in the memory card 7, still image shooting is completed. On the other hand, when the reading of the still image shooting mode is finished, the camera shifts to the thinning mode, and the EVF operation (through image display) is resumed in preparation for the next still image shooting. However, when the release button operation is only half-pressed and not fully pressed, when the AF control is completed, the AF mode operation (through image display) is resumed from the ranging mode to the thinning-out mode.

  Next, a new configuration that is not included in the conventional imaging device in the configuration of the embodiment will be described. The digital camera according to the embodiment has a still image shooting mode and a thinning mode, and there is no distance measuring mode used in a conventional imaging apparatus. The readout operation in the still image shooting mode is the same as the readout operation of the conventional imaging apparatus, and signal readout as shown in FIG. 5 is performed. However, since an AF photoelectric conversion cell is included, a part of the image signal shown in FIG. 5 is changed to the signals of the AF sensor cells S1 and S2. In the arrangement of FIG. 4, the G signal in the fourth line (fourth field) from the top of FIG. 5 is changed to the signal S1, and the G signal in the sixth line (second field) from the top is changed to the signal S2. ing.

  On the other hand, the readout operation in the thinning mode is significantly different from that of the conventional imaging apparatus. In this embodiment, as shown in FIG. 4, the thinned-out image signal is reduced, and the S1 and S2 AFs that are not thinned out. Signals are mixed and read. Since the readout operation in the still image shooting mode is the same as the operation of the conventional imaging apparatus, the description thereof will be omitted, and the readout operation in the thinning mode will be described in detail below. The thinning mode is used in EVF operation (through image display) and moving image shooting, which is progressive readout in which all signals are read out in a 1V period (one cycle of the VSync signal).

  As shown in FIG. 4, signals are read in order from the upper line to the lower line. That is, “R / G” is read out as the image signal of line number n, then the line including the signal of S1 and the line including the signal of S2 are sequentially read, and then the image of line number n + 1 The signal “G / B” is read out. Since the line including the S1 signal and the line including the S2 signal are not used for creating a through image (moving image), no line number is assigned to these lines. Also, in FIG. 4, the shaded lines in the left array are thinned lines, and no image signal is output. As shown in FIG. 5, in the still image shooting mode, all signals (including S1 and S2) including the shaded lines shown in FIG. 4 are read out.

  FIG. 9 shows signal readout timing in the thinning mode shown in FIG. FIG. 9A shows the readout timing of the embodiment, and FIG. 9B shows the readout timing of the conventional imaging apparatus shown in Patent Document 1. In the thinning mode, since a signal is read out in a 1V period, only one cycle of the VSync signal is shown in the upper half of FIGS. 9 (a) and 9 (b). The upper half of FIG. 9A represents a read operation from the previous line (line number n-1) shown in FIG. 4, and the beginning and end of the frame including the front and rear vertical blanking periods are omitted. Has been.

  In the figure, n-1, n, and n + 1 represent the line numbers of the image signals. Next to the image signal (R / G) of line number n, the S1 / B line is read out, followed by the S2 / B line, and then the image signal of line number n + 1 (G / B) continues, which is consistent with the signal read operation shown in FIG. The lower half of FIG. 9A shows the read operation of the S1 / B or S2 / B line. In the middle of the line, the S1 or S2 signal and the B signal are alternately read. It matches the sequence shown in 4.

  On the other hand, in the conventional imaging apparatus, only the line of the image signal is read as shown in the upper half of FIG. In the figure, n-2, n-1, n, n + 1, and n + 2 represent line numbers, respectively. The lower half of FIG. 9B represents the reading operation of one line in the lower half, and includes only the image signal. Since the upper half of FIG. 9B does not include the lines of AF sensors (S1 and S2), if the number of effective image lines is the same, the image signal can be transmitted in a shorter period than the upper half of FIG. 9A. Reading is finished. In order to avoid dropping the frame frequency, in the read operation shown in the upper half of FIG. 9A, a part of the vertical blanking period is assigned to the period for reading the S1 / B and S2 / B lines. The vertical blanking period is shortened by the number of lines of S1 / B and S2 / B, but there is no problem if it is long to some extent.

  Since it is desirable to provide the AF sensor array at several places on the image pickup screen, an example of a solid-state image pickup device provided with the AF sensor arrays at five places is shown in FIG. In FIG. 10A, reference numerals 1, 2, 3, 4, and 5 represent AF sensor array numbers, each of which is composed of a sensor array of S1 and S2. FIG. 4 shows the signal readout operation near one AF sensor array. However, if multiple AF sensor arrays are provided, an extra period is required to read out signals from other AF sensor arrays. .

  In FIG. 10A, if the AF sensor arrays 2 and 3 and 4 and 5 are arranged on the same line, the AF sensor array 1 is matched because each includes the S1 line and the S2 line. A total of 6 lines need to be read out. When signals are read out from the image sensor 2 shown in FIG. 10A in the thinning mode, the vertical blanking period is shortened by 6 lines. Considering this point, the frequency of the VSync signal and the HSync signal must be determined.

  A typical solid-state imaging device that performs readout as shown in FIGS. 4 and 5 is a CCD. Since the CCD reads out the signal in line units, it is suitable to place the S1 and S2 AF sensor arrays in the horizontal direction as shown in FIG. Since the image signal (B signal in the case of FIG. 4) on the line where S1 and S2 are placed is not used for creating a through image or a moving image, the time for reading out the signal is wasted. Because of the structure of the CCD, when reading the S1 and S2 signals, the image signal on the same line is read together, so it takes a long time to use the S1 and S2 signals. Yes.

  In FIG. 1, an image / AF signal separation circuit 21 is newly added to the conventional imaging apparatus. 2 is a control block in which an AWB correction value circuit 62 is newly added to the conventional imaging apparatus. The photographing operation of the digital camera according to the embodiment will be described focusing on the new control block according to the embodiment.

  When the exposure of the subject is finished, the CCD of the image sensor 2 shifts to the still image shooting mode, and the image signal of the shot still image is read out according to the procedure shown in FIG. The image signal is converted to digital data by the AFE 3 and sent to the image / AF signal separation circuit 21. In the still image shooting mode, the image signal is bypassed and the image data is directly input to the image processing circuit 22. The image processing circuit 22 first performs preprocessing, and the image data after the preprocessing is temporarily stored in the SDRAM 12.

  When the pre-processed still image data is stored in the SDRAM 12, the data read in the jump order (interlace) is arranged so as to line up in the line order (progressive) as shown in FIG. However, since this still image data includes AF signals (S1 and S2 signals), care must be taken when performing preprocessing. For example, when a defective pixel exists in the vicinity of the AF sensor (S1 and S2), the signal of S1 or S2 must not be used when correcting the signal of the defective pixel. Since correction of defective pixels is usually performed by interpolation with surrounding image signals, if the defective pixels near the AF sensor are simply corrected by this method, the signals of S1 and S2 may be used. Therefore, when the signal of S1 or S2 is referenced, the defective pixel is output as it is without correction. Such correspondence is possible because the position of the AF sensor is known in advance. To detect S1 and S2 signals from the flowing image signal, the line number (HSync signal count value based on the VSync signal) and the pixel number on the line (pixel clock based on the HSync signal) Count value).

  Since the OB clamp process only subtracts the OB level (signal average of the OB area) from the value of each pixel, even if signals of S1 and S2 are mixed, the process is performed without any problem. Since it is not necessary to apply OB clamping to the signals of S1 and S2, it is bypassed (may be processed). In that respect, the following sensitivity ratio adjustment is the same. Since the sensitivity ratio adjustment is a gain adjustment (multiplication) of R and B, even if signals of S1 and S2 are mixed, they are processed without any problem. Since it is not necessary to adjust the sensitivity ratio to the signals of S1 and S2, it is bypassed (may be processed).

  However, care must be taken in extracting AWB evaluation values. In the still image shooting mode, it is only necessary to extract the AWB evaluation value as described above. However, since the still image data includes the S1 and S2 signals, the S1 and S2 signals must be excluded when integrating the color signals. Since any of the AWB evaluation value windows shown in FIG. 7B includes S1 and S2 signals, the total number of each color signal differs in that window, and a correct AWB evaluation value cannot be obtained. Since the AWB evaluation value of the window including the signals S1 and S2 must be corrected later, it is necessary to prevent the signals S1 and S2 from being integrated as described above.

  The preprocessing of the image processing circuit 22 shown in FIG. 2 corresponds to such an exception. The pre-processed still image data is temporarily stored in the SDRAM 12 in the arrangement shown in FIG. 11 via the output buffer 63 and the image bus 34 shown in FIG. Next, the still image data is subjected to post-processing. Since the signals S1 and S2 are included as shown in FIG. 11, this portion must first be interpolated with surrounding image signals. . Since the signals S1 and S2 are only partially present at a plurality of locations, interpolation may be performed by cutting out the portions including the peripheral image signals.

  The image processing circuit 22 shown in FIG. 2 is provided with an AF signal interpolation circuit 61 that moves independently of the pre-processing and post-processing, and uses this to interpolate the signal portions S1 and S2. In this embodiment, since an AF signal interpolation circuit 61 having a hardware configuration is provided, the processing is performed in a very short time compared with the case where interpolation is performed by software using a microprocessor. For example, a bicubic method is used as an interpolation algorithm. When an edge of an image is present near the signals S1 and S2, if interpolation is performed in the direction across the edge, it becomes dull. Therefore, it is desirable to perform interpolation only from peripheral image signals in the edge direction. The AF signal interpolation circuit 61 needs to have such an edge determination function.

  In the example shown in FIG. 11, the signals S1 and S2 are replaced with G signals, but this is usually interpolated with G signals in the direction of ± 45 ° existing on adjacent R / G lines. Depending on the direction of the edge, it is inappropriate to interpolate with the G signal in the direction of ± 45 °. In this case, interpolation is performed using the outer G signal. When determining the edge direction of an image, it is preferable to refer to as many surrounding image signals as possible. When this interpolation processing is performed, an area including S1 and S2 signals is cut out together with peripheral image signals, and the data is input from the image bus 34 to the input buffer 59 shown in FIG. When a DMAC (not shown) is used, this data transfer is performed at high speed.

  Since this data transfer is performed on a rectangular area, image signals that are not necessary for interpolation are also included. However, in general, data transfer is faster, so there is no problem if unnecessary image signals are ignored. . The data input to the input buffer 59 is processed by the AF signal interpolation circuit 61, and the image signal created by the interpolation is stored in the output buffer 63. When the output buffer 63 is filled with the image signal generated by the interpolation, it is output to the SDRAM 12 and stored using the DMAC described above. Although the input buffer 59 and the output buffer 63 are also used in post-processing, in this case, post-processing is not performed, so the AF signal interpolation circuit 61 may occupy.

  As shown in FIG. 11, since the signals S1 and S2 exist at discontinuous addresses, it is difficult to overwrite only the G signal generated by interpolation. It is easier to overwrite the entire original rectangular area including the surrounding original image signal cut out together with the signals of S1 and S2.

  Subsequently, the AWB evaluation value is corrected using an image signal generated by interpolation. In the image processing circuit 22 shown in FIG. 2, an AWB correction value is first generated. In the case of the image data shown in FIG. 11, since the G signal is generated by interpolation, the AWB correction value is the integrated value. The AWB correction value is added to the AWB evaluation value (G integrated value) of the window in which the signals S1 and S2 were included. When the signals of S1 and S2 are distributed over a plurality of windows, an AWB correction value is calculated for each window and added to the AWB evaluation value of each window. 2 indicates that the AWB evaluation value is corrected with the AWB correction value for each window.

  When the correction of the AWB evaluation value is automatically performed, it is most easy to use. However, if this is difficult, the CPU 30 shown in FIG. The above interpolation processing and AWB evaluation value correction are performed for all locations where AF sensors (S1 and S2) are placed. For example, as shown in FIG. 10A, when five AF sensors are arranged, the above-described interpolation processing and AWB evaluation value correction are performed at the five positions.

  When the correction of the AWB evaluation value is completed, the R and B gains of the WB adjustment circuit are subsequently obtained. This is determined according to the AWB algorithm based on the AWB evaluation value. The gains R and B are set in the WB adjustment circuit 52 shown in FIG. Since the digital camera according to the embodiment corrects the AWB evaluation value in this way, the accuracy of AWB is high. Now that all the preparations for post-processing are complete, post-processing is performed on the still image data after interpolation. The interpolated image data is read from the SDRAM 12 and input from the image bus 34 to the input buffer 59 shown in FIG. This data transfer is performed by DMA transfer using a DMAC (not shown).

  The image data input to the input buffer 59 is first sent to the horizontal thinning circuit 51, and post-processing is started. When a large image such as a main image is created by post-processing, the horizontal thinning circuit 51 is bypassed instead of processing in strip-shaped blocks as shown in FIG. When creating a small image such as a thumbnail or a freeze image, the horizontal thinning circuit 51 first reduces the number of horizontal pixels instead of inputting large image data as it is. The subsequent processing is the same as that of the conventional imaging device described above.

  The YCbCr data created by post-processing is temporarily stored in the output buffer 63 shown in FIG. When the output buffer 63 is filled with YCbCr data, it is sent to the SDRAM 12 by DMA transfer and stored. In the case of the main image, this is repeated for each strip-shaped block, and when the YCbCr data of each block is combined on the SDRAM 12, one large main image is completed. The above is the operation of the digital camera of the embodiment in the still image shooting mode.

  Next, the operation in the thinning mode in the digital camera of the embodiment will be described. In the thinning mode, a signal is read from the CCD of the image sensor 2 as shown in FIGS. 4 and 9A. A line including an AF signal is read out by being mixed with an image signal whose number of lines has been thinned out. The signal is first converted into digital data by the AFE 3 and then input to the image / AF signal separation circuit 21 of FIG. The two types of signals are separated here and output separately. In FIG. 9A, a line including only image signals such as..., N-1, n, n + 1,... And a line including an AF signal sandwiched between lines n and n + 1. Separated and output separately from the image / AF signal separation circuit 21.

  Since the line containing the AF signal is known in advance, the image / AF signal separation circuit 21 is set so that separation is performed when the period starts. In the case of FIG. 9A, setting is made so that two lines following the line n are separated. The line including only the image signal output from the image / AF signal separation circuit 21 is output to the image processing circuit 22 shown in FIGS. 1 and 2 and subjected to image processing. In the thinning mode, pre-processing and post-processing are performed together, and live view and moving image data are created.

  However, there is no image signal during the period of the line in which the AF signal was included, and if the image processing circuit 22 is operated during this period, incorrect data will be mixed. During that period, the image processing circuit 22 must be stopped, and the image processing circuit 22 needs to have such a function. In the case of FIG. 9A, the image processing circuit 22 is set to stop during the period of two lines following the line n. The line including the AF signal is removed by the image / AF signal separation circuit 21, and the image processing circuit 22 performs image processing only on the image signal, so that a normal through image or moving image is created.

  In parallel with this image processing, 3A evaluation values are extracted. However, since the AF signal is output separately, it is only necessary to extract the AE evaluation value and the AWB evaluation value. The generated through image and moving image data are temporarily stored in the output buffer 63 of the image processing circuit 22, and then stored in the SDRAM 12 via the image bus 34 by DMA transfer. In the thinning mode, the AF signal interpolation circuit 61 and the AWB correction value 62 of the image processing circuit 22 are not used. The operation of this image processing is the same as that of the conventional imaging device described above.

  On the other hand, the line including the separated AF signal is further separated into an image signal and an AF signal in the image / AF signal separation circuit 21. The image / AF signal line shown in FIG. 9A shows the detailed timing below, and in part, the image signal and AF signal are synchronized with the pixel clock corresponding to the arrangement of FIG. Flow alternately. In the image / AF signal separation circuit 21, only the AF signal is further extracted from this line, and the remaining image signal is discarded without being used. Since the position of the AF signal is known in advance, only the AF signal is extracted from the count value of the pixel clock based on the HSync signal.

  The AF signal is temporarily stored in an output buffer (FIFO or the like) (not shown) provided in the image / AF signal separation circuit 21 and then stored in the SDRAM 12 by DMA transfer via the image bus 34 shown in FIG. . When AF sensors are arranged at a plurality of locations as shown in FIG. 10A, AF signals at all locations are stored in the SDRAM 12. Since there are only a few S1 and S2 photoelectric conversion cells that make up the AF sensor, and only the signals of S1 and S2 are taken out, all the five AF signals are shown as shown in Fig. 10 (a). Even if the data is transferred to the SDRAM 12, the data traffic does not become large.

  The AF signal is read from the SDRAM 12 and input to the phase difference calculation circuit 23 shown in FIG. In the phase difference calculation circuit 23, first, the phase difference between the signal sequence S1 and the signal sequence S2 is calculated. Subsequently, the phase difference calculation circuit 23 obtains the moving direction and the moving amount of the focusing lens from this phase difference. The moving direction and moving amount are stored in a register (not shown) in the phase difference calculation circuit 23 and read by the CPU 30. If an interrupt request is issued when these values are stored in the register, the CPU 30 does not have to perform a useless read operation, which is efficient.

  If the arrangement is as shown in FIG. 10 (a), the AF sensor that overlaps the main subject is selected from five locations and these calculations are performed. If two or more overlap the main subject, the phase difference is clear. Choose the AF sensor of your choice and perform the calculation. When the CPU 30 sends a command to the motor driver IC 4 in accordance with the read information, a focusing motor (not shown) is driven, and the focusing lens moves to a target position to focus on the subject.

  The digital camera according to the embodiment can create a through image and control AF during the thinning mode, so that there is no trouble of switching between the thinning mode and the ranging mode unlike a conventional imaging device. Therefore, a great merit that the through image and the video are not interrupted is born.

  By the way, although the AF signal is output for each frame in the thinning mode, there is little need to control AF for each frame during EVF operation (through image display) or moving image shooting. If it is still image shooting, it may be performed once every time the release button is pressed halfway. Even in the case of moving image shooting, AF control may be performed once at an appropriate frame interval. Therefore, storing unused AF signals in the SDRAM 12 is wasteful because of increased power consumption.

  Therefore, the image / AF signal separation circuit 21 can be set to perform or not perform separation and output of AF signals. As a result, the AF signal is separated and output only when necessary, so that power consumption can be reduced. Although this setting may be switched every time AF control is required, the image / AF signal separation circuit 21 can be provided with a more convenient function. That is, two modes are provided when AF signal separation and output are performed. The first is a single output mode in which AF signal is separated and output for one frame immediately after the setting becomes effective, and then stops. The second is AF for only one frame at a specified frame interval. This is a repeated output mode in which signal separation and output are performed.

  In the case of still image shooting, since it seems that most of the single output mode is sufficient, it can contribute to reduction of power consumption. The repeated output mode is convenient for moving image shooting. If the frame interval is set so that the AF signal is output when it is time to control the AF, the CPU 30 is notified when the AF signal is stored in the SDRAM 12, so a wasteful operation is performed. AF control starts. Useless power consumption is also eliminated. Further, when the AF signal is stored in the SDRAM 12, the phase difference calculation circuit 23 may automatically start the above-described calculation. If the frame interval in the repeated output mode is set to 1, the AF signal can be output every frame, so the flexibility is sufficiently high.

  By the way, it is expected that AF control is difficult only by the arrangement of the AF sensor array shown in FIG. In FIG. 10A, the array of S1 and the array of S2 are oriented in the horizontal direction. However, when the luminance of the subject hardly changes along the horizontal direction, the signal sequence between the signal sequence of S1 and the signal sequence of S2 It seems that it is not easy to obtain the phase difference. In that case, if the arrangement of the AF sensor array as shown in FIG. 10B is adopted, the phase difference may be easily obtained.

  However, the sequence as shown in FIG. 10 (b) is not suitable for CCD. In FIG. 10B, since the array of S1 and the array of S2 are oriented in the vertical direction, the photoelectric conversion cells of S1 and S2 included in each AF sensor array in the figure are arranged across several lines. become. Since the CCD reads out signals in line units, it takes a long time to read out signals from each AF sensor array. Since the AF signal cannot be read out quickly, the start of AF calculation is also delayed. In the case of a CCD, since it is not possible to read a signal by selecting a specific photoelectric conversion cell on the line, it takes a relatively long time to read the AF signal even with the arrangement as shown in FIG. In the case of the arrangement as shown in FIG. 10 (b), this disadvantage appears more prominently.

  In order to solve such a problem, a type capable of reading the solid-state imaging device 2 by the X-Y address method is adopted. Specifically, a solid-state image sensor such as a C-MOS sensor is used. Since the C-MOS sensor can also read out signals by selecting a specific photoelectric conversion cell, only the AF sensor signals can be read out together. When operating such a solid-state imaging device having an XY address type readout function in the above-described thinning mode, only the AF sensor signals are read first, and then the number of image signals reduced by thinning is reduced. I am trying to read it out.

  When the AF sensor is a horizontal array, for example, AF signals are collectively read in the order of arrows shown by solid lines in FIG. 12A, and when the AF sensor is a vertical array, for example, solid lines are shown in FIG. 12B. Read the AF signals together in the order of the arrows shown. In FIG. 12, broken arrows indicate the signal readout directions of S1 and S2 in each AF sensor array. As shown in FIGS. 11 and 4, even if pixels for image signals are included between S1 and S2 photoelectric conversion cells, signals can be read out by selecting only S1 or S2. The time it takes to read is negligible.

  Even if AF sensor arrays are arranged at five locations as shown in FIGS. 12 (a) and 12 (b), the total number of S1 and S2 photoelectric conversion cells is considered to be less than 1,000. With recent high-pixel solid-state image sensors, the number of horizontal pixels exceeds 3000, so if horizontal thinning is not performed in the thinning mode, all AF signals can be read during 1H period (HSync signal cycle). is there. If the AF signal is read out in the thinning mode of the solid-state imaging device, it is best to read it out in the vertical blanking period immediately after the VSync signal is output.

  That is, in FIGS. 9A and 9B, the AF signal is embedded in a line (not shown) immediately after the VSync signal and read. Since this part is a vertical blanking period, some lines are free and can be used as a period for reading out an AF signal. The merit of performing such reading is that the line including the AF signal is not inserted between the lines of the image signal read after that and the length of the vertical blanking period does not change. For this reason, the image processing operation is almost the same as that of the conventional imaging apparatus, which is one of the merits.

  As shown in FIG. 12C, an AF sensor composed of a horizontal array and an AF sensor composed of a vertical array can be mixed and arranged at a plurality of locations in the photographing screen. Even in the AF sensor arrangement as shown in FIG. 12C, the AF signals may be read together in the order of the solid line arrows in the same manner as those shown in FIGS. 12A and 12B. The broken-line arrows in the figure indicate the readout direction of the S1 and S2 arrays in each AF sensor. Note that either the S1 and S2 arrays or the horizontal and vertical arrays may be read first.

  As described above, in the thinning-out mode of the solid-state imaging device 2, since the AF signals are read together during the vertical blanking period immediately after the VSync signal is output, the AF / signal separation shown in FIG. The data is taken out by the circuit 21 and stored in the SDRAM 12 as before. Since the timing at which the AF signals are read is known, the image / AF signal separation circuit 21 is set so that they can be accurately extracted. When this is completed, the AF signal is read from the SDRAM 12 and input to the phase difference calculation circuit 23, where the phase difference between the two signals is obtained to control the AF. Since the AF signal is read out earlier, there is a merit that AF control is started earlier.

  On the other hand, when the vertical blanking period ends, reading of the line of the image signal starts. The image signal line is output from the image / AF separation circuit 21 and sent to the image processing circuit 22. The image signal is subjected to image processing to create through image data and moving image data. AF signal lines are not inserted and read between image signal lines, and it is not necessary to provide a period for stopping the image processing circuit 22 during image processing as before. Since the vertical blanking period and the effective image period in the thinning mode are almost the same as those of the conventional solid-state imaging device, the timing of the image processing is not severe.

  However, when the AF sensor includes an array in the vertical direction as shown in FIGS. 12B and 12C, new attention is required when performing image processing. When the AF sensor includes an array in the vertical direction, the lines including only the image signal have been greatly reduced. In such a solid-state imaging device, if the signal of the line including the AF sensor is completely removed, the number of lines of the through image and the moving image may be insufficient. Even if the image signal lines necessary to create a through image or video are secured, the lines including the AF sensor are thinned out together, so the image signal becomes non-uniform and the through image or video The image quality will deteriorate.

  Therefore, if the AF sensor includes an array in the vertical direction, the line including the AF sensor must also be used to create live view and video data. Even if AF sensors are arranged at five locations as shown in FIG. 12B, if attention is paid only to the vertical array, the total number of S1 and S2 photoelectric conversion cells included in one line is six at most. . As described above, if a single line includes only a small number of S1 and S2 photoelectric conversion cells, the line may be used to create a low-resolution image such as a through image or a moving image. However, in FIGS. 12A and 12C, the line including the array in the horizontal direction has few image signals and is not used for image processing. These lines are not read out in the thinning mode.

  On the other hand, the lines including the array in the vertical direction have many image signals and are used for image processing. In the thinning mode, these lines are read as many times as necessary. Since the AF signals are read together first, there is no signal read from the AF sensor when the image signal is read. When the vertical blanking period immediately after the VSync signal ends, the effective image period starts. As described above, only the lines used for image processing are read out during this period, so the image / AF shown in FIG. The signal separation circuit 21 may send all the lines in this period to the image processing circuit 22. Among them, the line including the vertical array must be handled separately when image processing is performed because there is no image signal in the S1 and S2 portions.

  In general, since the resolution of a through image or a moving image is low, pixels in the horizontal direction are thinned out. A solid-state imaging device having an X-Y address type readout function such as a C-MOS sensor can read out not only in vertical thinning but also in horizontal thinning in the thinning mode. Furthermore, pixels of the same color can be added and read out. By using such a readout function, it is possible to read out only the image signal by skipping the photoelectric conversion cells of S1 and S2, even for a line including a vertical array.

  If the signal is read out using such horizontal thinning, only the image signal is sent to the image processing circuit 22, so that a normal through image, moving image, or moving image is created even if the image processing is performed as it is. . Since the image signal that has been subjected to horizontal thinning in advance is read, horizontal thinning and horizontal resolution conversion in the image processing circuit 22 are also simplified. If horizontal thinning is not performed in advance using a solid-state imaging device, it must be handled by an image processing circuit. The simplest way of handling is to perform horizontal thinning similar to the horizontal thinning of the solid-state imaging device by the image processing circuit. What is necessary is just to make it discard the part of S1 and S2 in the case of this horizontal thinning.

  This horizontal thinning can be performed by the horizontal thinning circuit 51 shown in FIG. 2, but a signal including S1 and S2 flows in the preprocessing unit 22a of the image processing circuit 22, so that it is performed at the beginning of the preprocessing. Is desirable. In particular, when extracting an AE evaluation value or an AWB evaluation value, a signal that does not include the S1 and S2 portions is necessary, so it is essential to perform horizontal thinning at the beginning of preprocessing. If the horizontal thinning circuit 51 shown in FIG. 2 can be used for both pre-processing and post-processing, the number of gates of the image processing circuit 22 can be reduced.

  Next, a specific configuration of the phase difference calculation circuit 23 for achieving cost reduction without degrading performance will be described. Since the digital camera shown in FIG. 1 has a dedicated phase difference calculation circuit 23, the phase difference can be obtained quickly, and as a result, AF control can be performed at high speed. However, if this is configured with dedicated hardware, there will be inconvenience that it cannot be used for other purposes. Although the circuit scale increases, there are few opportunities for use, so the efficiency is poor.

  FIG. 13 shows an embodiment of the phase difference calculation circuit 23 that places importance on cost. In addition, the same code | symbol is attached | subjected with respect to the apparatus similar to the apparatus shown in FIG. 1, and it demonstrates centering around difference. The CPU 30 is a main CPU for controlling the entire camera, and the SRAM 71 is a memory that can be read / written at high speed from the main CPU 30. The first output of the image / AF signal separation circuit 21 is connected to the image processing circuit 22, and the second output is connected to the SRAM 71.

  When the solid-state imaging device 2 is operating in the thinning mode, the image signal separated by the image / AF signal separation circuit 21 is input to the image processing circuit 22 through the first output, where a through image or a moving image is input. Is created. On the other hand, the AF signal separated by the image / AF signal separation circuit 21 is directly stored in the SRAM 71 through the second output. The AF signal is read by the main CPU 30 via the CPU bus 35, and the phase difference between the two signals is first calculated by software processing of the main CPU 30. Next, the moving direction and the moving amount of the focusing lens are calculated from the phase difference information by software processing of the main CPU 30.

  When the main CPU 30 sends a command to the motor driver IC 4 shown in FIG. 1 according to the information, the focusing motor is driven, the focusing lens moves to the target position, and the subject is focused. Since the various calculations described above are performed by the software of the main CPU 30, it cannot be denied that the calculation speed is inferior to that of the dedicated phase difference calculation circuit 23 shown in FIG. Accordingly, the calculation of the AF signal is suppressed by storing the AF signal in the high-speed SRAM 71 shown in FIG. Normally, this SRAM 71 is mounted as an on-chip memory in the DSC engine 6 as with the main CPU 30.

  The SRAM 71 shown in FIG. 13 can be read / written from the main CPU 30 at a higher speed than the SDRAM 12 shown in FIG. The fact that the AF signal is directly stored in this SRAM means that it does not go through SDRAM or the like. This also contributes to speeding up AF calculation. Since the calculation speed changes according to the clock frequency of the main CPU 30, the performance is derived from this. Although the phase difference calculation circuit 23 shown in FIG. 1 has been deleted, since the SRAM 71 is newly provided, it may be felt that the circuit scale does not decrease. However, since the combination of the CPU 30 and the high-speed SRAM 71 can be used for various purposes, it has high flexibility and is much more useful than providing a dedicated phase difference calculation circuit 23.

  In addition, CPU cores for system-on-chip are often equipped with general-purpose high-speed SRAM in addition to cache memory (instructions and data). If this general-purpose SRAM is used as a high-speed memory for storing AF signals, additional hardware is required. Is not necessary. The storage capacity required to store the AF signal is very small. For example, in the arrangement shown in FIG. 12C, there are a total of 1024 photoelectric conversion cells of the AF sensor, and even if a 16-bit AF signal is output per photoelectric conversion cell, only 2 KB is occupied. CPU core general-purpose SRAMs are available in 4KB, 8KB, 16KB, etc., and these SRAMs can be used to store AF signals. However, when an on-chip general-purpose SRAM is used, the connection with the image / AF signal separation circuit 21 may need to be slightly changed.

  In the connection of FIG. 13, the SRAM 71 appears to have two ports for writing from the image / AF signal separation circuit 21 and for reading and writing from the main CPU 30. However, since the general-purpose SRAM described above generally has one read / write port, the output of the image / AF signal separation circuit 21 is sent to the CPU bus 35 in order to access it from both the image / AF signal separation circuit 21 and the main CPU 30. Then, the AF signal is stored in the SRAM 71 via the CPU bus 35. For this data transfer, DMA transfer by DMAC (not shown) may be used. As shown in FIG. 13, the combination of the main CPU 30 and the high-speed SRAM 71 achieves a highly flexible processing function and cost reduction at the same time.

  FIG. 14 shows another embodiment of the phase difference calculation circuit 23 in which performance and flexibility are emphasized. Compared with the phase difference calculation circuit shown in FIG. 13 described above, a coprocessor 72 is newly provided. The coprocessor 72 specializes in computation, and the main CPU 30 controls the entire camera. Similar to the phase difference calculation circuit shown in FIG. 13, a high-speed SRAM 71 is provided. This SRAM 71 can be read and written from the coprocessor 72 at high speed. The coprocessor 72 performs an operation according to a program. A coprocessor 72 such as a DSP (Digital Signal Processor) is suitable.

  In the phase difference calculation circuit 23 shown in FIG. 14, the SRAM 71 can be used as an integrated memory for programs and data. However, if the program memory and the data memory are separated, the execution speed (calculation speed) of the program increases. The embodiment follows that. The coprocessor 72 is assumed to incorporate a program memory (not shown), and the SRAM 71 is used exclusively for a data memory (work memory). The above-mentioned AF signal is stored in the SRAM 71. The coprocessor 72 is connected to the CPU bus 35, and the main CPU 30 downloads a program to the program memory via the CPU bus 35.

  When the main CPU 30 writes a command to a command register (not shown) via the CPU bus 35, the coprocessor 72 starts executing the program. When the program is executed, the SRAM 71 is used as a work memory (data memory). The coprocessor 72 includes a status register (not shown), and when the execution state of the program is written to this register, the main CPU 30 reads it. If an interrupt request is issued to the main CPU 30 when the coprocessor 72 writes to the status register, useless read operations are reduced.

  In addition, various registers are prepared, and the main CPU 30 and the coprocessor 72 communicate with the CPU bus 35 via these registers. With such a mechanism, the coprocessor 72 can perform various operations. When executing another calculation, the main CPU 30 downloads another program. The SRAM 71 is also connected to the CPU bus 35, and when the main CPU 30 prepares data necessary for calculation, the data is downloaded from here to the SRAM 71. When uploading calculation result data, read it from here. These downloads and uploads may be performed by DMA transfer using a DMAC (not shown).

  In the configuration shown in FIG. 14, the connection between the SRAM 71 and the coprocessor 72 is standard, and the connection between the main CPU 30 and the SRAM 71 is performed when data is downloaded or uploaded. As described above, the access of the coprocessor 72 and the main CPU 30 to the SRAM 71 is exclusive, and the SRAM 71 is deeply colored as a dedicated memory of the coprocessor 72. As the general-purpose SRAM of the main CPU 30, the one built in the CPU core itself is used. As described above, the subsystem including the coprocessor 72 and the SRAM 71 is a highly versatile computing unit.

  Next, AF calculation by the phase difference calculation circuit 23 shown in FIG. 14 will be described. The first output of the image / AF signal separation circuit 21 is connected to the image processing circuit 22, and the second output is connected to the SRAM 71. When the solid-state imaging device 2 is operating in the thinning mode, the image signal separated by the image / AF signal separation circuit 21 passes through the first output and enters the image processing circuit 22 where there is a through image or a moving image. Data is created. On the other hand, the AF signal separated by the image / AF signal separation circuit 21 is directly stored in the SRAM 71 through the second output.

  Although the dual port SRAM can be used in the phase difference calculation circuit 23 shown in FIG. 14, the second output of the image / AF signal separation circuit 21 is connected to the CPU bus 35 and the AF signal is written to the SRAM 71 therefrom. You can also. During this period, the coprocessor 72 is prevented from accessing the SRAM 71. When the AF signal is stored in the SRAM 71, the coprocessor 72 starts calculation in response to a command from the main CPU 30. The main CPU 30 downloads the program to the coprocessor 72 in advance.

  The coprocessor 72 first calculates the phase difference between the two signals, and then determines the moving direction and amount of the focusing lens from the phase difference. When these operations are completed, the coprocessor 72 writes information indicating the completion of the operation in the status register. As a result of this writing, an interrupt request is issued to the main CPU 30, and when the main CPU 30 reads the status register in response thereto, it can be seen that the calculation is completed. The calculation result stored in the SRAM 71 is read from the CPU bus 35. When the main CPU 30 sends a command to the motor driver IC 4 shown in FIG. 1 according to the information, the focusing motor is driven and the focusing lens moves to the target position. Focus on the subject.

  Since the AF calculation is performed using the coprocessor 72 specializing in the calculation, the calculation time is short and the AF control is performed at high speed. The coprocessor 72 and the SRAM 71 are provided, so that the circuit scale is slightly increased. However, the coprocessor subsystem is extremely versatile compared to a dedicated phase difference arithmetic circuit. That is, the coprocessor subsystem is frequently used and therefore has high cost performance. When the coprocessor subsystem is used, there is an advantage that the main CPU 30 can perform other processes in parallel. Thus, high cost performance is achieved in the digital camera of the embodiment.

  By the way, in the digital camera shown in FIG. 1 and the image processing circuit 22 shown in FIG. 2 according to the above-described embodiment, two dedicated hardwares, ie, a phase difference calculation circuit 23 and an AF signal interpolation circuit 61 are used. Since each circuit performs only a predetermined process, the utilization efficiency is poor. Therefore, if these processes are performed by one circuit, cost performance is improved.

  Therefore, AF signal interpolation is performed using the coprocessor subsystem shown in FIG. Since the two circuits are integrated into one, the overall circuit scale is reduced. Faster processing is possible by using a coprocessor subsystem that specializes in computing. When performing the interpolation processing, the signals S1 and S2 shown in FIG. 11 are read from the SDRAM 12 together with the surrounding image signals and stored in the SRAM 71. In the configuration shown in FIG. 14, this data is downloaded from the CPU bus 35. This data transfer may be performed by DMA transfer using a DMAC (not shown). The coprocessor 72 interpolates S1 and S2 signals using surrounding image signals. When a DSP is used for the coprocessor 72, bicubic interpolation can be performed at high speed.

  When the interpolation processing is completed, the coprocessor 72 subsequently obtains an AWB correction value based on the interpolated image signal, and corrects the AWB evaluation value extracted at the time of the preprocessing with the AWB correction value. According to the image processing circuit 22 shown in FIG. 2, since the AWB evaluation value is read from the CPU bus 35, the coprocessor 72 cannot directly read the AWB evaluation value. The main CPU 30 downloads the AWB evaluation value of the corresponding window to the SRAM 71 before the correction starts. When the correction of the AWB evaluation value is completed, the interpolated image signal is read from the SRAM 71 and stored in the SDRAM 12. This data is also uploaded from the CPU bus 35. DMA transfer can also be used for this.

  As shown in FIGS. 12A, 12B, and 12C, when AF sensor arrays are arranged at a plurality of locations in the shooting screen, the above-described interpolation processing and correction processing are performed for all of them. . When these processes are completed, the coprocessor 72 executes the AWB algorithm based on the corrected AWB evaluation value, and determines the R gain and B gain for WB adjustment. The AWB evaluation value necessary for the AWB algorithm is downloaded to the SRAM 71 by the main CPU 30. This download may be performed by DMA transfer.

  When the R gain and the B gain are determined, the main CPU 30 reads them from the SRAM 71 and sets them in the WB adjustment circuit 52 shown in FIG. Finally, when the interpolated image signal is input to the image processing circuit 22 shown in FIG. 14 and post-processing is performed, YCbCr data of the captured still image is created. While the coprocessor subsystem is performing the above processing, the main CPU 30 can perform another processing in parallel. Since data is downloaded to the SRAM 71 or data is uploaded from the SRAM 71, data frequently flows through the CPU bus 35. For this reason, the operation of the main CPU 30 is hindered. If the SRAM 71 can be connected to both the CPU bus 35 and the image bus 34, the CPU bus 35 becomes free and the main CPU 30 becomes easy to operate. Since the flow of image data and the processing of the main CPU 30 are generally asynchronous, if the bus is divided in this way, each operation is performed smoothly.

  As described above, since the phase difference calculation circuit 23 and the AF signal interpolation circuit 61 are integrated into one, the cost performance is further improved.

  As described above, according to one embodiment, in the thinning readout mode, signals from all AF sensor cells are read out in addition to signals from some of the pixels for imaging. When performing AF control in the mode, there is no need to switch the mode from the thinning readout mode to another mode, and the control sequence can be simplified.

  Further, according to one embodiment, the imaging pixels and the AF sensor cells are expanded in two dimensions having an X-axis address and a Y-axis address, respectively. Since the readout circuit is equipped with a readout circuit, the readout circuit reads out the signals of all AF sensor cells at the time of executing the readout readout mode, and then reads out the signals by thinning out some of the pixels for imaging. In addition to obtaining the AF signal early, the readout timing of the image signal can be made almost the same as that of the conventional imaging apparatus.

  According to one embodiment, the AF sensor cell is configured to have a photoelectric conversion unit that receives one or both of a pair of light beams that have passed through different regions of the pupil of the optical system, so that output from the array of AF sensor cells is performed. Using this signal, quick focus detection by the pupil division type phase difference detection method can be executed.

  According to one embodiment, when the thinning readout mode is executed, a signal output from the imaging pixel and a signal output from the AF sensor cell are separated, and an image is generated by the separated imaging pixel signal. At the same time, the focus adjustment state of the optical system is detected based on the signal from the separated AF sensor cell, and the focus adjustment of the optical system is performed based on the focus adjustment state. However, AF control can be performed in parallel.

  According to one embodiment, a signal for generating an image at the position of the AF sensor cell is obtained by interpolation based on the signal of the imaging pixel around the AF sensor cell, and based on the signal of the imaging pixel. The evaluation value used for white balance adjustment is calculated, and the evaluation value used for white balance adjustment is corrected based on the image sensor pixel signal and the AF sensor cell position signal obtained by interpolation when the all-pixel readout mode is executed. As a result, white balance accuracy can be improved.

  According to one embodiment, it is possible to set whether or not to separate the signal of the AF sensor cell, and when it is set to be separated, only one of the thinning readouts repeatedly performed in the thinning readout mode. Since either the mode for separating the AF sensor cell signal or the mode for separating the AF sensor cell signal every predetermined number of thinning-out readouts can be selected, the AF control can be performed flexibly and easily.

  According to one embodiment, the focus adjustment unit has a CPU and a high-speed readable / writable memory, and stores the signal of the separated AF sensor cell directly in the memory and the memory when executing the thinning readout mode. Since the CPU executes the focus detection calculation and the focus adjustment control based on the received signal, a highly flexible processing function can be realized and the cost can be reduced.

  According to one embodiment, the focus adjustment unit further includes a coprocessor dedicated to computation, and when executing the thinning readout mode, the signal of the separated AF sensor cell is directly stored in the memory and stored in the memory. The coprocessor executes focus detection calculation based on the signal, and the CPU executes focus adjustment control based on the focus detection calculation result, so that the calculation speed can be improved and a flexible processing function can be realized. High cost performance can be achieved.

  According to one embodiment, since the interpolation calculation and the evaluation value correction calculation are executed using a coprocessor and a high-speed readable / writable memory, the dedicated hardware can be reduced and the cost can be suppressed.

2; Image sensor, 5; Timing generator, 6; DSC engine, 21; Image / AF signal separation circuit, 22; Image processing circuit, 23; Phase difference calculation circuit, 30; CPU, 61; ; AWB correction value circuit 71; SRAM 72; Coprocessor

Claims (8)

Imaging in which a plurality of focus detection pixels that output a signal for focus detection are arranged on a part of an image sensor in which a plurality of imaging pixels that output a signal for generating an image are two-dimensionally arranged An element,
The imaging pixels and the focus detection pixels are expanded in a two-dimensional shape having an X-axis address and a Y-axis address, respectively, and include a readout circuit that reads out signals for each pixel by specifying the X and Y addresses,
The readout circuit includes an all-pixel readout mode for reading out signals from all the imaging pixels and all the focus detection pixels, and a thinning-out readout mode for reading out signals by thinning out some of the imaging pixels. , has a,
In the thinning readout mode, in addition to the signals of some of the pixels for imaging, all the signals of the focus detection pixels are read out within one frame readout period ,
When the thinning readout mode is executed, signals of all the focus detection pixels are read out during the vertical blanking period in the one frame, and one of the imaging pixels is read out after the vertical blanking period in the one frame. An image pickup device that reads out a signal by thinning out pixels of a part . The imaging device according to claim 1 ,
The focus detection pixel includes a photoelectric conversion unit that receives one or both of a pair of light beams that have passed through different regions of the pupil of the optical system. The image sensor according to claim 1 or 2 ,
A signal separation means for separating a signal output from the imaging pixel and a signal output from the focus detection pixel when the thinning readout mode is executed;
Image generating means for generating an image based on signals of the imaging pixels separated by the signal separating means;
Characterized in that it comprises, a focusing means for performing focus adjustment of the optical system by detecting the focusing state of the optical system, based on the focusing state by a signal of the focus detection pixels separated by the signal separating means An imaging device. The imaging device according to claim 3 .
Interpolation means for obtaining, by interpolation, a signal for generating an image at the position of the focus detection pixel based on the signal of the imaging pixel around the focus detection pixel;
White balance evaluation value calculating means for calculating an evaluation value used for white balance adjustment based on a signal of the imaging pixel;
An evaluation value correction unit that corrects an evaluation value used for the white balance adjustment based on a signal of the imaging pixel and a signal of the focus detection pixel position obtained by the interpolation unit when the all-pixel reading mode is executed; An imaging apparatus comprising: The imaging device according to claim 3 .
The signal separation means can set whether or not to separate the signals of the focus detection pixels. When the separation is set, only one of the thinning readouts repeatedly performed in the thinning readout mode. An image pickup apparatus, wherein either one of a mode for separating the signal for the focus detection pixel and a mode for separating the signal for the focus detection pixel at every predetermined number of decimation readouts can be selected. The imaging device according to any one of claims 3 to 5 ,
The focus adjustment unit includes a CPU and a memory capable of reading and writing at high speed, and directly stores the signal of the focus detection pixel separated by the signal separation unit in the memory when the thinning readout mode is executed. An image pickup apparatus, wherein the CPU executes focus detection calculation and focus adjustment control based on a signal stored in the memory. The imaging device according to claim 6 ,
The focus adjustment unit further includes a coprocessor dedicated to calculation, and stores the signal of the focus detection pixel separated by the signal separation unit directly in the memory when the thinning readout mode is executed, and the memory An image pickup apparatus, wherein the coprocessor executes a focus detection calculation based on a signal stored in the image, and the CPU executes a focus adjustment control based on the focus detection calculation result. The imaging apparatus according to claim 7 ,
The imaging apparatus, wherein the interpolation unit and the evaluation value correction unit execute an interpolation calculation and an evaluation value correction calculation using the coprocessor and the memory.

Images