JP2006041867A - Image processing method and image processor, imaging apparatus, and timing controller - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus having an electronic zoom function in which problems concerned with the circuit scale, cost, or power consumption, picture quality, etc. are solved. <P>SOLUTION: A dummy period is provided which does not contribute to readout processing and writing processing by extending cycles of a horizontal synchronizing signal supplied to a drive control section 96 (12c) at the same rate with the magnification of electronic zooming when pixel information is read out of a solid-state imaging device 10 in a horizontal scanning direction and the read-out pixel information is written in a line buffer memory 312. Without reference to the cycles of the horizontal synchronizing signal SHD, cycles of a pixel clock driving individual pixels of the solid-state imaging device 10 are made substantially as long as usual. The dummy output period is provided to secure a period in which interpolation processing is performed, and consequently readout processing for a pixel signal from the solid-state imaging device 10 and write processing to a line buffer memory 312, and horizontal interpolation processing in a horizontal interpolation processing part 314 and vertical interpolation processing in a vertical resolution conversion processing part 320 are performed simultaneously in parallel. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing method, an image processing apparatus, an imaging apparatus, and a timing control apparatus that controls drive timing of imaging devices that constitute the imaging apparatus. More specifically, the present invention relates to a mechanism for electronically performing image enlargement conversion processing.

  A resolution conversion process is known in which an image is enlarged or reduced by electronic interpolation processing on image data using a combination with a digital image signal processing LSI (Large Scale Integrated Circuit).

  For example, in an imaging device that captures an image using an imaging device (imaging device) such as a CCD (Charge Coupled Device) imaging device or a CMOS (Complementary Metal-Oxide Semiconductor) imaging device, a digital image signal processing LSI is used. Thus, it is possible to configure the camera function centered on electronic components, and a mechanism for performing electronic zoom processing using the resolution conversion processing described above without using an optical zoom system is considered. In this case, a mechanism for realizing a zoom function of a captured image using only a solid-state imaging device and a digital image signal processing LSI using a frame memory (or field memory) has been proposed (see, for example, the prior art in Patent Document 1). .

JP 2000-295530 A

  FIG. 11 is a diagram for explaining a conventional configuration example of a mechanism for realizing an electronic zoom function using a frame memory. 12 and 13 are diagrams for explaining an interpolation method using a frame memory.

  As illustrated in FIG. 11, the imaging apparatus 900 includes a solid-state imaging element 910 that is a pixel under the control of a drive control unit 996 that operates in conjunction with a zoom control unit (device control functional part) that controls an electronic zoom operation. The signal is read and supplied to the subsequent image signal processing unit 966. In the figure, functional parts such as correlated double sampling and AD conversion processing are omitted.

  Pixel data that has undergone predetermined signal processing by the image signal processing unit 966 is temporarily stored in the frame memory 967. Note that the frame memory 967 is a memory for recording all pixel data of one frame. However, for example, when image data of double zoom is desired, only a partial area of the solid-state image sensor 910 is read and stored.

  Next, pixel data is read from the frame memory 967 at a predetermined timing in accordance with an instruction from a zoom control unit (memory control functional part) 992 for controlling the electronic zoom operation, and the pixel data is passed to the subsequent interpolation processing unit 998.

  The interpolation processing unit 998 provides a gap corresponding to the zoom magnification between the pixels under the control of the zoom control unit (interpolation process control functional part) 992 that controls the electronic zoom operation, and an interpolation signal is obtained by signal processing in the gap. To construct an image of the same size as the original image.

  For example, vertical filter processing is performed using pixel data of multiple lines adjacent to the pixel of interest to be interpolated according to the zoom magnification to make the number of lines the desired number of lines, that is, the number of vertical pixels is converted to the desired number of pixels In addition to performing vertical resolution conversion processing, the horizontal filter processing is performed using the pixel data of the same line adjacent to the target pixel to be interpolated according to the zoom magnification to convert the number of horizontal pixels to the desired number of pixels. By doing so, resolution conversion processing in the horizontal direction is performed. By doing so, an electronic zoom processed image output is obtained from the interpolation processing unit 998.

  Here, the frame memory 967 is used in a drifting system in which a pixel signal is output as it is regardless of the zoom magnification from the solid-state image sensor 910. This is because in order to perform the reading process while interpolating, a mechanism that covers the time lag for the line interpolation is required (see FIG. 13).

  In other words, during enlargement zoom processing, an image signal sequentially read from the imaging device is sequentially stored in the line memory, and a predetermined number of line data is read in order to generate line data that is insufficient during enlargement (vertical resolution conversion). Therefore, it is necessary to perform vertical interpolation processing. If this is to be done in real time, first, a large-scale memory such as a frame memory that holds line data for the zoom target in the vertical direction is required. For example, since the pixel signal continues to enter the image signal processing unit 966 from the solid-state image sensor 910, a memory for 1/2 line (480 lines) of effective pixels is required at the time of double zooming.

  For example, FIG. 12A is a map in which the effective area captured by the solid-state imaging device 910 is two-dimensionally expressed. The solid-state imaging device 910 of this example is an imaging device of about 1.3 M pixels with a horizontal direction of 1280 pixels and a vertical direction of 960 lines.

  In the frame memory 967, pixel data is sequentially read out and stored in the pixel arrangement order of the solid-state imaging device 910, and at the time of electronic zooming, at least in the use area in the vertical direction of the image area used at the time of zooming according to the zoom magnification. Corresponding pixel data is stored. As an example, as shown in FIG. 12B, at the time of double zooming, only the pixel data of 480 lines (for example, the central portion in the vertical direction), which is half of the normal 960 lines, is stored in the frame memory 967. Remember.

  Further, as shown in FIG. 12C, the interpolation processing unit 998 reads out pixel data of a plurality of lines adjacent to the target pixel to be interpolated according to the zoom magnification from the line memory, and converts the pixel data of the plurality of lines. The vertical filter process using the interpolation process is performed. That is, the number of vertical pixels is converted to the desired number of pixels using raw pixel data of adjacent lines adjacent to the target pixel to be interpolated according to the zoom magnification or horizontal interpolation data obtained by horizontal interpolation processing. As a result, vertical resolution conversion processing is performed.

  At this time, as shown in FIG. 13, in order to perform vertical interpolation by sequentially using the pixel data read from the solid-state image sensor 910, the data is read from the solid-state image sensor 910 for interpolation processing of the interpolation line (difference line). A means for storing pixel data for a plurality of lines is required.

  Next, the interpolation processing unit 998 temporarily stores the pixel data (line data) after the resolution conversion processing in the vertical direction in a line memory (not shown). That is, the pixel signal of the area (for example, the center of the screen) to be used during electronic zooming is once stored in the frame memory 967 from the solid-state image sensor 910 and transferred to the interpolation processing unit 998 while managing the timing, and the interpolation processing unit 998. The horizontal interpolation process is performed by the interpolation process.

  For example, as shown in FIG. 12D, the interpolation processing unit 998 performs interpolation processing on only a region of 640 pixels (for example, the central portion in the horizontal direction) that is half of all 1280 pixels in the horizontal direction. Perform horizontal filtering using. That is, the horizontal resolution conversion processing is performed by converting the number of horizontal pixels into a desired number of pixels using raw pixel data on the same line adjacent to the target pixel to be interpolated according to the zoom magnification.

  However, such conventional electronic zoom processing requires a large-scale storage means such as a frame memory having a large circuit scale for storing pixel data for one frame or the number of lines corresponding to the zoom magnification. There is also a need for a digital processing LSI (interpolation processing unit 998) that performs pixel interpolation, resulting in an increase in hardware and a large circuit configuration. In addition, it is necessary to realize difficult control that creates a gap for interpolation in the readout control of pixel data from a large-scale storage means such as a frame memory, and the circuit configuration of the zoom control unit is also complicated and large-scale I have to be.

  As a whole, the system becomes a large-scale, high-power-consumption or high-cost system, so a solid-state imaging device that performs electronic zoom processing with a single sensor chip (imaging device), for example, a mobile phone that requires a low-cost or small-scale configuration There is a problem that it is difficult to introduce as an electronic zoom when realizing a camera for a camera or a mobile device.

  As a method for solving this problem, for example, a mechanism for performing an electronic zoom without using a large-capacity storage means such as a frame memory has been proposed (for example, see Patent Document 2).

JP 2001-197348 A

  However, in the mechanism described in Patent Document 2, the reading drive of the imaging device is the same as that during normal zooming (there is no change to sensor driving), and the period of the horizontal synchronization signal that reads out the pixel signal from the imaging device is enlarged. However, it is necessary to set the vertical interpolation calculation processing cycle fast.

  For this reason, there is a problem that power consumption increases during zooming. In addition, when the zoom control unit and the interpolation processing unit are configured integrally with the output imaging device in order to perform electronic zoom processing with a single sensor chip (imaging device), the cycle is faster than the cycle of the horizontal synchronization signal for reading out pixel signals. Noise caused by the pixel clock for the vertical interpolation calculation processing becomes a problem. For example, when an imaging device is built on the same chip as a zoom processing digital LSI manufactured by a CMOS process, the image quality may deteriorate in the enlarged zoom mode compared to the normal mode.

  As described above, the conventional electronic zoom mechanism is not sufficient in terms of the size, cost, power consumption, image quality, etc. of the image pickup device, and is not particularly applicable to image pickup devices for mobile use. There is a problem that it is difficult.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a resolution conversion or electronic zoom mechanism that can solve problems such as circuit scale, cost, power consumption, and image quality.

  The image processing method according to the present invention controls the writing and reading of pixel information to and from the storage unit in a relatively high-speed first scanning direction (typically the horizontal scanning direction), and also reads out pixel information read from the storage unit Is an image processing method for performing resolution conversion processing of an image by performing interpolation processing in a relatively slow second scanning direction (typically the vertical scanning direction) that is different from the first scanning direction. Thus, during resolution conversion processing for enlarging the image size in the second scanning direction, a dummy period that does not contribute to writing of pixel information to the storage unit is provided when writing pixel information to the storage unit in the first scanning direction. I made it.

  An image processing apparatus according to the present invention is an apparatus such as an IC (Integrated Circuit) suitable for carrying out the image processing method according to the present invention, and a storage unit for storing pixel information; A conversion control unit that controls writing and reading of pixel information to and from the storage unit in the first scanning direction and an operation of interpolation processing, and interpolation processing in the second scanning direction using the pixel information read from the storage unit. And a resolution conversion processing unit that performs image resolution conversion processing.

  An imaging apparatus according to the present invention is an imaging apparatus suitable for carrying out the image processing method according to the present invention, and includes a drive control unit that controls reading pixel information from the imaging device, and an imaging device. A storage unit for storing the read pixel information, a conversion control unit for controlling writing and reading of the pixel information to and from the storage unit in the first scanning direction and controlling the operation of the interpolation process, and a pixel read from the storage unit And a resolution conversion processing unit that performs resolution conversion processing of an image by performing interpolation processing in the second scanning direction using the information.

  Here, in any of the image processing apparatus and the imaging apparatus according to the present invention, the conversion control unit does not contribute to writing the pixel information to the storage unit when writing the pixel information to the storage unit in the first scanning direction. The storage unit is controlled so that a dummy period is provided, and the resolution conversion processing unit performs the interpolation process in the second scanning direction substantially using the dummy period, thereby changing the image size to the second size. Enlarge in the scanning direction. In the imaging apparatus, the drive control unit is controlled so that a dummy period that does not contribute to reading of pixel information from the imaging device is provided.

  Here, “substantially using the dummy period” does not necessarily mean that the interpolation process is performed in the dummy period, but one process in the first scanning direction in which the dummy period is provided. Any interpolation processing may be performed within the period.

  The timing control device according to the present invention is a device for controlling the timing of the imaging operation of the imaging device, and is a scanning reference signal for reading out pixel information from the imaging device in the first scanning direction, and the first scanning A reference signal supply unit is provided that provides a dummy period that does not contribute to reading of pixel information from the imaging device by supplying a scanning reference signal having a longer readout processing cycle in the direction unit than that in the normal state to the drive control unit.

  The invention described in the dependent claims defines further advantageous specific examples of the image processing method, the image processing apparatus, the imaging apparatus, and the timing control apparatus according to the present invention.

  For example, a specific configuration for providing the dummy period is realized by making the writing process cycle of the first scanning direction unit for writing pixel information to the storage unit in the first scanning direction longer than usual. May be. In this case, it is preferable to extend the writing process cycle in the first scanning direction at the same magnification as that for the normal magnification. By doing so, the reading process and the interpolation process of the pixel signal from the imaging device can be performed simultaneously and no wasteful time is generated.

  According to the present invention, at the time of resolution conversion processing for enlarging the image size in the second scanning direction, when reading pixel information from the imaging device in the first scanning direction or writing this pixel information to the storage unit, A dummy period that does not contribute to the reading process or the writing process is provided.

  In this way, when the reading process and the writing process are continuously performed for one screen, a free time for performing the zoom enlargement interpolation process can be made. A period that does not contribute to output of pixel information or writing to the storage unit can be used as a timing for generating and embedding interpolation pixel information in the second scanning direction. As a result, for one screen, the pixel signal reading process from the imaging device, the writing process to the storage unit, and the interpolation process can be performed simultaneously (in real time).

  Control of the storage unit only needs to control writing and reading of pixel information in the first scanning direction (typically the horizontal scanning direction), and does not require complicated control. Since the control only needs to be performed in the first scanning direction, a simple configuration such as a line memory can be used as the configuration of the storage unit.

  Therefore, the circuit scale is large, and the electronic zoom function can be realized at a low cost without using an expensive frame memory. The memory of the camera system can be reduced and the cost contribution is great. In particular, in the case of a mobile camera system, downsizing is required, but the present invention can also contribute to downsizing of the system. That is, since an electronic zoom system can be constructed at a low cost, it is suitable as an electronic zoom for realizing a camera for a mobile phone or a mobile device.

  Further, since a pixel clock having the same period can be used in the pixel signal reading process, the writing process to the storage unit, or the interpolation process regardless of the zoom magnification, as described in Patent Document 2, from the imaging device. It is not necessary to set the cycle of vertical interpolation calculation processing (vertical resolution conversion processing by data processing) faster than the cycle of the horizontal sync signal for reading out the pixel signal. It is advantageous.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overall configuration of digital still camera; CCD type>
FIG. 1 is a schematic configuration diagram showing a first embodiment of an imaging apparatus (camera system) according to the present invention. The image pickup apparatus according to the first embodiment is an example of the solid-state image pickup device 10, for example, a CCD solid-state image pickup device 11 capable of reading all pixels by an interline transfer (IT) method, an image pickup lens 50 for capturing an optical image of a subject Z, and A camera module 3 having a drive control unit 96 for driving the CCD solid-state imaging device 11 and a video signal are generated based on an imaging signal obtained by the camera module 3 and output to a monitor or an image is stored in a predetermined storage medium. The digital still camera 1 includes a main body unit 4. In addition, you may make it comprise as an imaging device module with the form which integrated the camera module 3 and the main body unit 4. FIG.

  The digital still camera 1 is specifically applied as a camera that can capture a color image during a still image capturing operation using a frame readout method. Further, the frame readout method is not limited to the general two-field readout method by combining with the CCD solid-state imaging device 11, but can be of various fields such as three fields, four fields, five fields, or more. The reading method is applicable. In addition to the still image capturing mode, a moving image capturing mode is also prepared at a frame rate close to 30 frames / second (for example, 10 frames / second or more) using thinning-out reading.

  The drive control unit 96 in the camera module 3 receives a timing signal generation unit 40 that generates various pulse signals for driving the CCD solid-state imaging device 11 and a pulse signal from the timing signal generation unit 40. A driver (drive unit) 42 for converting the drive pulse to drive the CCD solid-state image sensor 11 and a drive power supply 46 for supplying power to the CCD solid-state image sensor 11 and the driver 42 are provided.

  The solid-state imaging device 2 is configured by the solid-state imaging device 10 (CCD solid-state imaging device 11 in this example) and the drive control unit 96 in the camera module 3. The solid-state imaging device 2 is provided as a CCD solid-state imaging device 11 and a drive control unit 96 arranged on a single circuit board or formed on a single semiconductor substrate. Is good.

  The CCD solid-state imaging device 11 is not shown in its configuration example. For example, a sensor unit (photosensitive unit) including a photodiode as an example of a light receiving element corresponding to a pixel (unit cell) on a semiconductor substrate. A large number of photocells) arranged in a two-dimensional matrix in the horizontal (row) direction and vertical (column) direction. These sensor units convert incident light incident from the light receiving surface into signal charges having a charge amount corresponding to the amount of light, and accumulate the signal charges.

  As a color image capturing application, the sensor unit is provided with a color filter of any one of color separation filters composed of a combination of color filters on a light receiving surface such as a photodiode on which light is incident. As an example, sensor units (unit pixels) arranged in a square lattice using three colors of red (R), green (G), and blue (B) using a basic color filter of a so-called Bayer array. They are arranged so as to correspond to color filters (primary color filters). Or it is good also as a thing of the complementary color filter structure which combined four colors, cyan (C), magenta (M), yellow (Y), and green (G).

  When the signal processing is configured to perform primary color signal processing, if the primary color filter is used, red (R), green from the imaging signal (a combination of pixel signals of a plurality of colors) obtained by the CCD solid-state imaging device 11 is used. The primary color separation unit that separates the primary color signals of (G) and blue (B) can be omitted.

  The CCD solid-state imaging device 11 has a vertical CCD (V register unit, vertical transfer unit) provided with a plurality of vertical transfer electrodes corresponding to 6-phase or 8-phase drive for each vertical column of the sensor unit. The transfer direction of the vertical CCD is the vertical direction in the figure, and a plurality of vertical CCDs 13 are arranged in this direction. Further, a read gate (ROG) is interposed between the vertical CCD and each sensor unit, and a channel stop is provided at a boundary portion of each unit cell. An imaging area is configured by a plurality of vertical CCDs that are provided for each vertical column of the sensor units and vertically transfer signal charges read from the sensor units by the read gate unit.

  The signal charge accumulated in the sensor unit is read out to the vertical CCD by applying a drive pulse corresponding to the readout pulse XSG to the readout gate unit. The vertical CCD is driven to transfer by drive pulses φV1 to φV6 (φV8) based on 6-phase (or 8-phase) vertical transfer clocks V1 to V6 (or V8), and the read signal charges are stored in one horizontal blanking period. The unit sequentially transfers the portions corresponding to one scanning line (one line) in the vertical direction. This vertical transfer for each line is called a line shift.

  The CCD solid-state imaging device 11 includes a horizontal CCD (H register) extending in a predetermined (for example, left and right) direction adjacent to each transfer destination side end of a plurality of vertical CCDs, that is, the vertical CCD in the last row. Part, horizontal transfer part) is provided for one line. The horizontal CCD is driven to transfer by drive pulses φH1 and φH2 based on, for example, two-phase horizontal transfer clocks H1 and H2, and the signal charge for one line transferred from a plurality of vertical CCDs is transferred after the horizontal blanking period. The images are sequentially transferred in the horizontal direction during the horizontal scanning period. For this reason, a plurality of (two) horizontal transfer electrodes corresponding to two-phase driving are provided.

  For example, a charge / voltage conversion unit having a floating diffusion amplifier (FDA) configuration is provided at the end of the transfer destination of the horizontal CCD. The charge / voltage converter sequentially converts the signal charges transferred horizontally by the horizontal CCD into voltage signals and outputs the voltage signals. This voltage signal is derived as a CCD output (Vout) corresponding to the amount of incident light from the subject. The interline transfer type CCD solid-state imaging device 11 is configured as described above.

  The processing system of the digital still camera 1 is roughly composed of an optical system 5, a signal processing system 6, a recording system 7, a display system 8, and a control system 9. Needless to say, the camera module 3 and the main unit 4 are accommodated in an exterior case (not shown), and an actual product (finished product) is finished.

  The optical system 5 includes a shutter 52, a lens 54 for condensing a light image of a subject, an imaging lens 50 having a diaphragm 56 for adjusting the light amount of the light image, and photoelectrically converting the collected light image into an electric signal. It comprises a CCD solid-state imaging device 11 for conversion. The light L1 from the subject Z passes through the shutter 52 and the lens 54, is adjusted by the diaphragm 56, and enters the CCD solid-state imaging device 11 with appropriate brightness. At this time, the lens 54 adjusts the focal position so that an image composed of the light L <b> 1 from the subject Z is formed on the CCD solid-state imaging device 11.

  The signal processing system 6 includes an amplification amplifier that amplifies an analog imaging signal from the CCD solid-state imaging device 11, a CDS (Correlated Double Sampling) circuit that reduces noise by sampling the amplified imaging signal, and the like. A preamplifier unit 62, an analog signal output from the preamplifier unit 62, an A / D (Analog / Digital) converter unit 64 that converts the analog signal into a digital signal, and a predetermined image processing on the digital signal input from the A / D converter unit 64. The image signal processing unit 66 is configured as a camera signal processing LSI (Large Scale Integrated Circuit) configured by a DSP (Digital Signal Processor) to be applied.

  For example, the image signal processing unit 66 separates and synchronizes primary color signals of red (R), green (G), and blue (B) from complementary color imaging data, and primary color imaging data (R, G). , B pixel data), vertical stripe noise correction processing for correcting vertical stripe noise components caused by smear phenomenon and blooming phenomenon, WB control processing for controlling white balance (WB) adjustment, Gamma correction processing for adjusting the degree of gradation, resolution conversion processing for electronically enlarging or reducing an image by image data processing (that is, electronic zoom processing), or luminance data (Y) instead of using an optical zoom lens And YC signal generation processing for generating color data (C). It also has a function of generating a synchronization signal indicating a reference of a timing pulse for driving the CCD solid-state imaging device 11. Note that components unique to the present embodiment relating to the electronic zoom process (resolution conversion process) will be described in detail later.

  The image signal processing unit 66 configured by the DSP can be configured such that all processing of each functional part is performed by a digital processing circuit using dedicated hardware, and part of these functional parts is performed by software processing. It can also be set as the structure to perform.

  Although the mechanism for performing predetermined processing by software can flexibly cope with parallel processing and continuous processing, the processing time becomes longer as the processing becomes complicated, so that a reduction in processing speed becomes a problem. On the other hand, it is possible to construct an accelerator system with a higher speed by using a hardware processing circuit. Even if the processing is complicated, the accelerator system can prevent a reduction in processing speed and can obtain a high throughput.

  The recording system 7 includes a memory (recording medium) 72 that can be attached to and detached from a device such as a flash memory that stores image data, and the image data processed by the image signal processing unit 66 is encoded (compressed) into the memory 72. A CODEC (Compression / Decompression) 74 is recorded, read out, decoded (expanded), and supplied to the image signal processing unit 66.

  The display system 8 includes a D / A (Digital / Analog) conversion unit 82 that converts the image signal processed by the image signal processing unit 66 into an analog, and a liquid crystal that functions as a finder by displaying an image corresponding to the input video signal. (LCD; Liquid Crystal Display), a video monitor 84 made of organic EL (Electro Luminescence), and the like, and a video encoder 86 that encodes an analog image signal into a video signal suitable for the video monitor 84 in the subsequent stage. ing. Note that the arrangement of the D / A converter 82 and the video encoder 86 may be reversed, and the encoding process may be performed digitally. In this case, the video encoder 86 can be taken into the image signal processing unit 66.

  First, the control system 9 includes a synchronization signal generation unit 41 that generates a reference signal such as a horizontal synchronization signal SHD and a vertical synchronization signal SVD that defines a reference for driving a CCD solid-state image sensor 11 that is an example of the solid-state image sensor 10, and a digital signal. And a central control unit 92 including a CPU (Central Processing Unit) for controlling the entire still camera 1. The synchronization signal generation unit 41 generates a horizontal synchronization signal SHD and a vertical synchronization signal SVD having a predetermined cycle based on an instruction from the central control unit 92 and supplies the generated signals to the timing signal generation unit 40 on the solid-state imaging device 10 side. The synchronization signal generation unit 41 and the central control unit 92 constitute a reference signal supply unit and a timing control device according to the present invention.

  The control system 9 includes a ROM (Read Only Memory) 93a that is a read-only storage unit, a RAM (Random Access Memory) 93b that is an example of a volatile storage unit that can be written and read at any time, and a nonvolatile memory. An RAM (described as NVRAM) 93c, which is an example of a storage unit, and an EEPROM (Electrically Erasable and Programmable ROM) 93d, which is an example of a non-volatile storage unit that stores individual device data such as white spot position information and various adjustment data A storage unit (memory unit) 93 is provided. Various memories in the storage unit 93 excluding the central control unit 92 such as a CPU and the EEPROM 93d can be taken into the image signal processing unit 66 constituted by a DSP.

  In the above description, the “volatile storage unit” means a storage unit in which the stored contents are lost when the power of the digital still camera 1 is turned off. On the other hand, the “non-volatile storage unit” means a storage unit that maintains the stored contents even when the main power of the digital still camera 1 is turned off. Any memory device can be used as long as it can retain the stored contents. The semiconductor memory device itself is not limited to a nonvolatile memory device, and a backup power supply is provided to make a volatile memory device “nonvolatile”. You may comprise as follows. Note that the special application is not limited to a semiconductor memory element, but may be configured using an external medium such as a magnetic disk or an optical disk by using an external drive device.

  In the digital still camera 1 configured as such an electronic computer, when a series of processing is executed by software, a program configuring the software is installed from a recording medium (in this example, the ROM 93a). . This software includes an OS (operating system; basic software) running on the computer.

  The program for causing the central control unit 92 to execute a predetermined process may be distributed (acquired or updated) through any portable storage medium such as a non-volatile semiconductor memory card such as a CD-ROM or a flash memory. Alternatively, the program may be downloaded and acquired from a server or the like via a network such as the Internet or updated.

  The central control unit 92 reads the control program stored in the ROM 93a of the storage unit 93 constituted by a semiconductor memory or the like, and the entire digital still camera 1 based on the read control program or a command from the user. Control the operation and signal processing. A configuration is realized in which the digital still camera 1 is configured in software using a CPU or memory, that is, the digital still camera 1 is functioned in software using the function of a computer (electronic computer) such as a personal computer.

  In such a configuration, the central control unit 92 controls the entire system via the system bus 99. The ROM 93a stores data common to the apparatus such as a control program of the central control unit 92. The RAM 93b is configured by an SRAM (Static Random Access Memory) or the like, and stores program control variables, data for various processes, and the like. The RAM 93b includes an area for temporarily storing image data read by the solid-state imaging device 10, image data edited by a predetermined application program, image data read from the memory 72, and the like.

  The control system 9 also includes an exposure controller 94 that controls the shutter 52 and the diaphragm 56 so that the brightness of the image sent to the image signal processing unit 66 is kept at an appropriate level, and the image signal processing unit from the CCD solid-state imaging device 11. 66 from a drive control unit 96 having a timing signal generation unit (timing generator; TG) 40 for controlling the operation timing of each functional unit up to 66, and keys and switches for the user to input shutter timing, zoom operation or other commands. It has the operation part 98 which becomes.

  The central control unit 92 controls the image signal processing unit 66, the CODEC 74, the memory 72, the exposure controller 94, and the timing signal generation unit 40 connected to the system bus 99 of the digital still camera 1.

  The digital still camera 1 includes automatic control devices such as auto focus (AF), auto white balance (AWB), and automatic exposure (AE). These controls are processed using an output signal obtained from the CCD solid-state imaging device 11. For example, the exposure controller 94 has its control value set by the central control unit 92 so that the brightness of the image sent to the image signal processing unit 66 is kept at an appropriate level, and controls the diaphragm 56 according to the control value. . Specifically, the central control unit 92 acquires an appropriate number of luminance value samples from the image held in the image signal processing unit 66, and the average value thereof falls within a predetermined appropriate luminance range. The control value of the diaphragm 56 is set so as to be within the range.

  The timing signal generation unit 40 is controlled by the central control unit 92 and generates timing pulses required for the operation of the CCD solid-state imaging device 11, the preamplifier unit 62, the A / D conversion unit 64, and the image signal processing unit 66. Supply to each part. The operation unit 98 is operated when the user operates the digital still camera 1.

  In the illustrated example, the preamplifier unit 62 and the A / D conversion unit 64 of the signal processing system 6 are built in the camera module 3. However, the configuration is not limited to such a configuration, and the preamplifier unit 62 and the A / D conversion unit 64 are included. The structure provided in the main body unit 4 can also be taken. A configuration in which the D / A conversion unit is provided in the image signal processing unit 66 can also be adopted.

  Further, although the timing signal generation unit 40 is built in the camera module 3, the configuration is not limited to such a configuration, and a configuration in which the timing signal generation unit 40 is provided in the main unit 4 can also be adopted. In addition, the timing signal generation unit 40 and the driver 42 are separate components, but the configuration is not limited to this, and the timing signal generation unit 40 and the driver 42 may be integrated (timing generator with built-in driver). By doing so, a more compact (small) digital still camera 1 can be configured.

  Further, the timing signal generator 40 and the driver 42 may each be configured by a circuit with individual discrete members, but are provided as an IC (Integrated Circuit) formed on a single semiconductor substrate. Is good. By doing so, not only can it be made compact, but the handling of the members becomes easy, and both can be realized at low cost. In addition, the digital still camera 1 can be easily manufactured. In addition, the timing signal generator 40 and the driver 42 which are strongly related to the CCD solid-state image sensor 11 to be used are integrated on the same substrate as the CCD solid-state image sensor 11 or integrated in the camera module 3. When integrated by mounting, handling and management of the members become simple. In addition, since these are integrated as a module, the digital still camera 1 (completed product) can be easily manufactured. The camera module 3 may be composed only of the CCD solid-state imaging device 11 and the optical system 5.

  An outline of a series of operations of the digital still camera 1 provided with such a CCD solid-state imaging device 11 is as follows. First, the timing signal generation unit 40 generates various pulse signals such as transfer clocks V1 to V6 (V8) for vertical transfer and a read pulse XSG. These pulse signals are converted into drive pulses of a predetermined voltage level by the driver 42 and then input to predetermined terminals of the CCD solid-state imaging device 11.

  When the subject Z is imaged, an optical image of the subject Z formed on the light receiving surface of the CCD solid-state imaging device 11 via the imaging lens 50 (shutter 52 and lens 54) is received by each sensor unit including a photodiode or the like. The signal charge is converted into an amount corresponding to the amount of incident light.

  The signal charges accumulated in each of the sensor units are applied to the transfer channel terminal electrode of the read gate unit by the read pulse XSG generated from the timing signal generation unit 40, and the potential below the transfer channel terminal electrode is deepened. Data is read out to the vertical CCD through the readout gate section. Then, the vertical CCD is driven based on the 6-phase (8-phase) vertical drive pulses φV1 to φV6 (φV8), and sequentially transferred to the horizontal CCD.

  If the accumulated signal charge can be swept by the shutter gate pulse, a so-called electronic shutter function for controlling the charge accumulation time (shutter speed) can be realized. In this case, the shutter 52 of the imaging lens 50 can be removed, and the optical system 5 can be made compact.

  The horizontal CCD is generated on one line vertically transferred from each of the plurality of vertical CCDs based on the two-phase horizontal drive pulses φH1 and φH2 which are emitted from the timing signal generation unit 40 and converted to a predetermined voltage level by the driver 42. Corresponding signal charges are sequentially horizontally transferred to the charge-voltage converter.

  The charge-voltage conversion unit accumulates signal charges sequentially injected from the horizontal CCD in a floating diffusion (not shown), converts the accumulated signal charges into a signal voltage, and performs timing, for example, via an output circuit having a source follower configuration (not shown). An image pickup signal (CCD output signal) Vout is output under the control of the reset pulse RG generated from the signal generator 40.

  That is, in the CCD solid-state imaging device 11, the signal charges detected in the imaging area formed by two-dimensionally arranging the sensor units in the vertical and horizontal directions are converted into horizontal CCDs by the vertical CCDs provided corresponding to the vertical columns of the sensor units. The signal charges are then transferred in the horizontal direction by the horizontal CCD based on the two-phase horizontal transfer pulses H1 and H2. Then, the operation of converting to a potential corresponding to the signal charge from the horizontal CCD in the charge voltage conversion unit and outputting is repeated.

  The voltage signals sequentially read from the CCD solid-state imaging device 11, that is, R, G, and B pixel signals corresponding to the pixels, are converted into CDS by the preamplifier 62 based on the sample pulses from the timing signal generator 40. After being processed and converted into digital R, G, and B pixel data by the A / D conversion unit 64, the data is temporarily stored in the RAM 93 b of the storage unit 93.

  The R, G, and B pixel data stored in the RAM 93b are subjected to a synchronization process, a gamma correction process, and the like by the image signal processing unit 66, and then the luminance data Y and the color (chroma) data U, V. (Or Cr, Cb) (collectively referred to as YC data) and temporarily stored in the RAM 93b of the storage unit 93.

  The display system 8 can display a through image, a captured still image, and the like by reading the YC data stored in the RAM 93b and outputting the YC data to a video monitor 84 made of liquid crystal or the like.

  The YC data after photographing is compressed into a predetermined format such as JPEG (Joint Photographic Experts Group) by a CODEC 74 having a compression / decompression function, and then recorded on a recording medium such as the memory 72. Further, in the playback mode, the image data recorded in the memory 72 or the like is decompressed by the CODEC 74 and then output to the video monitor 84 to display the playback image.

<Overall configuration of digital still camera; CMOS type>
FIG. 2 is a schematic configuration diagram showing a second embodiment of the imaging apparatus according to the present invention. In the imaging apparatus of the second embodiment, the CCD solid-state imaging device 11 in the first embodiment is changed to a CMOS imaging device 12, and in accordance with this change, the preamplifier unit 62, the A / D conversion unit 64, and drive control are performed. The circuit unit having the same function as the unit 96 is changed so as to be taken into the CMOS image sensor 12. Other points are the same as in the first embodiment.

  The image signal processing unit 66 constituted by the DSP generates a central control unit 92 composed of a CPU and the like, various memories of the storage unit 93 excluding the EEPROM 93d, and further generates a synchronization signal that defines the operation timing of the CMOS image sensor 12. The timing signal generation unit 40 is captured, and the image pickup device module 3a is configured to include the camera module 3 having the CMOS image pickup device 12 as a main part, the image signal processing unit 66, and the EEPROM 93d.

  The CMOS image sensor 12 is not shown in its configuration example, but first, the point of using a photodiode or the like in the sensor unit and the point of providing a color separation filter for color image capturing are the same as in the case of the CCD solid-state image sensor 11. It is the same.

  The CMOS image sensor 12 is a pixel in which a plurality of sensor units (pixels) including photoelectric conversion elements such as photodiodes that output an electrical signal corresponding to the amount of incident light are arranged in rows and columns (that is, in a two-dimensional matrix). A column processing unit 12b that includes a unit (imaging unit) 12a, a signal output from each pixel is a voltage signal, and performs data processing such as a CDS processing function unit and a digital conversion unit (ADC). It is provided in parallel. In addition, in the periphery of the imaging unit composed of a large number of sensor units, there is a drive control unit 96 in the case of a CCD, and in addition to a drive control unit 12c that drives and controls the CMOS image sensor 12, there is a column processing unit 12b. A reference signal generator for supplying a reference voltage for AD conversion, an output circuit, and the like are provided.

  “The data processing unit is provided in parallel with the column” means that a plurality of CDS processing function units and digital conversion units are provided substantially in parallel with the vertical signal lines in the vertical column. Each of the plurality of functional units may be arranged only on one edge side (output side) in the column direction with respect to the imaging unit when the device is viewed in plan, or the pixel unit However, it may be divided into one end side (output side) in the column direction and the other end side on the opposite side. In the latter case, it is preferable that the horizontal scanning unit that performs readout scanning (horizontal scanning) in the row direction is also arranged separately on each edge side so that each can operate independently.

  The drive control unit has a control circuit function for sequentially reading signals from the imaging unit. For example, the control circuit functions include a horizontal scanning unit (column scanning circuit) that controls column addresses and column scanning, a vertical scanning unit (row scanning circuit) that controls row addresses and row scanning, and an internal clock. And a communication / timing control unit having the above functions.

  The horizontal scanning unit has a function of a reading scanning unit that reads a count value from the column processing unit. Each element of these drive control units is integrally formed in a semiconductor region such as single crystal silicon using a technique similar to the semiconductor integrated circuit manufacturing technique together with the imaging unit, and is a solid-state imaging device (example of a semiconductor system) Imaging device).

  The vertical scanning unit selects a row of the pixel portion 12a and supplies a control pulse (line shift pulse) necessary for the row. For example, a pulse is supplied to a vertical decoder that defines a readout row in the vertical direction (selects a row of the pixel portion 12a) and a row control line for a unit pixel on a readout address (in the row direction) defined by the vertical decoder. And a vertical driving circuit for driving the Note that the vertical decoder selects a row for electronic shutter in addition to a row from which a signal is read.

  The horizontal scanning unit sequentially selects the column AD circuits of the column processing unit 12b in synchronization with the low-speed clock, and guides the signal to a horizontal signal line (horizontal output line). For example, a horizontal decoder that selects a read column in the horizontal direction (selects each column AD circuit in the column processing unit 12b) and each signal of the column processing unit 12b according to a read address defined by the horizontal decoder And a horizontal drive circuit that leads to the signal line. For example, if the number of horizontal signal lines is n (n is a positive integer) handled by the column AD circuit, for example, 10 (= n), 10 horizontal signal lines are arranged corresponding to the number of bits.

  A CMOS sensor is an example of an XY address type solid-state imaging device. Signals can be extracted from a pixel at an arbitrary position by addressing, and signal charges obtained from the pixels are selected by a shift register in order. Unlike the CCD (Charge Coupled Device) type image sensor that reads out the pixel signals, the order in which the pixel signals are read out can be set relatively freely.

  For example, in a still image imaging technique represented by a digital still camera, a multi-pixel CMOS solid-state image sensor is used as an imaging device, and a still image is obtained by independently reading out pixel information of all pixels. Is well known, but in addition to this mode, for example, a “decimation readout mode” that reads out several rows or columns at a time, for example, selects several rows or columns (not limited to adjacent pixels). Thus, operations such as “addition readout mode” in which readout, addition, and output are performed can be easily realized on the sensor side.

  In the thinning readout mode, for example, when the subject is being confirmed (monitoring mode), a rough image (low resolution image) corresponding to the number of pixels of the liquid crystal monitor is output, or pixel information is thinned out for moving images. This is used when the information amount is reduced and transmitted. The addition reading mode is used for the purpose of expanding the dynamic range by outputting signals from a plurality of rows (for example, two rows) and adding them.

  Also, a reverse reading mode (mirror processing) for performing reverse reading may be required for the purpose of preventing the image from being inverted on the monitor when the camera portion is rotated 180 degrees. The reverse direction reading mode is a mode in which scanning is performed in the reverse direction with respect to the order of address scanning in the forward direction reading mode, and this mode can also be easily realized on the sensor side. For example, in the forward reading mode, the row and column addresses are scanned in order from the smallest, and in the backward reading mode, the row and column addresses are scanned from the larger to the smaller.

  Depending on the interface between the column processing unit 12b and the output unit and the configuration of the horizontal scanning unit, horizontal reverse reading may not always be realized on the sensor side. In this case, the horizontal mirror processing may be realized by the image signal processing unit 66.

  In the digital still camera 1 including the CMOS image sensor 12 having such a configuration, the pixel signal output from the unit pixel is supplied to the column AD circuit of the column processing unit 12b via the vertical signal line for each vertical column. Is done. Amplifying transistors constituting the unit pixel are connected to each vertical signal line.

  Each column AD circuit of the column processing unit 12b receives a pixel signal for one column and converts the signal into digital data. The details of the configuration of the AD circuit are omitted, but as an example, a ramp-like reference signal (reference voltage) is supplied to the comparator (voltage comparator) and at the same time, counting with a clock signal is started. The analog pixel signal input through the vertical signal line is compared with the reference signal, and the AD conversion is performed by counting until a pulse signal is obtained.

  At this time, by devising the circuit configuration, the signal level (noise level) immediately after the pixel reset and the true (light receiving) for the voltage mode pixel signal input through the vertical signal line as well as AD conversion. It is possible to perform processing for obtaining a difference from the signal level (in accordance with the amount of light). Thereby, it is possible to remove a noise signal component called fixed pattern noise (FPN) or reset noise.

  The pixel data digitized by the column AD circuit is transmitted to a horizontal signal line via a horizontal selection switch (not shown) driven by a horizontal selection signal from a horizontal scanning unit, and further input to an output circuit.

  With such a configuration, from the pixel unit 12a in which the light receiving elements as charge generation units are arranged in a matrix, a driving pulse (vertical CCD) is supplied from a vertical scanning unit (row scanning circuit) that controls row address and row scanning. Based on the transfer clock), pixel signals are sequentially output for each vertical column for each row. Then, one image corresponding to the pixel portion 12a in which light receiving elements (photoelectric conversion elements such as photodiodes) are arranged in a matrix, that is, a frame image, is shown as a set of pixel signals of the entire pixel portion 12a.

<< Electronic Zoom Processing Function; First Example >>
FIG. 3 is a block diagram of a first example focusing on the electronic zoom processing function in the digital still camera 1 shown in FIGS. 1 and 2. This first example is characterized in that resolution conversion is performed by interpolation processing using primary color data as a processing target.

  As shown in the figure, as a signal processing function in the image processing unit (DSP) 66, a digital clamping unit 200 that clamps a black reference of imaging data acquired by the solid-state imaging device 10 and digitized by the A / D conversion unit 64, In addition, after extracting primary color data of R, G, B from the image data clamped by the digital clamp unit 200, sharpness correction, gamma correction, contrast correction, or other luminance is applied to the primary color data R, G, B. A color signal processing unit 220 that performs signal processing and color signal processing to convert to luminance data Y (or lightness data L) and two color data U and V (or color difference data RY and BY) and outputs the data. I have.

  Each data generated by the color signal processing unit 220 is sent to the recording system 7 (for example, CODEC 74) for image recording, or sent to the display system 8 (for example, video encoder 86) for display output. Or

  As an example, the color signal processing unit 220 extracts primary color data of R, G, and B from the image data clamped by the digital clamp unit 200 and simultaneously synchronizes the primary color separation / synchronization processing unit 222. The white balance amplifier 224, which is an example of a color signal amplifier capable of adjusting the gain of the pixel data, is provided. The color signal processing unit 220 includes a gamma correction unit 226 and a color difference matrix unit 228. Of course, this configuration example is merely an example, and the color signal processing unit 220 includes components other than these processing functions.

  The gamma correction unit 226 performs gamma (γ) correction for faithful color reproduction based on the R signal Sr3, the G signal Sg3, and the B signal Sb3, and outputs the output signals R, G and B are input to the color difference matrix unit 228. The color difference matrix unit 228 inputs the color difference signals RY and BY obtained by performing the color difference matrix processing to the video encoder 86.

  The video encoder 86 digitally modulates the color difference signals RY and BY with a digital signal corresponding to the color signal subcarrier, and then synthesizes the digital signal with a luminance signal Y generated by a luminance signal generation unit (not shown). Video signal VD (= Y + S + C; S is a synchronization signal, C is a chroma signal, or Y signal and chroma signals Cb, Cr), and then input to D / A converter 82. The D / A converter 82 converts the digital video signal VD into an analog video signal V.

  In addition, the image signal processing unit 66 is a feature part of the present embodiment. The resolution conversion processing unit 300 is a functional element that performs resolution conversion processing (that is, electronic zoom processing) that electronically enlarges or reduces an image by image data processing. Are provided between the gamma correction unit 226 and the color difference matrix unit 228.

  The resolution conversion processing unit 300 performs an interpolating process between pixels in the horizontal direction according to the horizontal magnification, and converts a horizontal resolution by interpolating between the horizontal pixels, and interpolates a vertical line according to the vertical magnification. A vertical resolution conversion processing unit 320 that converts the resolution in the vertical direction by processing and a conversion control unit 330 that controls the resolution conversion processing function are provided.

<< Electronic zoom processing function; second example >>
FIG. 4 is a block diagram of a second example focusing on the electronic zoom processing function in the digital still camera 1 shown in FIGS. 1 and 2. This second example is characterized in that the primary color data is not processed, but is converted to YCbCr and then the resolution is converted by interpolation processing.

  That is, as illustrated, a resolution conversion processing unit 300 is provided between the video encoder 86 and the D / A conversion unit 82. The resolution conversion processing unit 300 performs resolution conversion processing using interpolation processing on the Y signal supplied from the video encoder 86 and the two chroma signals Cb and Cr as processing targets.

  As in the first example, the resolution conversion processing unit 300 performs horizontal interpolation between pixels in the horizontal direction according to the horizontal magnification, thereby converting the horizontal resolution, and the vertical magnification. And a vertical resolution conversion processing unit 320 for converting the resolution in the vertical direction by performing interpolation processing on the vertical line, and a conversion control unit 330 for controlling the resolution conversion processing function.

<Detailed Configuration Example of Resolution Conversion Processing Unit>
FIG. 5 is a block diagram illustrating a detailed configuration example of the resolution conversion processing unit 300. The configuration for realizing the electronic zoom function of the present embodiment stores pixel signals for one screen read from the solid-state imaging device 10 or an amount corresponding to the magnification of the electronic zoom (for example, 1/2 frame at the time of double zoom). A large-scale storage means such as a frame memory is not provided, and a simple memory such as a line memory is used.

  In addition to performing pixel interpolation in the resolution conversion processing unit 300 first, in addition to performing pixel interpolation in the resolution conversion processing unit 300 in order to simultaneously perform pixel signal reading processing and interpolation processing from the solid-state imaging device 10, In conjunction with the pixel interpolation operation, the reading timing of the solid-state imaging device 10 and the writing process of the read pixel information to the storage unit are controlled. For this reason, by exchanging information between the conversion control unit 330 and the drive control unit 96 (12c), the two operate together.

  In order to perform pixel interpolation processing in the resolution conversion processing unit 300, the horizontal resolution conversion processing unit 310 first receives pixel data (primary color data from the gamma correction unit 226 or video encoder received from the previous stage of the resolution conversion processing unit 300. 86) and the pixel data processed by each functional unit in the resolution conversion processing unit 300 is temporarily stored for each horizontal line. Also, the horizontal resolution conversion processing unit 310 refers to pixel data in one horizontal line (hereinafter also referred to as line data) stored in the line buffer memory 312 so that the desired pixel data amount specified by the conversion control unit 330 is obtained. The horizontal interpolation processing unit 314 performs enlargement conversion or reduction conversion (hereinafter collectively referred to as scaling processing) by performing interpolation processing in the horizontal direction.

  The conversion control unit 330 controls writing and reading of pixel information (pixel data) from the solid-state imaging device 10 to the line buffer memory 312 in the horizontal scanning direction, which is a relatively high-speed first scanning direction, and also in the horizontal direction. The operation of the interpolation processing in the interpolation processing unit 314 and the vertical resolution conversion processing unit 320 is controlled.

  Here, when the conversion control unit 330 writes pixel information for one unit in the horizontal scanning direction to the line buffer memory 312, a line is provided so that a dummy period that does not contribute to writing the pixel information to the line buffer memory 312 is provided. The writing operation to the buffer memory 312 is controlled. By doing this, the pixel signal reading operation from the solid-state image sensor 10 and the pixel information writing operation to the line buffer memory 312 and the relatively slow second scanning direction different from the horizontal scanning direction are vertical. A period for performing interpolation processing substantially simultaneously in the direction is prepared.

  The line buffer memory 312 is not all pixel data for one horizontal line received from the previous stage, but pixel data corresponding to the horizontal use area of the image area used at the time of zooming according to the zoom magnification, specifically, Only the pixel data in the area corresponding to the inverse of the electronic zoom magnification is stored in the line buffer memory 312 as a processing target.

  The horizontal interpolation processing unit 314 processes only pixel signals in a region corresponding to the reciprocal of the electronic zoom magnification, and generates horizontal pixel data that is excessive or insufficient according to the electronic zoom magnification by interpolation processing. Further, the horizontal interpolation processing unit 314 stores the line data after the scaling process in the line buffer memory 312 as necessary (according to the processing of the vertical resolution conversion processing unit 320).

  Further, the vertical resolution conversion processing unit 320 refers to the pixel data in units of lines whose horizontal resolution has been converted by the horizontal interpolation processing unit 314 for a plurality of lines, and has a desired pixel data amount designated by the conversion control unit 330. As described above, the vertical interpolation processing unit 324 performs enlargement conversion or reduction conversion (magnification processing) by performing interpolation processing in the vertical direction. The vertical interpolation processing unit 324 generates only pixel data in the vertical direction corresponding to the reciprocal of the magnification of the electronic zoom, and generates vertical pixel data that is excessive or insufficient according to the magnification of the electronic zoom by interpolation processing. To do.

  During the interpolation process, the vertical interpolation processing unit 324 reads the line data after horizontal interpolation from the line buffer memory 312, receives the line data immediately after the horizontal interpolation by the horizontal interpolation processing unit 314, and receives these multiple lines. Perform vertical interpolation using minute line data. For example, at the time of “double zoom”, pixel data for two lines (N lines and “N + 1” lines are used to obtain pixel data for the line between them (N + 0.5 lines) by interpolation processing). For the minute, the horizontal interpolation processing unit 314 performs horizontal interpolation processing, and the processed pixel data is temporarily stored in the line buffer memory 312.

  Then, each time the horizontal interpolation processing unit 314 performs horizontal interpolation processing for a predetermined horizontal position for “N + 1” lines, pixel data that has been subjected to horizontal interpolation processing for N lines at the same horizontal position is read from the line buffer memory 312. To perform vertical interpolation immediately. Alternatively, for the “N + 1” lines, the horizontal interpolation processing unit 314 performs horizontal interpolation processing, temporarily stores the processed pixel data in the line buffer memory 312, and the “N + 1” lines are stored. After all the horizontal interpolation processes are completed, the pixel data that has been subjected to the horizontal interpolation process at the same horizontal position of N and “N + 1” is read out from the line buffer memory 312 and the vertical interpolation process is performed. Good. Even in this case, the final data output is only delayed by one line compared to the former.

<Outline of electronic zoom processing>
FIG. 6 is a diagram for explaining an example of processing outline of electronic zoom. When enlarging by electronic zoom, at least in the vertical direction, only the portion to be enlarged is read and the pixel data is stored in the line buffer memory 312.

  For example, FIG. 6 shows an example of “double zoom” when the electronic zoom magnification is set to double in both vertical and horizontal directions. In this case, of the pixel data for one screen output from the solid-state image sensor 10, an area portion (for example, the center of the screen or an arbitrary position) that is ¼ of the length and width of the image and ¼ in area. Are used as a processing target.

  Then, the pixel spacing is doubled in each of the horizontal direction (horizontal direction) and the vertical direction (vertical direction) by some means, and the pixel position of the gap which is excessive or insufficient according to the magnification (2 times) of this electronic zoom is set. Pixel data (interpolation data) to be embedded is obtained.

  By doing this, it is possible to enlarge the image size to the original image by halving the length and width of the original image, for example, by halving the length and width of the center portion. A zoom image can be obtained without changing.

<Electronic Zoom Processing Timing; First Example>
7 to 9 are timing charts for explaining a first example of detailed processing timing for realizing such a mechanism of electronic zoom. Here, FIG. 7 illustrates the relationship between the horizontal synchronization signal SHD supplied to the drive control unit 96 (or 21c) that controls the solid-state imaging device 10 and the pixel data output from the solid-state imaging device 10 at the time of double zooming. FIG. 8 and 9 are diagrams for explaining the outline of the interpolation processing at the time of double zooming in the first example of the processing timing.

  First, the conversion control unit 330 generates a blanking (BLK) period corresponding to the number of pixels of the horizontal synchronization signal SHD that determines the horizontal driving reference of the solid-state imaging device 10 at normal time (when the zoom magnification is 1). The added cycle. Further, when electronic zoom is performed (when the zoom magnification is other than 1), the synchronization signal is generated so that the period of the horizontal synchronization signal supplied to the drive control unit 96 (12c) is extended at the same rate as the zoom magnification. The unit 41 is controlled. The synchronization signal generation unit 41 generates a horizontal synchronization signal SHD and a vertical synchronization signal SVD having a predetermined cycle based on an instruction from the conversion control unit 330, and supplies the horizontal synchronization signal SHD and the vertical synchronization signal SVD to the drive control unit 96 (12c) on the solid-state imaging device 10 side. .

  The drive control unit 96 (12c) generates various timing pulses for driving the solid-state imaging device 10 in accordance with the horizontal synchronization signal SHD and the vertical synchronization signal SVD from the synchronization signal generation unit 41, but at the cycle of the horizontal synchronization signal SHD. Regardless, the period of the pixel clock for driving the individual pixels of the solid-state imaging device 10 is set to be substantially the same as that in the normal time (there may be some differences).

  In addition, the conversion control unit 330 sets the pixel clock for reading pixel information from the solid-state imaging device 10 and the pixel clock for writing pixel information to the line buffer memory 312 in the first scanning direction at the same cycle regardless of the presence or absence of electronic zoom. Keep it. For example, a pixel signal read from the solid-state image sensor 10 is read out from the solid-state image sensor 10 in the horizontal direction, such as a horizontal transfer clock in a CCD or a horizontal shift clock in a CMOS. Keep the same period as the clock period.

  In this way, a dummy output period that does not substantially contribute to the readout of the pixel signal is provided in addition to the normal readout period of the pixel signal during electronic zooming (particularly during enlargement). By providing this dummy output period, it is possible to secure a period for performing the interpolation process, thereby enabling the pixel signal reading process from the solid-state imaging device 10 and the memory writing process and the interpolation process to be performed simultaneously in parallel. Like that.

  At this time, the reading process of the pixel signal from the solid-state imaging device 10 and the writing process to the line buffer memory 312 and the interpolation process can be performed simultaneously in parallel, and so as not to waste time. It is preferable to extend the pixel signal reading processing cycle in the horizontal scanning direction and the pixel information writing processing cycle to the line buffer memory 312 at the same magnification as the normal magnification magnification.

  For example, as shown in FIG. 7, at the time of double zooming, the period of the horizontal synchronization signal SHD1 given to the solid-state image sensor 10 is doubled with respect to the normal horizontal synchronization signal SHD0, so that the horizontal synchronization signal SHD1 In addition to the pixel data output period Hrd that is the first half of the synchronizing signal SHD0, that is, the normal pixel data reading period, a dummy output period Hdm that is not subjected to reading of the pixel signal substantially is read from the pixel data. Provided in the latter half of the output period Hrd.

  In the drive control unit 96 (12c) that drives the solid-state image sensor 10, the cycle of the pixel clock that drives the individual pixels of the solid-state image sensor 10 is set to be the same as the normal time regardless of the cycle of the horizontal synchronization signal SHD. As a result, by making the cycle of the horizontal synchronization signal SHD1 longer than the horizontal synchronization signal SHD0 according to the zoom magnification (in this example, twice the normal time), the frame (field) shift pulse with respect to the normal time, That is, the solid-state imaging device 10 is driven so as to thin out the CCD vertical transfer clock and the drive pulse of the CMOS row address.

  Thereby, the reading process of the pixel signal from the solid-state image sensor 10 in the horizontal direction and the horizontal interpolation process in the horizontal interpolation processing unit 314 can be performed substantially simultaneously. Here, “substantially simultaneously in parallel” means that the pixel signal from the solid-state imaging device 10 has a period width of the horizontal synchronization signal SHD0 within one period of the horizontal synchronization signal SHD1 during electronic zoom. This means that the reading process and the interpolation process in the horizontal interpolation processing unit 314 are alternately performed. Actually, the interpolation processing in the horizontal interpolation processing unit 314 is performed in the dummy output period Hdm.

  Further, since the pixel data can be delayed in the vertical direction to create a gap in timing between the pixel lines, interpolation data can be easily input in the vertical direction. That is, the above-described horizontal processing and vertical interpolation processing in the vertical interpolation processing unit 324 can be performed in parallel.

  For example, after YCbCr conversion for all pixels on the N-th line, as shown in FIGS. 8 and 9, the horizontal resolution conversion processing unit 310 outputs pixel data that is the first half of the horizontal synchronization signal SHD0 in the horizontal synchronization signal SHD1. In the period Hrd, pixel data corresponding to the “use area at the time of double zoom” is temporarily stored in the line buffer memory 312 as the cutting process in the horizontal direction.

  Thereafter, the horizontal interpolation processing unit 314 reads the pixel data from the line buffer memory 312 in the dummy output period Hdm, which is the second half of the horizontal synchronization signal SHD0 in the horizontal synchronization signal SHD1, at the timing of every other pixel. Reading is performed with a gap (at sparse timing), horizontal interpolation pixel data is calculated from the periphery of the target pixel, and the calculated horizontal interpolation pixel data is embedded in the pixel position of the gap. At this time, the timing signal generation unit 40 drives the solid-state imaging device 10 based on the first pixel clock having substantially the same period as the pixel clock period as in the normal time, and the horizontal interpolation processing unit 314 performs horizontal resolution. Perform the conversion.

  Interpolation calculation in the horizontal direction is performed using an average value from two adjacent pixels as an example of primary interpolation using a plurality of (two or more) reference pixel data adjacent to the target pixel in the horizontal direction. It is possible to perform interpolation processing using, and weighted linear interpolation processing that performs linear interpolation according to the distance between pixels while using three or more pixels. Of course, the interpolation processing method is not limited to linear interpolation, and for example, any interpolation method such as bilinear (bilinear interpolation), cubic convolution interpolation, or spline interpolation can be used. When weighted linear interpolation processing, convolution interpolation, or the like is used, an image quality-oriented interpolation method can be realized as compared with interpolation processing using an average value from two lines. Since details of these various interpolation processes are known techniques, detailed description thereof is omitted here.

  In this way, the horizontal interpolation processing unit 314 stores the processed pixel data in the line buffer memory 312 for each line when the extraction process and the interpolation process in the horizontal direction of the Nth line are completed. Dummy pixels are output from the solid-state imaging device 10 for one horizontal period of the next horizontal synchronization signal SHD0, but the horizontal interpolation unit ignores it and does not process it.

  Similarly, the horizontal resolution conversion processing unit 310 performs horizontal cut-out processing and horizontal interpolation processing for the “N + 1” -th line as well.

  Next, for the Nth line and the “N + 1” th line, the vertical resolution conversion processing unit 320 outputs the line data subjected to the horizontal interpolation processing by the horizontal interpolation processing unit 314 as it is, while “N + 0.5”. When generating line data to be interpolated corresponding to the line, using the data after horizontal interpolation of the Nth line stored in the line buffer memory 312 and the pixel data after horizontal interpolation of the “N + 1” line, Interpolation for one line is performed.

  At this time, the vertical interpolation processing unit 324 is parallel to the reading process of the pixel signal from the solid-state imaging device 10 based on the first pixel clock having substantially the same period as the normal time and the horizontal interpolation processing in the horizontal interpolation processing unit 314. Then, the vertical resolution conversion process is performed based on the second pixel clock having substantially the same cycle as the first pixel clock (which may be slightly different). That is, the number of vertical pixels using raw pixel data of a plurality of adjacent lines adjacent to the target pixel to be interpolated according to the zoom magnification or horizontal interpolation data obtained by horizontal interpolation processing in the horizontal interpolation processing unit 314 Is converted into the desired number of pixels.

  At this time, as shown in FIG. 8, the horizontal interpolation processing unit 314 performs horizontal interpolation processing at a predetermined horizontal position for “N + 1” lines (at least one pixel for horizontal interpolation processing). The line buffer memory 312 includes the pixel data at the predetermined horizontal position of the line data immediately after the interpolation process for the “N + 1” line and the pixel data that has been subjected to the horizontal interpolation process for N lines at the same horizontal position. And the interpolation process in the vertical direction is immediately performed for “N + 0.5” lines which are interpolation lines of the N and “N + 1” lines. In this case, as can be seen from the figure, horizontal interpolation processing and vertical interpolation processing are performed during the “dummy output period”.

  Alternatively, as shown in FIG. 9, the horizontal interpolation processing unit 314 also performs horizontal interpolation processing on the entire “N + 1” line, and temporarily stores the processed pixel data in the line buffer memory 312. Thereafter, pixel data that has been subjected to horizontal interpolation processing at the same horizontal position of N and “N + 1” is read from the line buffer memory 312, and “N + 0.5” is an interpolation line of each of N and “N + 1” lines. "Interpolate vertically for the line.

  In this case, the final data output is delayed by one line compared to the processing shown in FIG. In this case, as can be seen from the figure, the horizontal interpolation process is performed during the “dummy output period”, but the vertical interpolation process is performed during the pixel signal read process and the read pixel signal write process to the line buffer memory 312. Will be. Even in this case, horizontal interpolation processing and vertical interpolation processing are performed within one processing period of SHD1 in the horizontal scanning direction in which the dummy output period is provided.

  Interpolation calculation in the vertical direction is adjacent as an example of primary interpolation using reference pixel data at the same pixel position in a plurality of lines adjacent in the vertical direction of the target pixel in the same manner as interpolation calculation in the horizontal direction. Interpolation processing using an average value from two lines or weighted linear interpolation processing that performs linear interpolation according to the distance between pixels while using three or more lines can be performed. Of course, also in the vertical direction, this interpolation processing method is not limited to linear interpolation, and for example, any interpolation method such as bilinear interpolation, cubic convolution interpolation, or spline interpolation can be used. Since details of these various interpolation processes are known techniques, detailed description thereof is omitted here.

  For example, if interpolation processing using an average value from two lines is used, it is sufficient to prepare a line memory for at least one line in the line buffer memory 312, which is the method that can save the line memory most. In a system that can secure a little more number of line memories, an interpolation method emphasizing image quality such as weighted linear interpolation processing and convolution interpolation may be realized.

  As described above, according to the present embodiment, the period of the pixel clock for driving the individual pixels of the solid-state imaging device 10 is maintained substantially the same as the normal time regardless of the period of the horizontal synchronization signal SHD. The period of the horizontal synchronization signal SHD1 during the electronic zoom process is made longer than the normal horizontal synchronization signal SHD0 in accordance with the zoom magnification, so that the frame (field) shift pulse is thinned out from the normal time so that the solid-state imaging device 10 is thinned. Drive. By securing a period for performing the interpolation processing according to the zoom magnification, expanding the space between the pixels according to the zoom magnification, and embedding the pixel data acquired by the interpolation processing in the pixel position of the gap, the vertical and horizontal 1 of the original image / Focusing on the length of the magnification, the vertical and horizontal directions were enlarged by the magnification respectively.

  As a result, by using an extremely small number of line memories, it is possible to realize electronic zoom conversion of an image in real time and obtain a zoom image without changing the image size from the original image. The circuit scale is large, and the electronic zoom function can be realized at low cost without using an expensive frame memory.

  Since simple line memory control management is required, the capacity of the line buffer memory 312 can be extremely small, and in addition to the conventional read control from the frame memory, a gap for interpolation is created. There is no need to realize difficult control, which is very advantageous for downsizing the electronic camera. In addition, since the number of pixel data reading processes can be reduced, the power consumption of the electronic camera is low, and the battery life of the electronic camera can be extended. Since an electronic zoom system can be constructed at a low cost, it is very suitable as an electronic zoom for realizing a camera for a mobile phone or a mobile device.

  The method described in Patent Document 2 is also common in that it does not use a large-scale storage means such as a frame memory and does not require reading control from the frame memory or the like. The first pixel for pixel signal readout processing or horizontal interpolation processing is required for vertical interpolation processing while maintaining the same readout driving of the imaging device as normal (no change to sensor driving) during enlargement A second pixel clock that is faster than the clock is required.

  On the other hand, in the configuration of the present embodiment, a pixel clock having the same period can be used in each processing (pixel signal reading processing and horizontal / vertical interpolation processing) regardless of the zoom magnification, and the memory. However, it is sufficient for at least one line, and it is more advantageous and simpler than the method described in Patent Document 2 (smaller circuit configuration), lower cost, and lower power consumption. This is advantageous, and there is little problem of image quality degradation even in the enlarged zoom mode.

  Furthermore, since the resolution conversion method of the present embodiment depends only on the resolution on the pixel data readout side, the number of pixels and the color filter array of each imaging device such as the CCD solid-state imaging device 11 and the CMOS imaging device 12 are different. A plurality of different electronic cameras can be used in common with the same apparatus configuration, and a versatile mechanism independent of the device structure can be constructed.

  As described above, when the conventional method performs the pixel signal reading process and the resolution conversion process in parallel (that is, in real time) when enlarging with the electronic zoom, as described in Patent Document 1, A large-scale line memory that stores line data for 1 / magnification frames (number of lines) is used, or a horizontal synchronization signal that reads out a pixel signal from the solid-state imaging device 10 as described in Patent Document 2 is used. It is necessary to set the cycle of vertical interpolation calculation processing (vertical resolution conversion processing by data processing) faster than the cycle.

  On the other hand, according to the configuration and processing procedure of the present embodiment, the second pixel clock for vertical interpolation processing with respect to the pixel signal readout clock and the horizontal interpolation processing pixel clock (first pixel clock). A dummy output period Hdm for performing interpolation processing for zoom enlargement is provided by changing the pixel signal readout cycle to 1 / magnification without increasing the cycle. There is no need for large-capacity storage means such as a frame memory that stores pixel data corresponding to the area used by the electronic zoom, and there is no need to set the interpolation calculation processing cycle fast, so high-speed processing is not required, and noise and power consumption From this point of view, it is superior to the mechanism of Patent Document 2.

  In the electronic zoom method of the present embodiment, a decrease in the frame rate that is delayed by the reciprocal of the zoom magnification (for example, 1/2 during double zooming) occurs. This is considered to be a commercialization due to a sense of incongruity due to a drop in the frame rate and an effect of cost reduction. However, assuming that a recent imaging device is used, it can be said that there is a region where there is no sense of incongruity even when the frame rate is lowered by using the electronic zoom function when a device with a high driving frequency can be used.

  Note that when the line information of the unused area that does not contribute to zoom enlargement is read out from the solid-state imaging device 10, it is read out (swept out) at a higher speed than the normal pixel clock, thereby improving the frame rate. May be.

<Electronic Zoom Processing Timing; Second Example>
FIG. 10 is a timing chart for explaining a second example of detailed processing timing for realizing the mechanism of the electronic zoom, and shows an example at the time of double zooming. FIG. 10 corresponds to FIG. 10, and shows the relationship between the horizontal synchronization signal SHD applied to the drive control unit 96 (or 21 c) that drives and controls the solid-state image sensor 10 and the pixel data output from the solid-state image sensor 10. It is a figure explaining.

  In the first example, the period of the horizontal synchronization signal SHD1 is longer than that of the horizontal synchronization signal SHD0 (in the first example, twice that of the normal time) so that the vertical transfer clock is thinned out from the normal time. As a result, it is only necessary to thin out the vertical transfer clock with respect to the normal time, and whether or not to change the cycle of the horizontal synchronization signal SHD is not an essential element.

  The horizontal synchronization signal SHD supplied from the synchronization signal generation unit 41 to the timing signal generation unit 40 is maintained at substantially the same cycle as normal time regardless of the presence or absence of the electronic zoom operation. During the zoom operation (the magnification is other than 1), the solid-state imaging device 10 may be driven by thinning out the line shift pulse, which is a scanning clock in the vertical direction, according to the magnification of the electronic zoom.

  For example, as shown in FIG. 10, the timing signal generation unit 40 performs a shift operation in the vertical direction, that is, a vertical transfer of a CCD or a column processing unit, at an interval of one line in a normal state at the time of double zooming. It suffices to perform readout processing in units of rows in the CMOS of the column parallel configuration, and only the operation of thinning out the line shift pulse for each line may be performed without changing the cycle of the horizontal synchronization signal SHD.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various changes or improvements can be added to the above-described embodiment without departing from the gist of the invention, and embodiments to which such changes or improvements are added are also included in the technical scope of the present invention.

  Further, the above embodiments do not limit the invention according to the claims (claims), and all combinations of features described in the embodiments are not necessarily essential to the solution means of the invention. Absent. The embodiments described above include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. Even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, as long as an effect is obtained, a configuration from which these some constituent requirements are deleted can be extracted as an invention.

  For example, in the above-described embodiment, the zoom magnification is specifically described by taking the double zoom as an example. However, the zoom magnification is not limited to double, and the first scanning direction (in order to secure a period for interpolation processing related to the electronic zoom) For example, when pixel information is read from the imaging device or written in the storage unit in the horizontal scanning direction), a dummy period that does not contribute to the pixel information reading process or writing process is provided, for example, three times Alternatively, zooming can be performed at an arbitrary integer multiple such as 4 × at a fixed magnification, and is not limited to an integer multiple, but M / N times (M> N; M and N are arbitrary integers). It is also possible to realize a variable zoom function in which a magnification can be freely specified, that is, a halfway magnification can be designated.

  In the above-described embodiment, the method of realizing the electronic zoom function using the image read from the imaging device as a processing target has been described. However, the processing target image is not limited to the image read from the imaging device. The mechanism for realizing the electronic zoom described in the above embodiment can be applied to any image processing target.

1 is a schematic configuration diagram illustrating a first embodiment of an imaging apparatus according to the present invention. It is a schematic block diagram which shows 2nd Embodiment of the imaging device which concerns on this invention. FIG. 3 is a block diagram of a first example focusing on an electronic zoom processing function in the digital still camera shown in FIG. 1 and FIG. 2. FIG. 3 is a block diagram of a first example focusing on an electronic zoom processing function in the digital still camera shown in FIG. 1 and FIG. 2. It is a block diagram explaining the detailed structural example of a resolution conversion process part. It is a figure explaining the process outline | summary example of an electronic zoom. It is a figure explaining the relationship between the horizontal synchronizing signal given to the drive control part which drive-controls a solid-state image sensor at the time of 2 times zoom, and the pixel data output from a solid-state image sensor. In the first example of processing timing, it is a diagram for explaining the outline of interpolation processing at the time of double zoom. In the first example of processing timing, it is a diagram for explaining the outline of interpolation processing at the time of double zoom. It is a timing chart explaining the 2nd example of the detailed processing timing for implement | achieving the mechanism of an electronic zoom. It is a figure explaining the example of a conventional structure of the structure which implement | achieves an electronic zoom function using a frame memory. It is a figure explaining the interpolation method using a frame memory (the 1). It is a figure explaining the interpolation method using a frame memory (the 2).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital still camera (imaging device), 2 ... Solid-state imaging device, 3 ... Camera module, 3a ... Imaging device module, 4 ... Main unit, 5 ... Optical system, 6 ... Signal processing system, 7 ... Recording system, 8 ... Display system, 9 ... Control system, 10 ... Solid-state imaging device, 11 ... CCD solid-state imaging device, 12 ... CMOS imaging device, 50 ... Imaging lens, 66 ... Image signal processing unit, 74 ... CODEC, 84 ... Video monitor, 92 ... Central control unit, 94 ... exposure controller, 98 ... operation unit, 99 ... system bus, 300 ... resolution conversion processing unit, 310 ... horizontal resolution conversion processing unit, 312 ... line buffer memory, 314 ... horizontal interpolation processing unit, 320 ... vertical Resolution conversion processing unit, 324 ... vertical interpolation processing unit, 330 ... conversion control unit

Claims (14)

The writing and reading of the pixel information to and from the storage unit are controlled in the relatively fast first scanning direction, and the pixel information read out from the storing unit is used and the pixel information read out from the storing unit is relatively slow and different from the first scanning direction. An image processing method for performing resolution conversion processing of an image by performing interpolation processing in a second scanning direction,
At the time of resolution conversion processing for enlarging the image size in the second scanning direction, a dummy period that does not contribute to writing of pixel information to the storage unit is written when pixel information is written in the storage unit in the first scanning direction. An image processing method characterized by being provided. By performing an interpolation process while reading out pixel information from the storage unit, a resolution conversion process for enlarging the image size in the first scanning direction is performed.
The image processing method according to claim 1, wherein the image size is enlarged in the second scanning direction using the pixel information enlarged in the first scanning direction. The dummy period is provided by making a writing processing cycle of the first scanning direction unit for writing the pixel information to the storage unit in the first scanning direction longer than a normal time. 2. The image processing method according to 1. 4. The image processing method according to claim 3, wherein the writing processing cycle in the first scanning direction is extended at the same magnification as that for the normal time in the enlargement. An image processing apparatus that performs resolution conversion processing of an image by performing interpolation processing,
A storage unit for storing pixel information;
A conversion control unit that controls writing and reading of pixel information to and from the storage unit in a relatively high-speed first scanning direction and controls the operation of the interpolation processing;
A resolution conversion processing unit that performs resolution conversion processing of an image by performing interpolation processing in a second scanning direction that is relatively low speed different from the first scanning direction using the pixel information read from the storage unit; With
The conversion control unit controls the storage unit so that a dummy period that does not contribute to writing of the pixel information to the storage unit is provided when writing pixel information to the storage unit in the first scanning direction. ,
The resolution conversion processing unit enlarges an image size in the second scanning direction by performing interpolation processing in the second scanning direction substantially using the dummy period. Processing equipment. The resolution conversion processing unit performs interpolation processing in the first scanning direction while reading out pixel information from the storage unit, thereby enlarging the image size in the first scanning direction, and in the first scanning direction. The image processing apparatus according to claim 5, wherein the image size is enlarged in the second scanning direction using the enlarged pixel information. The conversion control unit controls the storage unit so that a writing processing cycle in the first scanning direction unit for writing the pixel information to the storage unit in the first scanning direction is longer than normal. The image processing apparatus according to claim 5, wherein the dummy period is provided. The image processing apparatus according to claim 7, wherein the conversion control unit extends the writing processing cycle in the first scanning direction at a magnification that is the same as a magnification with respect to a normal time in the enlargement. An imaging device that captures an image of a subject using an imaging device and performs image resolution conversion processing by performing interpolation processing on the captured image,
A drive controller for controlling the pixel information to be read from the imaging device;
A storage unit for storing pixel information read from the imaging device;
A conversion control unit that controls writing and reading of pixel information to and from the storage unit in a relatively high-speed first scanning direction and controls the operation of the interpolation processing;
A resolution conversion processing unit that performs resolution conversion processing of an image by performing interpolation processing in a second scanning direction that is relatively low speed different from the first scanning direction using the pixel information read from the storage unit; With
The conversion control unit controls the drive control unit so that a dummy period that does not contribute to reading of pixel information from the imaging device is provided,
The resolution conversion processing unit enlarges an image size in the second scanning direction by performing interpolation processing in the second scanning direction substantially using the dummy period. apparatus. The resolution conversion processing unit performs interpolation processing in the first scanning direction while reading out pixel information from the storage unit, thereby enlarging the image size in the first scanning direction, and in the first scanning direction. The image pickup apparatus according to claim 9, wherein the image size is enlarged in the second scanning direction using the enlarged pixel information. The conversion control unit controls the drive control unit so that a reading process period in the first scanning direction unit for reading out pixel information from the imaging device in the first scanning direction is longer than normal. The imaging apparatus according to claim 9, wherein the dummy period is provided. The imaging apparatus according to claim 11, wherein the conversion control unit extends the writing processing cycle in the first scanning direction at the same magnification as that for the normal time in the enlargement. A drive control unit that reads out pixel information from the imaging device in a relatively high-speed first scanning direction and controls the pixel information to be read out in a relatively low-speed second scanning direction different from the first scanning direction. A timing control device for controlling the timing of the driving operation of the imaging device,
A scanning reference signal for reading pixel information from the imaging device in the first scanning direction, and at the time of resolution conversion processing for enlarging the image size in the second scanning direction, a readout processing period in units of the first scanning direction Timing control, comprising: a reference signal supply unit that provides a dummy period that does not contribute to reading pixel information from the imaging device by supplying a scanning reference signal longer than normal to the drive control unit apparatus. The timing control device according to claim 13, wherein the reference signal supply unit extends the writing processing cycle in the first scanning direction at the same magnification as that for the normal time in the enlargement.

Images