JP2006157153A - Imaging apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus and an imaging method capable of producing a shutter image representing an image imaged at a desired shutter speed, which is viewed with smooth movement, and an imaged image at a frame rate different from an imaging frame rate. <P>SOLUTION: A memory control section 15 writes an image signal DB of an imaged image to a memory 16, reads the image signal written in the memory 16, supplies the image signal to a signal processing section 154, and applies filtering processing to the read image signals by a plurality of frames to produce the shutter image wherein time resolution is preserved. In the case of producing the shutter image, a tap range in response to the shutter speed is moved in response to a frame rate of the shutter image and the image signal by a plurality of frames within the tap range is read from the memory 16. Further, the filtering processing produces the image signal at a frame rate higher than the imaging frame rate. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus and an imaging method. Specifically, an image signal of a captured image is generated and stored in a memory, and a filtering process is performed using a plurality of frames of image signals read from the memory to generate a shutter image that preserves time resolution. Also, filtering processing is performed to generate an image signal having a higher frame rate than the image signal generated by the imaging means.

  In the imaging apparatus, it is possible to display a movement of the subject at a speed different from the actual speed by changing a frame rate (hereinafter referred to as “imaging frame rate”) when imaging the subject. For example, if an image signal obtained by imaging at an imaging frame rate is reproduced at a frame rate lower than the imaging frame rate, a slow reproduction image in which the movement of the subject is slower than the actual speed can be obtained.

  Further, in the imaging apparatus, as in the video camera of Patent Document 1, the same image is output continuously for two fields, thereby extending the charge accumulation period to about two field periods or reading from the imaging device for each field. The image signal of the previous field is stored in the memory, and the average value is output as the image signal in addition to the stored image signal and the current image signal, thereby generating a slow shutter image. It has been broken.

JP-A-6-90402

  By the way, if the same image is output continuously for two fields as described above, a slow shutter image with smooth movement cannot be obtained. In addition, when reading from the image sensor for each field and outputting an average value as an image signal in addition to the image signal of the previous field and the current image signal, an image obtained by averaging two fields as a slow shutter image Can only get. Furthermore, the image signal generated by the video camera is fixed to the imaging frame rate of the imaging device.

  Therefore, the present invention provides an imaging apparatus and an imaging method capable of generating a shutter image having a smooth motion and a desired shutter speed, and a captured image having a frame rate different from the imaging frame rate.

  An imaging device according to the present invention stores an image sensor that generates an image signal of a captured image, a conversion unit that converts the image signal generated by the image sensor into a digital image signal, and an image signal obtained by the conversion unit A memory control unit, a memory control unit for controlling writing of an image signal to the memory and a reading of the image signal from the memory, and a signal processing unit for performing a filtering process using the image signal read from the memory by the memory control unit. The signal processing means performs a filtering process using the image signals of a plurality of frames read by the memory control means, and generates a shutter image that preserves the time resolution.

  In addition, an image sensor that generates an image signal of a captured image, a conversion unit that converts the image signal generated by the image sensor into a digital image signal, a memory that stores the image signal obtained by the conversion unit, and a memory A memory control unit that controls writing of the image signal and reading of the image signal from the memory, and a signal processing unit that performs a filtering process using the image signal read from the memory by the memory control unit. Filtering processing is performed to generate an image signal having a higher frame rate than the image signal generated by the image sensor.

  An imaging method according to the present invention includes an imaging process for generating an image signal of a captured image, a conversion process for converting the image signal generated in the imaging process into a digital image signal, and a memory for the image signal obtained in the conversion process A memory control process for controlling writing to and reading of an image signal from the memory, and a signal processing process for performing a filtering process using the image signal read from the memory in the memory control process. In the signal processing process, Filtering processing is performed using the image signals of a plurality of frames read out by the memory control process, and a shutter image storing time resolution is generated.

  In addition, an imaging process for generating an image signal of a captured image, a conversion process for converting the image signal generated in the imaging process into a digital image signal, writing of the image signal obtained in the conversion process into a memory, and the memory A memory control process for controlling reading of an image signal from the memory, and a signal processing process for performing a filtering process using the image signal read from the memory by the memory control process. In the signal processing process, an imaging process is performed by performing a filtering process. An image signal having a frame rate higher than that of the image signal generated in (1) is generated.

  In the present invention, the image signal of the captured image is written in the memory, the image signal written in the memory is read and supplied to the filter, and the filtering process of the image signals of a plurality of frames read from the memory is performed. A shutter image storing the time resolution is generated. In the generation of the shutter image, the tap range corresponding to the shutter speed is moved according to the frame rate of the shutter image, and image signals of a plurality of frames within the tap range are read from the memory, and filtering processing is performed. . Further, when the frame rate of the shutter image is “1 / m times (m: positive integer)” of the imaging frame rate, this tap range is shifted by m frames. Further, when the imaging frame rate is FA [frames / second] and the shutter speed is SS [seconds], a range including at least captured images for “ceil (FA × SS)” frames is set as a tap range. In the filtering process, an image signal having a frame rate higher than the imaging frame rate is generated.

  According to the present invention, a signal filtering process is performed using the image signals of a plurality of frames read from the memory, and a shutter image in which time resolution is stored is generated. For this reason, before recording the captured image when the shutter operation is performed on the recording medium, it can be confirmed using, for example, an electronic viewfinder. Moreover, a relatively bright photographed image can be obtained even in a dark place.

  Further, the tap range corresponding to the shutter speed is moved according to the frame rate of the shutter image, and image signals of a plurality of frames within this tap range are read out and subjected to filtering processing. When the frame rate of the shutter image is “1 / m times (m: positive integer)” of the imaging frame rate of the generated image signal, the tap range is shifted by m frames. Therefore, it is possible to obtain a shutter image having a shutter time longer than the frame interval of the display frame rate with time resolution of the display frame rate. In addition, a shutter image with smooth movement can be obtained.

  Furthermore, when the imaging frame rate of the image signal is FA [frames / second] and the shutter speed is SS [seconds], a range including at least a captured image of “ceil (FA × SS)” frames is set as the tap range. Therefore, it is possible to obtain a shutter image having a desired shutter speed in which various filter processes and the like are performed using captured images of frames longer than the shutter time.

  Further, since an image signal having a frame rate higher than that of the image signal can be generated by performing the filtering process, an image signal of a higher-speed captured image can be generated.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 10.

  The imaging unit 11 generates an imaging signal at an imaging frame rate using, for example, a CMOS type or CCD type solid-state imaging device. The imaging frame rate is set to a frame rate equal to or higher than a recording frame rate when recording a captured image on a recording medium or a display frame rate of an electronic viewfinder display image.

  FIG. 2 is a block diagram illustrating a configuration of the imaging unit 11 configured using, for example, a column amplifier type CMOS solid-state imaging device. For example, the imaging unit 11 outputs a pixel signal of four pixels in parallel at one clock, thereby realizing an imaging frame rate four times that of an imaging unit that outputs a pixel signal of one pixel at one clock.

  The vertical scanning control circuit 111 selects a line from which a pixel signal is read. The horizontal scanning control circuit 112 drives the pixel column selection circuit 113 to select a pixel position in the horizontal direction for reading out a pixel signal. The pixel column selection circuit 113 includes a switch group 113sw in which one terminal is connected to a pixel column in a direction orthogonal to the line direction and the other terminal is connected to the output amplifier 114.

  The output amplifier 114 is provided in parallel according to the number of pixel signals output in one clock, and the switch group 113sw of the pixel column selection circuit 113 is distributed and connected to each output amplifier 114. For example, when four pixel signals are output in parallel in one clock, four output amplifiers 114-1 to 114-4 are provided in parallel. Further, the output amplifier 114-1 is connected to a switch 113sw- (4L + 1) connected to the pixel row positioned at the “4L + 1 (L is 0 or positive integer)” position. Similarly, the output amplifiers 114-2 to 114-4 are provided with switches 113sw- (4L + 2) to 113sw- (4L + 4) connected to the pixel rows positioned in the “4L + 2” to “4L + 4” positions, respectively. Connecting.

  Here, the vertical scanning control circuit 111 reads the pixel signal of the pixels on the first line, and the horizontal scanning control circuit 112 simultaneously turns on the switches 113sw-1 to 113sw-4 of the pixel column selection circuit 113. At this time, the pixel signals Sp- (1,1) to Sp- (4,1) of the pixels P (1,1) to P (4,1) are output in parallel from the output amplifiers 114-1 to 114-4. Is output. The horizontal scanning control circuit 112 simultaneously turns on the switches 113sw-5 to 113sw-8 of the pixel column selection circuit 113 at the next clock. At this time, the pixel signals Sp- (5,1) to Sp- (8,1) of the pixels P (5,1) to P (8,1) are output from the output amplifiers 114-1 to 114-16. The Similarly, by repeatedly outputting pixel signals in parallel in units of four pixels, an image signal SA having a frame rate four times that of an imaging unit that outputs a pixel signal of one pixel in one clock can be output.

  In addition, as shown in FIG. 3, for example, an imaging unit 11a may be configured using a column A / D conversion type CMOS solid-state imaging device to output a digital image signal DA. In this case, the A / D conversion processing unit 113ad that performs analog-digital conversion is provided on the signal input side or signal output side of the switch 113sw of the pixel column selection circuit 113. The A / D conversion processing unit 113ad converts an analog pixel signal read from each pixel into a digital pixel signal and supplies the digital pixel signal to the output amplifier 114. The pixel signal supplied to the A / D conversion processing unit 113ad is a pixel signal of pixels that are continuous in the direction orthogonal to the line direction. On the other hand, the image signal SA output in parallel from the output amplifier 114 in FIG. 2 is a continuous signal of pixels separated by four pixels in the horizontal direction. For this reason, if the analog-digital conversion is performed after removing the aliasing components of the image signals SA output in parallel through the pre-filters, the filter processing result may affect the pixel positions separated according to the number of parallel outputs. There is. However, if a pre-filter (not shown) is provided in the A / D conversion processing unit 113ad, pixel signals of pixels that are continuous in the direction orthogonal to the line direction are supplied, so that the pre-filter is a separated pixel. The aliasing component can be satisfactorily removed without affecting the position.

  The pre-processing unit 13-1 performs gain adjustment and black level adjustment of the image signal SA-1 output from the output amplifier 114-1 of the imaging unit 11, and supplies the result to the A / D conversion processing unit 14-1. The A / D conversion processing unit 14-1 converts the image signal supplied from the pre-processing unit 13-1 into a digital image signal DB-1, and supplies it to the memory control unit 15. The pre-stage processing units 13-2 to 13-4 and the A / D conversion processing units 14-2 to 14-4 perform the same processing as the pre-stage processing unit 13-1 and the A / D conversion processing unit 14-1. The obtained digital image signals DB-2 to DB-4 are supplied to the memory control unit 15. The A / D conversion processing units 14-1 to 14-4 may remove the aliasing components.

  In the case of using the image signals DA-1 to DA-4 from which the aliasing components are output that are output from the imaging unit 11a illustrated in FIG. 3, the pre-processing units 13-1 to 13-4 are used as the image signals DA-1 to DA-. 4 is used to perform gain adjustment and black level adjustment, and the adjusted image signals are supplied to the memory control unit 15 as image signals DB-1 to DB-4.

  In addition, the pre-stage processing units 13-1 to 13-4 make it possible to control the adjustment amount of gain adjustment and black level adjustment independently by each pre-stage processing unit. In this way, by making the adjustment amount controllable independently, each image signal can be adjusted correctly even if the signal levels of the image signals output in parallel from the imaging unit 11 (11a) vary.

  The memory control unit 15 stores the supplied digital image signal DB in the memory 16. Further, the image signal stored in the memory 16 is read and supplied to the multiplexer 18.

  FIG. 4 is a block diagram showing a configuration of the memory control unit 15. The memory control unit 15 includes a timing signal generation unit 151, a control information register 152, and a writing / reading processing unit 153. The timing signal generator 151 writes the supplied image signal DB in the memory 16 (hereinafter, the image signal written in the memory 16 is referred to as an image signal DC), or reads out the image signal DC written in the memory 16. A timing signal TM serving as a reference for outputting (hereinafter, an image signal read from the memory 16 is referred to as an image signal DD) is generated. The timing signal TM is generated based on a clock signal TSck and synchronization signals TS-m and TS-c supplied from a synchronization signal generator 31 described later. The control information register 152 is connected to an operation control unit 35 to be described later, and holds control information supplied from the operation control unit 35, information on the configuration of the memory 16, the operation state of the write / read processing unit 153, and the like.

  The write / read processing unit 153 generates a write control signal WC and a read control signal RC based on the timing signal TM generated by the timing signal generation unit 151 and the control information JH held in the control information register 152. By supplying to the memory 16, an image signal is written into a desired area of the memory unit 6 or an image signal is read from the desired area of the memory 16. FIG. 4 shows a case where four image signals DD-1 to DD-4 are output in parallel as the image signal DB as the image signal DD.

  Further, the writing / reading processing unit 153 has a buffer (not shown) that temporarily stores an image signal to be written to the memory 16 and an image signal read from the memory 16. For this reason, the image signal DB is temporarily stored in the buffer even if the timing at which the image signal DB at the imaging frame rate is supplied and the timing at which the image signal DB is written to the memory 16 do not match. 16 can be written. Also, when the image signal is read out from the memory 16 and output, the read image signal is temporarily stored in the buffer, so that even if the image signal cannot be read out at the timing of the desired frame rate, the desired signal is output. The frame rate image signal DD can be output from the memory control unit 15. For example, an image signal can be output at a display frame rate or a recording frame rate.

  Thus, by storing the image signal of the captured image in the memory 16, the frame rate of the image signal DB supplied to the memory control unit 15 and the frame rate of the image signal DD output from the memory control unit 15 are made independent. Can be.

  In addition, a digital signal processing unit 154 is provided, performs filtering processing on the image signal output from the writing / reading processing unit 153 as necessary, and supplies the image signal after filtering processing to the multiplexer 18. The digital signal processing unit 154 is not limited to being provided in the memory control unit 15 as shown in FIG. 4, and may be provided in the VF process unit 21 or the main line process unit 23.

  The memory 16 stores an image signal with a high imaging frame rate or writes and reads a signal so that the image signal can be read out and output at a desired frame rate while the image signal with the imaging frame rate is written to the memory 16. A memory that can be performed at high speed is used. For example, it is configured using a DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory) capable of performing signal writing and reading at both rising and falling of the clock signal.

  The multiplexer 18 time-division multiplexes the image signals DD-1 to DD-4 read from the memory 16 to generate an image signal in units of frames. Here, when the memory control unit 15 reads the image signal from the memory 16 so as to generate an image signal at a display frame rate to display the captured image, the multiplexer 18 displays the generated frame-unit image signal. The image signal DE is supplied to the VF process unit 21. Further, when the memory control unit 15 reads the image signal from the memory 16 so as to generate an image signal at a recording frame rate for recording the captured image, the multiplexer 18 converts the generated image signal in units of frames into an image. The signal DF is supplied to the main line processing unit 23.

  The VF process unit 21 includes a pixel number conversion circuit according to the capability of the connected electronic viewfinder 41, an edge enhancement processing circuit for making the focus easy to understand, and a zebra mix for superimposing a mark on a signal of a predetermined video level. It is composed of a box cursor display circuit showing area information such as a circuit and an effective image frame. The VF process unit 21 performs various processes for assisting the photographer in the recording mode or the standby mode which is a recording standby state. The VF process unit 21 performs various signal processing on the image signal DE by the above-described circuit, and supplies the obtained display signal DG to the D / A converter 22. The D / A converter 22 converts the display signal DG into an analog display signal Vvf and supplies it to the electronic viewfinder 41. The electronic viewfinder 41 displays an image being captured or a captured image stored in the memory 16 based on the supplied display signal Vvf.

  The main line processing unit 23 shoots with a detection circuit for exposure control, an edge enhancement processing circuit for image creation, a linear matrix circuit for color adjustment, a gamma correction circuit for correcting monitor gamma, and a recording device 42. It is composed of a YC matrix processing circuit as an interface for recording images. The main line processing unit 23 performs various signal processing on the image signal DF using the above-described circuit, and supplies the obtained video output signal Vout to the recording device 42. The recording device 42 records the supplied video output signal Vout on a recording medium such as a tape or disk.

  The synchronization signal generator 31 generates a clock signal TSck and supplies it to each unit. In addition, a synchronization signal TS-m serving as a reference for generating and processing an image signal of a display frame rate or a recording frame rate is generated, and a process unit provided at the subsequent stage of the memory control unit 15 and the memory control unit 15 Etc. Further, a synchronization signal TS-c serving as a reference for generating and processing an image signal at the imaging frame rate is generated, and the memory controller 15, the drive signal generator 32, and the upstream processor of the upstream of the memory controller 15 13 and A / D conversion processor 14.

  The drive signal generation unit 32 generates a drive signal RD based on the clock signal TSck and the synchronization signal TS-c supplied from the synchronization signal generation unit 31, supplies the drive signal RD to the imaging unit 11 (11a), and an image signal at the imaging frame rate. The imaging unit 11 (11a) is driven so as to generate SA (DA).

  The operation control unit 35 is configured using a CPU (Central Processing Unit), and generates a control signal CS based on an operation signal PS corresponding to a user operation from a user interface unit 36 connected to the operation control unit 35. . By supplying the generated control signal CS to each unit, the imaging apparatus is operated according to a user operation. For example, when the operation mode is set to the standby mode, which is a recording standby state, the operation control unit 35 causes the electronic viewfinder 41 to display an image captured by the imaging unit 11 (11a) in real time so that focus or exposure is performed. Adjustment and angle of view setting. When the operation mode is set to the recording mode, the image captured by the imaging unit 11 (11a) is supplied to the recording device 42.

  Next, the operation of the imaging apparatus will be described. When the solid-state imaging device of the imaging unit 11 (11a) has a so-called HD size of 2200 pixels in the horizontal direction and 1125 lines in the vertical direction, for example, this solid-state imaging device is driven at a clock frequency of 74.25 MHz, When the pixel signal is read, the imaging frame rate is 30 frames / second.

  Here, when pixel signals of four pixels are read in parallel in one clock as described above, pixels for one line can be read in (2200/4) = 550 clocks. That is, the imaging frame rate is (30 × 4) = 120 frames / second.

  In addition, when the word length of the image signal DB supplied to the memory control unit 15 is 16 bits, the bus width between the memory control unit 15 and the memory 16 is 64 bits, and the memory 16 is composed of a single port memory, the image signal is written. When reading is simultaneously performed, the memory 16 is accessed at 148.5 MHz (the overhead for accessing the memory is ignored). That is, the writing and reading of the image signal each use a band of 74.25 MHz × (4 × 16) bits.

  On the other hand, if the electronic viewfinder 41 has an HD size image display function of 2200 pixels in the horizontal direction and 1125 lines in the vertical direction and displays images at 30 frames / second, the display frame rate is 30 frames / second. A band of 74.25 MHz × 16 bits may be allocated to the output of the image signal DE. Therefore, a band of 74.25 MHz × (64-16) bits can be allocated to the output of the image signal DF. That is, the multiplexer 18 selects an image signal read from the memory 16 using a band of 74.25 MHz × 16 bits to display a captured image, and converts the image signal DE of 30 frames / second into a VF process unit. 21. Further, the multiplexer 18 selects an image signal read from the memory 16 using a band of 74.25 MHz × 48 bits in order to record the captured image, and converts the image signal DF of the recording frame rate into the main line processing unit 23. Can supply.

  FIG. 5 shows a synchronization signal when the imaging frame rate is 120 frames / second and the display frame rate is 30 frames / second as described above. In FIG. 5, FIG. 5A is a synchronization signal TS-cf indicating a frame period at the imaging frame rate, FIG. 5B is a synchronization signal TS-cl indicating a line period at the imaging frame rate, and FIG. 5C is a display frame rate. The synchronization signal TS-mdf indicating the frame period and FIG. 5D respectively show the synchronization signal TS-mdl indicating the line period at the display frame rate.

  When the operation mode is set to the standby mode, the memory control unit 15 performs frame thinning of the image signal DB of the captured image and writes it in the memory 16 as an image signal having a display frame rate different from the captured frame rate. In addition, the memory control unit 15 reads the image signal so that an image signal having a display frame rate can be generated from the memory 16, and supplies the read image signal DD to the multiplexer 18. The multiplexer 18 selects the image signal DD, generates an image signal DE having a display frame rate, and supplies the image signal DE to the VF process unit 21. The VF process unit 21 generates a display signal DG based on the image signal DE, supplies the display signal DG to the D / A converter 22, and supplies the display signal Vvf to the electronic viewfinder 41. For this reason, in the standby mode, the electronic viewfinder 41 can confirm what kind of image is captured by the imaging unit 11 (11a), and adjustment and image adjustment such as focus and exposure can be performed before recording. Corner setting and the like can be performed.

  When the recording mode is set, the memory control unit 15 writes the image signal DB of the captured image in the memory 16. Further, the memory control unit 15 reads the image signal so that an image signal having a recording frame rate different from the imaging frame rate can be generated from the memory 16, and supplies the read image signal DD to the multiplexer 18. The multiplexer 18 selects the image signal DD, generates the image signal DE of the recording frame rate, and supplies it to the main line processing unit 23. Therefore, the video output signal Vout output from the main line process unit 23 can be supplied to the recording device 42 to record the captured image at the recording frame rate.

  Next, the operation when the operation mode is the standby mode will be described with reference to FIG. In this case, the memory control unit 15 writes the image signal DB of the captured image KDB (FIG. 6A) having an imaging frame rate of 120 frames / second to the memory 16 while writing the captured image KDC (FIG. 6B) stored in the memory 16. The image signal DC is read out and supplied to the multiplexer 18 so that an image signal having a recording frame rate can be generated. The multiplexer 18 generates an image signal DE of the captured image KDE (FIG. 6C) at the display frame rate and supplies it to the VF process unit 21. Here, since the imaging frame rate is 120 frame rates / second and the display frame rate is 30 frames / second, a process of thinning out three frames of the captured image after reading out one frame of the captured image (three-frame thinning process) is performed. If the image signal DC is repeatedly read from the memory 16, the image being captured can be displayed in real time at a display frame rate of 30 frames / second.

  Note that the thinning of the image signal in the memory control unit 15 is performed based on, for example, control information supplied as one of the control signals CS from the operation control unit 35, or one of the control signals CS from the operation control unit 35. It may be performed based on the gate signal supplied as one.

  FIG. 7 is a diagram for explaining the operation when the operation mode is the recording mode. In this case, the memory control unit 15 sequentially writes the image signal DB of the captured image KDB (FIG. 7A) having an imaging frame rate of 120 frames / second into the memory 16 as the image signal DC of the captured image KDC (FIG. 7B). The image signal of the captured image KDC stored in the memory 16 is read out and supplied to the multiplexer 18 so that an image signal of the display frame rate and an image signal of the recording frame rate can be generated. The multiplexer 18 selects the read image signal so as to generate the display frame rate image signal, and generates the image signal DE of the captured image KDE (FIG. 7C) having a display frame rate of 30 frames / second. . Further, the read image signal is selected so that an image signal of the display frame rate can be generated, and the image signal DF of the captured image KDF (FIG. 7D) with the maximum recording frame rate of 90 frame rate / second is generated. .

  Here, in order to display the image being captured by the electronic viewfinder 41 in real time, when the image signal DC is read from the memory 16, the memory control unit 15 thins the image signal DC in units of frames according to the display frame rate. Read out. That is, since the imaging frame rate is 120 frame rates / second and the display frame rate is 30 frames / second, if the image signal DC is sequentially read out by repeating the three-frame thinning process, the recording mode and recording standby In the standby mode, which is a state, an image being captured can be displayed in real time at a display frame rate of 30 frames / second.

  In the recording mode, when all the captured images are recorded by the recording device 42 without thinning out, the pre-recording image signal DC-ur (FIG. 7E) stored in the memory 16 is written and read out of the captured image. When the recording frame rate when performing the above is 90 frames / second, it increases at a rate of 30 frames / second. Therefore, the memory control unit 15 stops writing the image signal DB when the pre-recording image signal DC-ur reaches the predetermined amount ML frame. By controlling the writing operation of the image signal DB in this way, in order to increase the utilization efficiency of the memory 16, the newly captured image KDB is sequentially stored in the storage position of the oldest captured image KDB, and the storage area is cyclically stored. When used for recording, it is possible to prevent the captured image before recording from being rewritten with a new captured image and recording the discontinuous captured image in the recording device 42. When the storage area is used cyclically, if the recording frame rate is increased, the difference from the imaging frame rate is reduced, so that the time until the writing of the image signal DB is stopped can be increased.

  Here, as described above, when a captured image having an imaging frame rate of 120 frames / second is recorded without being thinned out, and the recorded captured image is reproduced at, for example, 30 frames / second, a 1/4 × speed slow reproduction image is obtained. Can be displayed.

  By the way, when recording the video output signal Vout by the recording device 42, when there is a sufficient bandwidth in reading the image signal from the memory 16, the image signal of the captured image can be read at a frame rate higher than the desired frame rate. it can. For example, a 74.25 MHz × (4 × 16) bit band for writing the image signal DB and an output of the image signal DE which is 30 frames / second is a 74.25 MHz × 16 bit band, 30 frames / second. If a band of 74.25 MHz × 16 bits is assigned to the output of the image signal DF, a margin of 74.25 MHz × 32 bits is generated. For this reason, the image signal is read by further using the band of 74.25 MHz × 32 bits, an image signal having a frame rate higher than the desired frame rate is generated, and the filtering process is performed. It is possible to generate an image signal with a frame rate of.

  FIG. 8 is a diagram for explaining the operation when the filtering process is performed. The memory control unit 15 writes the image signal DB of the captured image KDB (FIG. 8A) having an imaging frame rate of 120 frames / second in the memory 16 as the image signal DC of the captured image KDC (FIG. 8B). Further, while writing the signal, the image signal DC of the captured image KDC stored in the memory 16 is read by repeating the three-frame thinning process so that an image signal having a display frame rate of 30 frames / second can be generated. The read signal passes through the digital signal processing unit 154 and is supplied to the multiplexer 18. The multiplexer 18 generates an image signal DE of the captured image KDE (FIG. 8C) at the display frame rate and supplies it to the VF process unit 21.

  Further, the memory control unit 15 reads the captured image KDC stored in the memory 16 so that an image signal with a frame rate of 90 frames / second can be generated, and the image signal of the captured image KDF ′ (FIG. 8D) is obtained. The image signal DD is supplied to the digital signal processing unit 154. The digital signal processing unit 154 includes, for example, a transversal filter circuit, and the filter coefficients z1, z2, and z3 are set to “1, 2, 1” so as to perform weighted addition of three frames. For this reason, the digital signal processing unit 154 performs a filtering process of weighting and adding the image signal DD of the captured image KDF ′ for 3 frames for each pixel using the captured image KDF ′ for 3 frames as a tap range. An image signal of the captured image is generated and supplied to the multiplexer 18. The multiplexer 18 uses the signal supplied from the digital signal processing unit 154 to generate the image signal DF of the captured image KDF (FIG. 8E) at the recording frame rate and supplies it to the main line processing unit 23.

  In this way, when the frame addition is performed to generate the image signal DF, the dynamic range is expanded, and a bright captured image can be recorded even if imaging is performed at 120 frames / second. Also, when the memory bandwidth is large, if the image signal is read out at 120 frames / second and the filtering process for adding the image signals for four frames is performed, the image is captured at the recording frame rate. A captured image having the same brightness and the like can be obtained. Furthermore, since the dynamic range is expanded by adding high-frame-rate captured images, the charge accumulated in the imaging unit is not saturated as in the case of imaging at a low frame rate, for example. A signal level image signal can be generated. Furthermore, when the signal level of the signal after the frame addition is compressed and the gain is maintained at a unity, the noise level of the random noise is compressed, so that a captured image with less random noise can be obtained.

  Here, when performing the frame addition process, the tap range corresponding to the shutter speed is moved according to the frame rate of the shutter image while the image signal of a plurality of frames within the tap range is read and the filtering process is performed. Thus, it is possible to generate a slow shutter image in which time resolution is preserved.

  FIG. 9 is a diagram for explaining a slow shutter operation that preserves time resolution. The memory control unit 15 stores the image signal DB of the captured image KDB (FIG. 9A) having an imaging frame rate of FA frames / second in the memory 16 while writing the image signal DB of the captured image KDC (FIG. 9B) to the memory 16. The image signal DC of the captured image KDC within the tapped range is read out.

  Here, when the shutter speed is SS seconds, the tap range is a range including at least “ceil (FA × SS)” frames. Note that “ceil (FA × SS)” indicates the smallest integer value that is equal to or greater than the multiplication result of “FA” and “SS”. In addition, when the frame rate of the shutter image is “1 / m times (m: positive integer)” of the imaging frame rate, the tap range is shifted by m frames to generate the shutter image. Do.

  For example, if the imaging frame rate is 120 frames / second as described above and the shutter speed is (1/30) seconds, the tap range is 4 frames. If the shutter image frame rate is 60 frames / second, the tap range is shifted by two frames.

  Therefore, when the memory control unit 15 supplies the image signal in the tap range to the digital signal processing unit 154 and performs the addition process, the image signal of the captured image KDF equivalent to the captured image with the shutter speed set to 1/30 seconds. DF can be obtained. Further, since the tap range is sequentially moved in accordance with the frame rate of the shutter image, the shutter image has a smooth motion and stores a time resolution of 60 frames / second. That is, it is possible to obtain a slow-motion slow-smooth image having a shutter opening period longer than the frame interval of the desired frame rate while maintaining the time resolution of the desired frame rate. Further, by making the tap range wider than the “ceil (FA × SS)” frame, it is possible to perform filtering processing using captured images before and after the shutter opening period. It is possible to obtain a good slow shutter image subjected to the filtering process.

  In this way, since an image signal having a desired shutter speed that preserves the time resolution can be generated, for example, when the shutter speed is slowed down so that a bright captured image can be obtained because the subject is dark, the shutter open period is a recording frame. Even when the frame interval of the rate is longer, the image signal DF having the recording frame rate can be generated at a desired shutter speed. In addition, noise can be reduced. Note that noise reduction and dynamic resolution are in a trade-off relationship. When the noise reduction effect is increased, the dynamic resolution is lowered, and when the dynamic resolution is increased, the noise reduction effect is lowered.

  Note that, when the image signal DF is generated by performing the filtering process, the image signal DE is multiplied by a coefficient corresponding to the filtering process so that the signal level matches the image signal DF. For example, when the image signal DF of the slow shutter image is generated by simply adding the captured images for four frames as described above in the filtering process, the signal level of the image signal DF is 4 of the signal level of the image signal DE. Doubled. Therefore, by multiplying the image signal DE by the coefficient “4”, the signal level of the captured image KDE is made equal to that of the captured image KDF. The adjustment of the signal level of the image signal DE may be performed by the digital signal processing unit 154, or may be performed by the VF process unit 21. Thus, if the signal level of the image signal DE is adjusted according to the filtering process, it is possible to prevent the captured image KDF and the captured image KDE displayed on the electronic viewfinder 41 from being greatly different.

  Furthermore, it is possible to generate an image signal DF of the captured image KDF having a frame rate higher than the imaging frame rate by performing frame up-conversion by applying filtering in the time axis direction. FIG. 10 is a diagram for explaining the frame up-conversion operation, and FIG. 10A shows the captured image KDB at the imaging frame rate. Here, the memory control unit 15 reads the image signal of the captured image from the memory 16 so that the captured image is included in the captured frame rate at a frame rate that is three times the captured frame rate. When the read image signal is subjected to a filtering process with a filter coefficient zc of, for example, [1, 2, 2, 1], a captured image KDF based on the image signal DF after the filtering process is as shown in FIG. When the captured image KDB is 120 frames / second, a captured image of 360 frames / second can be obtained as the captured image KDF.

  In the filtering process, not only can a shutter image storing time resolution be generated, but also other functions can be provided by changing filter coefficients. For example, if the number of frames to be added is 3 and the filter coefficient is [−1, 2, −1], it has a high-pass filter function, and only the high-frequency component in the time direction is output, A certain part can be extracted.

  In addition, when recording a captured image that has been subjected to filtering processing, if the image signal DE is generated in the same manner as the generation of the image signal DF when the operation mode is the standby mode, the captured image that has been subjected to filtering processing is recorded. Is displayed on the electronic viewfinder 41, and it is possible to confirm in advance what kind of captured image is recorded.

  If the same processing is possible in the recording mode and the standby mode, for example, the output of the imaging unit 11 always performs the filtering process according to the purpose while keeping the same conditions (exposure time, reading speed, etc.), and the filtering process in the time direction Alternatively, the captured image can be recorded at various frame rates by changing the number of frame additions.

  Furthermore, the filtering processing result obtained as described above can be written back to the memory 16 and the captured image can be corrected using the written image signal. For example, when a black image captured with the aperture closed is added to a plurality of frames and written to the memory 16 as a captured image correction signal, and when the image signal is read from the memory 16, the gain is adjusted and the captured image correction signal is read from the image signal. By subtracting, an image signal from which the influence of fixed pattern noise is removed can be obtained.

  By the way, in the recording mode, if the reading of the image signal from the memory 16 is performed by thinning out the frame in the same manner as in the standby mode, the image signal DF in which the movement speed of the subject is changed can be generated. For example, when the image signal DC is read out by repeating the pulling process for 3 frames to generate the image signal DF, when the captured image recorded in the recording device 42 is reproduced at 30 frames / second, a 1 × speed reproduced image is obtained. be able to. In addition, when the image signal DC is read out every other frame and the image signal DF is generated, when a captured image recorded in the recording device 42 is reproduced at 30 frames / second, a reproduction image at 1/2 times speed can be obtained. it can. Further, when performing frame thinning, if a plurality of consecutive frame image signals are read out and subjected to the filtering process as described above to generate one frame image signal, the filtering process after changing the speed of movement of the subject is performed. An image signal DF can be generated.

  Further, when the image signal is read from the memory 16 by switching the reading speed and the image signal is read, it is possible to further reduce the sense of incongruity at the changing portion of the reproduction speed.

  FIG. 11 is a diagram for explaining the operation when the reading speed is switched. FIG. 11 shows a case where the reading speed is switched from the 1 × speed playback speed to the 1/4 × speed slow playback image and then returned to the 1 × speed playback speed step by step.

  Here, from the image signal DC of the captured image KDC (FIG. 11A) that is 120 frames / second stored in the memory 16, for example, 30 frames / second can be obtained so that a captured image at a playback speed of 1 × speed can be obtained. The captured image corresponding to the readout speed is read out, and the image signal DF of the captured image KDF (FIG. 11B) at 30 frames / second is generated. That is, the image signal DC is read from the memory 16 while thinning out the three frames, and the image signal DF is generated. Next, the readout speed is switched, and the captured image corresponding to the readout speed of 60 frames / second is read out. That is, the image signal DC is read out from the memory 16 while thinning out every other frame to generate the image signal DF. Further, the readout speed is switched to read out the captured image corresponding to the readout speed of 120 frames / second, and the image signal DC is read out from the memory 16 without performing frame thinning, and the image signal DF is generated. Thereafter, the reading speed is switched between 60 frames / second and 30 frames / second.

  When the readout speed is switched in this way, the speed when the captured image recorded by the recording device 42 is reproduced is as follows: 1 × speed playback → 1/2 × speed playback → 1/4 × speed playback → 1/2 × speed playback → 1 × speed When switching from 1 × speed playback to 1/4 × speed playback, the playback speed is switched step by step to obtain a playback image with less discomfort at the part where the frame rate is switched. it can. Also in this case, as described above, if a plurality of consecutive frames of image signals are read out and subjected to filter processing to generate one frame of image signal, a sense of incongruity in the portion where the playback speed has changed is more enhanced in the captured image after the filter processing. Can be reduced.

  It should be noted that the frame rate, image size, and the like of the above-described form are shown as examples for ease of understanding, and of course are not limited to the frame rate, image size, and the like.

  As described above, the imaging apparatus and imaging method according to the present invention are useful when performing a high-speed imaging operation, and are suitable for performing a high-speed imaging operation to generate a shutter image or the like having a desired shutter speed.

It is a figure which shows the structure of an imaging device. It is a figure which shows the structure of an imaging part. It is a figure which shows the other structure of an imaging part. It is a figure which shows the structure of a memory control part. It is a figure which shows a synchronizing signal. It is a figure for demonstrating operation | movement in standby mode. It is a figure for demonstrating operation | movement of video recording mode. It is a figure for demonstrating operation | movement when performing a filtering process. It is a figure for demonstrating slow shutter operation | movement. It is a figure for demonstrating a frame up-conversion operation | movement. It is a figure for demonstrating operation | movement when the reading speed is switched.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Imaging device 11, 11a ... Imaging part, 13 ... Pre-stage processing part, 14 ... A / D conversion processing part, 15 ... Memory control part, 16 ... Memory, 18 ... Multiplexer, 21 ... VF process section, 22 ... D / A converter, 23 ... Main line process section, 31 ... Synchronization signal generation section, 32 ... Drive signal generation section , 35... Operation control unit, 36... User interface unit, 41... Electronic viewfinder, 42... Recording device, 111 .. vertical scanning control circuit, 112. 113 ... Pixel column selection circuit, 113ad ... A / D conversion processing unit, 113sw ... switch, 114 ... output amplifier, 151 ... timing signal generation unit, 152 ... control information register, 153... Read / write processing unit, 15 4 Digital signal processing unit

Claims (10)

An image sensor for generating an image signal of the captured image;
Conversion means for converting an image signal generated by the image sensor into a digital image signal;
A memory for storing an image signal obtained by the conversion means;
Memory control means for controlling writing of image signals to the memory and reading of image signals from the memory;
Signal processing means for performing a filtering process using an image signal read from the memory by the memory control means;
The image processing apparatus, wherein the signal processing unit performs a filtering process using the image signals of a plurality of frames read by the memory control unit, and generates a shutter image in which time resolution is stored. 2. The memory control unit reads an image signal of a plurality of frames within the tap range while moving a tap range according to a shutter speed according to a frame rate of the shutter image. The imaging device described. When the frame rate of the shutter image is “1 / m times (m: a positive integer)” of the frame rate of the image signal generated by the image sensor, the memory control unit sets the tap range to m frames. The imaging apparatus according to claim 2, wherein the imaging apparatus is shifted and moved. When the frame rate of the image signal generated by the image sensor is FA [frames / second] and the shutter speed is SS [seconds], the memory control unit captures an image of “ceil (FA × SS)” frames. The imaging device according to claim 2, wherein a range including at least the tap range is set as the tap range. An image sensor for generating an image signal of the captured image;
Conversion means for converting an image signal generated by the image sensor into a digital image signal;
A memory for storing an image signal obtained by the conversion means;
Memory control means for controlling writing of image signals to the memory and reading of image signals from the memory;
Signal processing means for performing filtering using the image signal read from the memory by the memory control means;
The image processing apparatus, wherein the signal processing unit generates an image signal having a frame rate higher than that of the image signal generated by the image sensor by performing a filtering process. An imaging process for generating an image signal of the captured image;
A conversion step of converting the image signal generated in the imaging step into a digital image signal;
A memory control step for controlling writing of the image signal obtained in the conversion step into the memory and reading of the image signal from the memory;
A signal processing step of performing a filtering process using the image signal read from the memory in the memory control step,
In the signal processing step, a filtering process is performed using the image signals of a plurality of frames read out in the memory control step, and a shutter image in which time resolution is stored is generated. 7. The memory control step of reading image signals of a plurality of frames within the tap range while moving the tap range according to the shutter speed according to the frame rate of the shutter image. The imaging method described. In the memory control step, when the frame rate of the shutter image is “1 / m times (m: positive integer)” of the frame rate of the image signal generated in the imaging step, the tap range is set to m frames. The imaging method according to claim 7, wherein the imaging method is shifted. In the memory control step, when the frame rate of the image signal generated in the imaging step is FA [frames / second] and the shutter speed is SS [seconds], a captured image of “ceil (FA × SS)” frames The imaging method according to claim 7, wherein a range including at least the tap range is set as the tap range. An imaging process for generating an image signal of the captured image;
A conversion step of converting the image signal generated in the imaging step into a digital image signal;
A memory control step for controlling writing of the image signal obtained in the conversion step into the memory and reading of the image signal from the memory;
A signal processing step of performing a filtering process using the image signal read from the memory by the memory control step;
In the signal processing step, a filtering process is performed to generate an image signal having a higher frame rate than the image signal generated in the imaging step.

Images