JP2019179951A - Imaging device - Google Patents

Abstract

To provide an imaging device that can detect an accurate motion vector value in a pre-read pixel and can feed back correct motion vector information.SOLUTION: An imaging device comprises: an imaging element that has a first pixel and a second pixel that can read while thinning out pixels as a thinning-out unit; imaging element control means that reads a signal from the second pixel first and subsequently reads a signal from the first pixel; pixel signal rearranging means that rearranges the signal from the first pixel and the signal from the second pixel into image signals in a pixel array configuration of the imaging element; first motion vector detection means that performs motion vector detection processing on the signal from the second pixel; and second motion vector detection means that performs motion vector detection processing on the pixel signals rearranged by image signal rearranging means. The first motion vector detection means is controlled on the basis of the thinning-out unit of the second pixel, and the second motion vector detection means is controlled on the basis of a result of detection performed by the first motion vector detection means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to motion vector detection.

  In order to perform camera shake correction on video shot using an imaging device such as a digital still camera or a digital video camera, the amount of motion between frame images is detected and alignment is performed on a plurality of images. There is a need. As a method for detecting the amount of motion between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of motion from a captured frame image, and the like.

  Various methods for estimating the amount of motion using a frame image have been proposed in the past. A typical example is motion vector detection by template matching. In template matching, first, one of two frame images in a video is used as an original image and the other is used as a reference image. Then, a rectangular area having a predetermined size arranged on the original image is used as a template block, and a correlation with a distribution of pixel values in the template block is obtained at each position of the reference image. At this time, the position where the correlation is highest in the reference image is the destination of the template block, and the direction and amount of movement to the destination when the position of the template block on the original image is used as a reference is the motion vector. .

  In recent years, with the increase in the number of pixels of an imaging apparatus such as a digital still camera or a digital video camera, motion vector detection by template matching has a problem that the amount of calculation at 1V becomes enormous due to expansion of the template size and search range.

  Patent Document 1 shows a technique for speeding up a ranging calculation for autofocus processing by first reading out pixels for focus detection in an image sensor and reading out the remaining main line image pixels later. It is a thing.

  By applying this technology, simple motion vector detection with a small amount of calculation is performed on the prefetched pixels in advance, and the result is fed back to the main motion vector detection processing using the main video signal. Thus, it is conceivable to reduce the amount of calculation of the main motion vector detection process by limiting the template size and the search range.

Japanese Patent Laid-Open No. 2006-72695

  When performing simple motion vector detection with pre-read pixels, the motion vector value is not accurate unless the template shape, size, and search range are changed in consideration of information lost by pixel thinning. Is required.

  Similarly, if the calculated motion vector value is not fed back to the main motion vector detection process in consideration of information lost due to pixel decimation, incorrect motion vector information may be fed back.

In order to solve the above problems, an imaging apparatus according to the present invention provides:
An imaging device having a first imaging pixel and a second imaging pixel that can be read by thinning out one or more pixels as a thinning unit, and that converts a subject image formed by an optical system into an image signal;
After the image sensor is exposed, an image sensor control unit that reads the signal of the second image pickup pixel first and reads the signals of the remaining first image pickup pixels that have not been read later; and the first image pickup pixel And a signal of the second imaging pixel are rearranged into an image signal having a pixel array configuration of the imaging device, and a motion vector detection process is performed on the signal of the second imaging pixel. A first motion vector detecting means; and a second motion vector detecting means for performing a motion vector detection process on the image signals rearranged by the image signal rearranging means, and the first motion vector detecting means The means is controlled based on the thinning unit of the second imaging pixel, and the second motion vector detection means is based on the motion vector detection result of the first motion vector detection means. Te, wherein the control is performed.

  According to the imaging apparatus of the present invention, it is possible to detect an accurate motion vector value for a pre-read pixel, and to feed back correct motion vector information to the motion vector detection process for the main line video signal.

1 is a block diagram of a digital camera that is an imaging apparatus according to an embodiment of the present invention. It is explanatory drawing of the read-out order of the image pick-up element 0103 which concerns on embodiment of this invention. It is explanatory drawing of the rearrangement control of the pixel rearrangement means 0106 which concerns on embodiment of this invention. It is a template arrangement example of the motion vector detection process with respect to the main line image which concerns on embodiment of this invention. It is a template arrangement | positioning example of the motion vector detection process with respect to the prefetch image which concerns on embodiment of this invention. It is a timing chart of the motion vector detection process which concerns on embodiment of this invention. It is an example of search range control of the motion vector detection process with respect to the main line image which concerns on embodiment of this invention.

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a block diagram of a digital camera which is an imaging apparatus according to the present invention.

  In FIG. 1, an optical system 0101 includes a lens, a diaphragm, and the like, and at the time of shooting, an optical system driving unit 0102 performs focus adjustment and exposure adjustment to form an optical image on an image sensor 0103. The image sensor 0103 has a photoelectric conversion function for converting an optical image into an electrical signal, and is configured by a CCD, a CMOS sensor, or the like.

  The image sensor 0103 reads out the pre-read pixels used in the pixel rearrangement unit 0105, the display image processing unit 0108, and the first motion vector detection unit 0110, and the pixel arrangement in the image sensor driving unit 0104 at the time of shooting. Control of readout of post-read pixels used in the replacement unit 0105 is performed.

  The pixel rearranging unit 0105 returns the pre-read pixel group and the post-read pixel group output from the image sensor 0103 to the original raster scan pixel configuration and writes them in the memory 0113.

  The main line image processing means 0107 performs various image processing such as noise reduction and development processing on an image having a raster scan pixel configuration (hereinafter referred to as a main line image) arranged in the memory 0113, The processed image is written back to the memory 0113. Thereafter, the main line image subjected to the main line image processing is transferred to the recording apparatus via the memory 0113.

  The display image processing unit 0108 performs various image processing such as noise reduction, development processing, and horizontal resizing processing on the prefetched pixel group output from the image sensor 0103, and writes the processed image in the memory 0113. Thereafter, the prefetched pixel group that has undergone display image processing is transferred to the display device and the first motion vector detecting means 0111 via the memory 0113.

  The main line image processing unit 0107 and the display image processing unit 0108 are different from each other in that the display image processing unit 0108 generates an image to be output to a display unit such as a panel or EVF. In contrast to 0107, the circuit configuration takes into consideration higher speed and power saving.

  The first motion vector detection unit 0110 performs a motion vector detection process using the prefetched pixel group output from the image sensor 0103 and the prefetched pixel group 1V before arranged in the memory 0113 as inputs. The motion vector detection result is acquired via the CPU 0112.

  The second motion vector detection unit 0111 performs a motion vector detection process using the main line image and the main line image 1V before arranged in the memory 0113 as inputs.

  The motion vector detection result is acquired via the CPU 0112. The CPU 0112 is configured by a microcomputer or the like that controls the entire imaging processing apparatus, and gives operation instructions to each functional block to execute various control processes.

  A memory (DRAM) 0113 is a memory for storing data, and has a sufficient storage capacity for storing a predetermined number of still images, moving image data for a predetermined time, constants for operating the CPU 112, programs, and the like. .

  Next, readout control of the image sensor 0103 and rearrangement processing of the pixel rearrangement unit 0105 will be described. As specific drive control of the image sensor 0103, for example, the configuration shown in the prior art document 1 and drive control can be considered. Here, a pixel reading order in the present invention will be described.

  FIG. 2 illustrates details of the pixel readout order.

  Reference numeral 0201 denotes a pixel constituting the image sensor 0103. A pixel row 0202 indicated by gray indicates a pre-read pixel, and a pixel row 0203 indicated by white indicates a post-read pixel. In the embodiment of the present invention, the prefetch pixels are in units of rows, and there are prefetch rows in a four-row cycle. (Row numbers 0000, 0004, 0008, 0012...) The image sensor 0103 reads the prefetch row 0202 according to a predetermined drive, and after reading all the prefetch rows 0202, reads the post-read row 0203. In the following description, an image composed of the prefetch line 0202 is referred to as a prefetch image, and an image composed of the postfetch line 0203 is referred to as a preread image.

  The pixel rearrangement unit 0106 performs writing control by adding an offset to the write address of the memory when writing the pre-read image and the post-read image in the memory 0113. FIG. 3 shows the concept of the memory address offset for the pre-read image and the post-read image by the pixel rearrangement unit 0106. As a concept of offset of the prefetch image, first, the prefetch pixel group of the row number 0000 is written in order from the start address.

  Subsequently, for the pre-read pixel group of the row number 0004, the write start of the row number 0004 is started by adding the write area for three rows of the row numbers 0001, 0002, and 0003 as the offset offset1 to the write end address of 0000. Start writing as an address. Subsequently, for the pre-read pixel group of row number 0008, writing is started with an address obtained by adding offset offset1 to the write end address of 0004 as the write start address of row number 0008. This procedure is performed until the input of the prefetch image is completed.

  As a way of thinking about the offset of the post-reading image, first, for the post-reading pixel group of the row number 0001, an address obtained by adding the already written area of the row number 0000 as the offset offset2 to the start address is sequentially written as a start address. Go.

  Subsequently, after-reading pixel groups of row numbers 0002 and 0003 are also written continuously. For the post-reading pixel group at the line number 0005, the address obtained by adding the offset offset2 to the write end address at the line number 0003 is sequentially written as a start address.

  Subsequently, post-reading pixel groups of row numbers 0006 and 0007 are also written continuously. This procedure is performed until the input of the post-reading image is completed. When all of the procedures are completed, the main line images are arranged on the memory 0113 in the raster scan order. The pixel rearrangement method may be a method in which the pixel rearrangement process is performed directly in the circuit as a configuration in which the pixel rearrangement unit 0113 includes a buffer (SRAM) for storing the pre-read image. In addition, when writing to the memory 0113, rearrangement is not performed, and when reading the main line image from the memory 0113, a method of reading in the raster scan order by adding an offset may be used.

  Next, processing and control of the first motion vector detection unit 0110 and the second motion vector detection unit 0111 will be described. Since the configuration of the motion vector detection units 0110 and 0111 is common, the second motion vector detection unit 0111 will be described.

  Based on the template arrangement determined by the CPU 0112, a motion vector between the two input frame images is detected. With respect to the motion vector detection method, a template matching method is used in the present embodiment. The image signal of the current V is the original image, and the image signal before 1 V is the reference image. FIG. 4 shows an example of the template arrangement for the main line image in the second motion vector detection means 0111. A template 0402 is arranged at an arbitrary position in the image 0401, and a correlation value between the template 0402 and each region of the reference image is calculated. Reference numeral 0403 denotes the center of gravity of the template. At this time, calculating the correlation value for the entire area of the reference image requires a large amount of calculation. Therefore, as shown in 0403, a rectangular area for calculating the correlation value on the reference image is actually searched for. Set as.

  Here, the position and size of the search range 0403 are not particularly limited, but a correct motion vector cannot be detected if the search range 0403 does not include an area corresponding to the movement destination of the template 0402. . In this embodiment, a sum of absolute difference (hereinafter abbreviated as SAD) is used as an example of a correlation value calculation method. The calculation formula of SAD is shown in Formula 1.

In Expression 1, f (i, j) represents a pixel value at the coordinates (i, j) in the template 0402, and g (i, j) represents a correlation value calculation target area in the search range 0403. Each pixel value is represented. The correlation value calculation target area is the same size as the template 0402.

  In SAD, the absolute value of the difference is calculated for each pixel value f (i, j) and g (i, j) in both blocks, and the correlation value S_SAD can be obtained by obtaining the sum. Therefore, the smaller the correlation value S_SAD, the smaller the difference in luminance value between the two blocks, that is, the similarity between the template 202 and the texture in the correlation value calculation area.

  In this embodiment, SAD is used as an example of the correlation value, but the present invention is not limited to this, and other correlation values such as sum of squares of differences (SSD) and normalized cross correlation (NCC) are used. Also good.

  Regarding the template arrangement, for example, as shown in FIG. 4, the template 0402 has a horizontal size of TX pixels and a vertical size of TY pixels, and the motion vector search range 0403 has a horizontal SX pixel and a vertical SY pixel. The distance between the templates 0402 is set by horizontal DX pixels and vertical DY pixels, and a frame is arranged on the lattice. Further, the templates may be arranged in a staggered pattern so that a wide range of motion vectors can be observed with a small number of templates. Parameters such as TX, TY, DX, DY, SX, and SY can be changed to arbitrary values according to the purpose. In the present embodiment, the template arrangement is fixed, but the template position (template centroid 0403) as shown in Japanese Patent Application Laid-Open No. 2017-72913 may be determined based on the image feature point analysis result. .

  FIG. 5 shows a template arrangement example for the prefetched image in the first motion vector detecting means 0111. In the present embodiment, the prefetched image has a vertical size of 1/4 with respect to the main line image. Since the pixels are thinned out in the vertical direction, the setting in the vertical direction is changed according to the thinning amount with respect to the setting of the main line image. In the present embodiment, the template vertical size TY ′, the vertical search SY ′, and the inter-template distance DY ′ are set to ¼. With this setting, the motion vector detection setting has the same effect as the main line image. In the present embodiment, the fixed template arrangement is set in accordance with the motion vector detection setting of the main line image. However, the template arrangement may be determined based on the feature point analysis result of the prefetched image. Further, for the purpose of reducing the amount of calculation, a template arrangement different from the main line image motion vector detection setting may be used.

  Next, control of motion vector detection processing in this embodiment will be described. FIG. 6 shows a timing chart of motion vector detection processing in the present embodiment. When the motion vector detection processing of the first motion vector detection unit 0110 for the pre-read image is completed, the CPU 0112 acquires a first motion vector detection result (vector value). The CPU 0112 determines the setting of the second motion vector detection unit 0111 based on the first motion vector detection result. Since the first motion vector detection result is processed by the thinned-out image signal, the present embodiment is compared with the second motion vector detection unit 0111 in FIGS. 7 (a) and 7 (b). Feedback is implemented in any of the three forms of (c) or a combination thereof. In the present embodiment, the search range is controlled to reduce the amount of calculation for template matching.

  Only the direction of the motion vector value in the horizontal direction (sign of the motion vector value) obtained by the first motion vector detection means is referred to. The second motion vector detection means 0111 does not search for a direction opposite to the horizontal direction obtained from the first motion vector detection result. For example, the template 0702 has a search range in which the left direction is not searched because the first motion vector detection result is the right direction.

  The horizontal motion vector value obtained by the first motion vector detection means is directly used by the CPU for image stabilization control, and the second motion vector detection means searches only in the vertical direction.

  Reference is made to the direction of the motion vector values in both the horizontal and vertical directions (sign of the motion vector value) obtained by the first motion vector detection means. The second motion vector detection unit 0111 does not search for a direction opposite to the horizontal direction / vertical direction obtained from the first motion vector detection result. For example, the template 0702 has a search range in which the left direction and the upward direction are not searched because the first motion vector detection result is the right direction and the downward direction.

  In the search range of each template, as an index for performing a composite process such as switching between (a), (b), and (c), low contrast determination or repeated subject determination obtained by the first motion vector detection means 0110 It is conceivable to use the reliability (unsatisfactory) determination result such as.

  In this embodiment, the feedback target is limited to the search range setting of the second motion vector detecting unit 0112. However, the feedback may be fed back to the settings such as the template size and the template arrangement.

  As described above, according to the present embodiment, the setting of the first motion vector detection unit is appropriate according to the thinning configuration of the prefetched image. Furthermore, the amount of calculation of the template matching process is appropriately reduced by narrowing down the information and feeding it back to the second motion vector detecting means.

  In the present invention, the two motion vector detection means may be configured by one motion vector detection means, and the motion vector detection process for the prefetched image and the motion vector detection process for the main line image may be processed in a time division manner. Good.

  In the present invention, the pre-read pixel group output from the image sensor 0103 is a display image signal, but may be a focus detection signal.

0101 Optical system, 0102 Optical system driving means, 0103 Image sensor

Claims (9)

An imaging device having a first imaging pixel and a second imaging pixel that can be read by thinning out one or more pixels as a thinning unit, and that converts a subject image formed by an optical system into an image signal;
After the image sensor is exposed, an image sensor control unit that first reads out the signal of the second image pickup pixel and reads out the signal of the remaining first image pickup pixel that has not been read out;
Pixel signal rearranging means for rearranging the signal of the first imaging pixel and the signal of the second imaging pixel into an image signal having a pixel arrangement configuration of the imaging element;
First motion vector detection means for performing motion vector detection processing on the signal of the second imaging pixel;
A second motion vector detecting means for performing a motion vector detection process on the image signals rearranged by the image signal rearranging means;
The first motion vector detection means is controlled based on a thinning unit of the second imaging pixel,
2. The imaging apparatus according to claim 1, wherein the second motion vector detection unit is controlled based on a motion vector detection result of the first motion vector detection unit. The first motion vector detection unit adjusts at least one of a template size, a search range, and a template arrangement position of template matching processing based on a thinning unit of the second imaging pixel. Item 2. The imaging device according to Item 1. The second motion vector detection means determines at least one of a template size, a search range, and a template arrangement position for template matching processing based on the direction of the motion vector obtained by the first motion vector detection means. The imaging apparatus according to claim 2, wherein adjustment is performed. The imaging apparatus according to claim 3, wherein the thinning unit of the second imaging pixel is one or more pixel rows. The first motion vector detection unit reduces at least one vertical size of a template size and a search range of template matching processing based on a thinning-out unit of the second imaging pixel. 5. The imaging device according to 4. The said 2nd motion vector detection means does not make the direction opposite to the direction of a motion vector calculated | required by the said 1st motion vector detection means the search range of a template matching process. The imaging device described. The imaging apparatus according to claim 5, wherein the second motion vector detection unit does not perform horizontal motion vector detection. The imaging apparatus according to claim 1, wherein the second imaging pixel is a pixel used for at least one of display processing and focus detection processing. An imaging device having a first imaging pixel and a second imaging pixel that can be read by thinning out one or more pixels as a thinning unit, and that converts a subject image formed by an optical system into an image signal;
After the image sensor is exposed, an image sensor control unit that first reads out the signal of the second image pickup pixel and reads out the signal of the remaining first image pickup pixel that has not been read out;
Pixel signal rearranging means for rearranging the signal of the first imaging pixel and the signal of the second imaging pixel into an image signal having a pixel arrangement configuration of the imaging element;
A motion vector in which the first motion vector detection process for the signal of the second imaging pixel and the second motion vector detection process for the image signal rearranged by the image signal rearranging unit are performed in a time division manner. Detection means,
Have
The first motion vector detection process is controlled based on a thinning unit of the second imaging pixel,
The imaging apparatus according to claim 2, wherein the second motion vector detection process is controlled based on a motion vector detection result of the first motion vector detection process.