JP2019149600A - Imaging apparatus - Google Patents

Abstract

To reduce an amount of calculation of an area that does not require motion vector detection in accordance with an attitude of an imaging apparatus and a subject distance, so that deterioration in accuracy of motion vector detection is suppressed, and good motion vector detection is realized even when a detection cycle is short.SOLUTION: The imaging apparatus includes: attitude acquisition means for acquiring an attitude of an imaging apparatus; distance acquisition means for acquiring a distance from the imaging apparatus to a subject; image dividing means for dividing an image; feature point extraction means for calculating feature point coordinates and feature amounts in the image; feature point information holding means for holding the feature point coordinates and the feature amount as feature point information; and motion vector detecting means for detecting a motion vector using the feature point information held by the feature point information holding means and the image divided by the image dividing means. The motion vector detection means selects an image divided by the image dividing means in accordance with a result of the temporal change of the attitude acquisition means and the distance to the subject of the distance acquisition means.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging apparatus.

  A technique is known in which motion vector detection between frame images is obtained by template matching in order to perform camera shake correction and dynamic range expansion processing on a video shot using an imaging device.

  In template matching, one frame image is used as a template, and a position correlated with the template is searched from the other frame image. For example, Patent Document 1 describes a technique for dividing an image into a plurality of blocks, performing motion vector detection by template matching for each divided block, and statistically processing the plurality of motion vectors.

Japanese Patent No. 2956056

  Since the technique of Patent Document 1 performs motion vector detection for each divided block, the amount of calculation is very large. For this reason, when the motion vector detection cycle is short, such as when shooting a movie, the motion vector detection is completed within the cycle, so that the total calculation amount is reduced, such as reduction of the input image size and reduction of the search range. Thus, the accuracy of motion vector detection was reduced.

  An imaging apparatus according to the present invention includes an attitude acquisition unit that acquires the attitude of the imaging apparatus, a distance acquisition unit that acquires a distance from the imaging apparatus to a subject, an image division unit that divides an image, feature point coordinates in the image, and A feature point extracting means for calculating a feature quantity; a feature point information holding means for holding the feature point coordinates and the feature quantity as feature point information; and a motion vector detecting means for detecting a motion vector using an image. The motion vector detecting means selects the image divided by the image dividing means according to a result of temporal change of the posture acquiring means and a distance to the subject of the distance acquiring means. .

  According to the present invention, the amount of calculation of a region that does not require motion vector detection is reduced according to the orientation of the imaging device and the subject distance. By doing so, even when the detection cycle is short, it is possible to suppress the deterioration in accuracy of motion vector detection and realize good motion vector detection.

The figure for demonstrating the imaging device of this invention The figure for demonstrating the pixel arrangement example of the image pick-up element of this invention The figure for demonstrating the relationship between the imaging lens of this invention, and a unit pixel part. The figure for demonstrating the image signal waveform of this invention The figure for demonstrating the structure of this invention The figure for demonstrating the distance map of this invention The figure for demonstrating the pattern matching of this invention The figure for demonstrating the correlation table of this invention Flowchart for explaining the processing of the present invention The figure for demonstrating the process of this invention The figure for demonstrating the process of this invention The figure for demonstrating the process of this invention Flowchart for explaining the processing of the present invention

[Example 1]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 according to the present embodiment.

  The control unit 101 is, for example, a CPU, and controls the operation of each block included in the digital camera 100 by reading a program from the ROM 102, developing the program in the RAM 103, and executing the program. The ROM 102 is a rewritable nonvolatile memory, and stores parameters necessary for control of each block in addition to a program executed by the control unit 101. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output from each block included in the digital camera 100.

  The optical system 104 forms a subject image on the imaging unit 105. The imaging unit 105 is an imaging element such as a CCD or a CMOS sensor, for example, photoelectrically converts the subject image formed by the optical system 104, and outputs the obtained analog image signal to the A / D conversion unit 106. The A / D conversion unit 106 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data to the RAM 103.

  The posture detection unit 112 includes an acceleration sensor, an angular velocity sensor, a vibration gyro sensor, and the like, and detects a shift amount and a rotation angle as a posture change of the digital camera 100.

  Specifically, the shift amount (cX, cY, cZ) of the digital camera 100 is detected with respect to the three axes of the X axis as the horizontal direction (left and right direction), the Y axis as the vertical direction (vertical direction), and the Z axis as the optical axis direction. To do. Further, the rotation angles (cPi, cYa, cRo) are detected for the three axes of the pitch having the X axis as the rotation axis, the yaw having the Y axis as the rotation direction, and the roll having the Z axis as the rotation direction. The attitude detection unit 112 outputs the detected shift amount (cX, cY, cZ) and rotation angle (cPi, cYa, cRo) of the digital camera 100 to the RAM 103. Note that the posture detection unit 112 is applied with a widely known detection mechanism, and a detailed configuration thereof is omitted.

  The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, reduction / enlargement, filtering, encoding, and decoding to the image data stored in the RAM 103. Further, the image processing unit 107 performs the above-described processing on the image data stored in the RAM 103 according to the acquired shift amount (cX, cY, cZ) and rotation angle (cPi, cYa, cRo) of the digital camera 100. Applied image processing can be applied. A detailed description of the image processing unit 107 in this embodiment will be described later.

  The recording medium 108 is, for example, a detachable memory card, and the image data processed by the image processing unit 107 stored in the RAM 103, the image data output from the A / D conversion unit 106, and the like are in a predetermined format. Used to record as a data file. The communication unit 109 transmits an image data file or the like recorded on the recording medium 108 to an external device by wire or wireless.

  The display unit 110 displays image data obtained by shooting, image data read from the recording medium 108, and various menu screens. In some cases, a live view image is displayed to function as an electronic viewfinder.

  The operation unit 111 is an input device group for a user to input various instructions and settings to the digital camera 100. The operation unit 111 includes keys and buttons of a general digital camera such as a shutter button, a menu button, a direction key, and a determination key. It has. Further, when the display unit 110 is a touch display, it also serves as the operation unit 111. The operation unit 111 may have a configuration that does not require a physical operation, such as a combination of a microphone and a voice command recognition unit.

  Next, a method for obtaining the distance to the subject will be described with reference to FIGS. FIG. 2 is a schematic diagram illustrating a pixel arrangement example of the imaging unit 105. In the unit pixel unit 500, R (Red) / G (Green) / B (Blue) color filters are arranged in a Bayer array. In each unit pixel unit 500, a sub pixel a and a sub pixel b are arranged. In the figure, reference numeral 501 denotes a first photoelectric conversion unit constituting the subpixel a, and 502 denotes a second photoelectric conversion unit constituting the subpixel b.

  Here, a (x, y) is image data corresponding to the sub-pixel a501 obtained from the unit pixel unit 500, and b (x, y) is image data corresponding to the sub-pixel b502 obtained from the unit pixel unit 500. (X and y are integers of 0 or more). The imaging unit 105 of the present embodiment outputs b (x, y) and basic image data g (x, y) in the following expression as a pair of captured image data.

g (x, y) = a (x, y) + b (x, y) (Formula 1)
That is, the imaging unit 105 outputs the image data obtained by the subpixel b502 and the basic image data obtained by adding the image data obtained by the subpixel b502. Note that one set of image data output by the imaging unit 105 may be another combination. For example, a combination of a (x, y) and g (x, y) or a combination of a (x, y) and b (x, y) may be used.

  FIG. 3 is a schematic diagram for explaining the relationship between the light beam emitted from the optical system 104 and the unit pixel unit 500. The unit pixel unit 500 includes 501 (subpixel a) and 502 (subpixel b). The color filter 201 and the microlens 202 are formed on the unit pixel unit 500, respectively.

  An optical axis 204 indicates the center of the light beam emitted from the photographing lens of the optical system 104 with respect to the pixel portion having the microlens 202. The light that has passed through the photographing lens of the optical system 104 enters the unit pixel unit 500 around the optical axis 204. Regions 205 and 206 represent partial regions in the photographic lens of the optical system 104, respectively.

  As shown in FIG. 3, the light beam passing through the region 205 is received by the 501 (sub-pixel a) through the microlens 202. The light beam passing through the region 206 is received by the 502 (sub-pixel b) through the microlens 202. Therefore, each of the sub-pixel a and the sub-pixel b receives light that passes through different areas of the photographic lens of the optical system 104. Therefore, the phase difference type focus detection can be performed by comparing the output signals of the sub-pixel a and the sub-pixel b.

  FIG. 4 is a schematic diagram showing an image signal waveform obtained from the subpixel a and an image signal waveform obtained from the subpixel b. The horizontal axis indicates the pixel position in the horizontal direction, and the vertical axis indicates the signal output level.

  FIG. 4A illustrates the image signal waveforms when the image is out of focus. The image signal waveforms obtained from each of the sub-pixels a and b do not match and are shifted from each other. When approaching the in-focus state, as shown in FIG. 4B, the shift between the image signal waveforms becomes small, and both waveforms overlap in the in-focus state. As described above, the defocus amount (defocus amount) is detected based on the correlation between the image signal waveforms obtained from the sub-pixels a and b, and the distance to the subject can be obtained based on the detection result.

  Next, functions of the image processing unit 107 according to the present embodiment will be described with reference to FIGS. FIG. 5 is a schematic diagram illustrating functions of the image processing unit 107. In FIG. 5, for convenience, it is described that there are functional blocks corresponding to various types of image processing to which the image processing unit 107 can be applied. However, FIG. 5 schematically shows image processing steps for performing motion vector detection as functional blocks, and does not necessarily match the actual configuration. For example, when the image processing unit 107 is a programmable processor such as a GPU or a DSP, each functional block may be implemented by software, and input / output lines between the functional blocks may not exist.

  The image dividing unit 310 converts the basic image data g (x, y) and the sub-pixel b (x, y) input from the imaging unit 105 or the RAM 103 into the horizontal division number and the vertical number set in advance by the control unit 101. Divide into basic blocks according to the number of divisions. The image dividing unit 310 outputs basic image data g (x, y) of the basic block to the image holding unit 302 in response to a request from the control unit 101. Further, the image dividing unit 310 outputs the basic image data g (x, y) and the sub-pixel b (x, y) of the basic block to the distance map 301 generation unit in response to a request from the control unit 101.

  The distance map generation unit 301 calculates a sub pixel a (x, y) from the basic image data g (x, y) of the basic block input from the image holding unit 302 and the sub pixel b (x, y). The distance map generation unit 301 compares the corresponding subpixel a and subpixel b while shifting the subpixel a (x, y) and subpixel b (x, y) in the horizontal direction. Here, the shift amount when shifting the subpixel a to the left is minus, and the shift amount when the subpixel b is shifted to the right is plus, and the shift movement of -S to S is performed.

  The sum of the data with the smaller value at each position in the horizontal direction during each shift movement is calculated as the correlation value for one line. The correlation value calculation method is not limited to this. Any calculation method can be applied as long as it is a calculation method that indicates the correlation between the sub-pixel a (x, y) and the sub-pixel b (x, y). The correlation value of the distance map calculated by the distance map generator 301 performs signal processing such as known noise reduction processing, matrix processing, and degeneration processing, and forms a distance map def (x, y) including distance information to the subject. And output to the image holding unit 302.

  FIG. 6 shows a conceptual diagram when the distance map generator 301 generates a distance map def (x, y) of 7 horizontal pixels and 5 vertical pixels.

  The image holding unit 302 divides the arbitrary block basic image data g (x, y) input from the image dividing unit 310 and the arbitrary block distance map def (x, y) input from the distance map generating unit 301. Associating and holding each previous frame. The image holding unit 302 has a storage medium composed of a RAM (not shown), and can hold at least two frames of basic image data g (x, y) and a distance map def (x, y). In response to a request from the control unit 101, the image holding unit 302 receives an arbitrary frame, arbitrary basic image data g (x, y) and a distance map def (x, y) as a feature point extraction unit 303, a template matching unit 304, It outputs to the distance information acquisition part 311.

  The feature point extraction unit 303 performs feature point extraction on the basic image input from the image holding unit 302, and outputs the extracted feature point coordinates to the template matching unit 304 and the distance acquisition unit 311.

  For the feature point extraction, for example, there is a known Harris corner detector or Shi and Tomasi technique. The luminance value at the pixel (x, y) of the image is expressed as I (x, y). In both methods, first, an autocorrelation matrix H expressed by Equation 1 is created from the results Ix and Iy obtained by applying horizontal and vertical first-order differential filters to an image, respectively.

In Equation 2, G represents smoothing by the Gaussian distribution shown in Equation 2.

The Harris detector extracts a point having a higher feature evaluation value (hereinafter referred to as a feature amount) as a feature point from the feature evaluation equation shown in Equation 4.

In Equation 4, det represents a determinant, and tr represents the sum of diagonal components. Moreover, (alpha) is a constant and the value of 0.04-0.15 is good experimentally. On the other hand, in Shi and Tomasi, a feature evaluation formula shown in Formula 5 is used.

Expression 5 represents that the smaller eigenvalue of the eigenvalues of the autocorrelation matrix H of Expression 2 is used as the feature quantity.

  The distance acquisition unit 311 reads a corresponding distance map from the image holding unit 302 based on the feature point coordinates input from the feature point extraction unit 303, and acquires the distance of the feature point coordinates. The distance acquisition unit 311 outputs the distance of the acquired feature point coordinates to the priority order control unit 312.

  The priority determination unit 312 receives the distance of the feature point coordinates input from the distance acquisition unit 311, the shift amount (cX, cY, cZ) and the rotation angle (cPi, cYa, cRo) of the digital camera 100 held in the RAM 103. ) To determine the priority of correlation table processing generated by the template matching unit 304. A specific determination method by the priority order determination unit 312 will be described later.

  The template matching unit 304 generates a correlation table between corresponding frames based on the feature point coordinates input from the feature point extraction unit 303 and the determination of the priority order determination unit 312. The template matching unit 304 outputs the generated correlation table to the correlation table holding unit 305.

  FIG. 7 shows an outline of template matching. 7A shows an arbitrary block of the current frame (hereinafter referred to as an original image), and FIG. 4B shows an arbitrary block of the past frame (hereinafter referred to as a reference image). These images are stored in the image holding unit 302. Is the basic image data g (x, y) stored in.

  Then, as indicated by 401 in FIG. 7, a template is arranged at the coordinates of the feature points in the original image, and a correlation value between the template 401 and each area of the reference image is calculated. At this time, since calculating the correlation value for the entire area of the reference image requires a large amount of computation, the rectangular area for calculating the correlation value on the reference image is actually set as the search range as indicated by 402. Set as. The position and size of the search range 402 are not particularly limited, but a correct motion vector cannot be detected unless the search range 402 includes an area corresponding to the destination of the template 401.

  In this embodiment, a sum of absolute difference (hereinafter abbreviated as SAD) is used as an example of a correlation value calculation method. Formula 6 shows the SAD calculation formula.

  In Equation 6, f (i, j) represents the luminance value at the coordinates (i, j) in the template block 401, and g (i, j) is the template block for which the correlation value is calculated in the search range 402. Each luminance value in 403 is represented. In SAD, the absolute value of the difference is calculated for each luminance value f (i, j) and g (i, j) in both blocks, and the correlation value S_SAD can be obtained by calculating the sum. Therefore, the smaller the correlation value S_SAD, the smaller the difference between the blocks, that is, the textures in the blocks of the template 401 and the template block 403 are similar.

  In this embodiment, SAD is used as an example of a correlation value, but the present invention is not limited to this, and other correlation values such as sum of squares of differences (SSD) and normalized cross correlation (NCC) may be used. .

  Then, when the correlation value is calculated by moving the template block 403 for the entire area of the search range 402, a correlation table as shown in FIG. 8 can be created.

  The example shown in FIG. 8 shows a correlation table obtained by SAD by setting rectangular regions of ± 2 in the horizontal direction and ± 2 in the vertical direction as search ranges for the feature points.

  In the figure, the minimum correlation value is the minimum value 8, and it can be determined that a texture very similar to the template block 401 exists in the region where the minimum value 8 is calculated on the reference image. In this way, by calculating the correlation value between the template block 401 and the search range 402 and determining the position where the value is the smallest, to which position the template block on the original image has moved in the reference image That is, it becomes possible to detect a motion vector between images.

  The correlation table holding unit 305 holds the correlation table input from the template matching unit 304 in association with each block. The correlation table holding unit 305 has a storage medium composed of a RAM (not shown), and holds a correlation table calculated from each basic block of the basic image data g (x, y) and the distance map def (x, y). can do. The correlation table holding unit 305 generates a correlation table calculated from basic image data g (x, y) and a distance map def (x, y) of an arbitrary basic block in response to a request from the control unit 101. To 307.

  The motion vector detection unit 307 detects a motion vector based on the correlation table held in the correlation table holding unit 305. The motion vector detection unit 307 outputs the detected motion vector.

  In the motion vector detection method, the minimum shift amount in the correlation table is the motion vector. For example, in the correlation table shown in FIG. 8, since the minimum value is “8”, if the center of the reference block is the origin before movement (position where the SAD value is “26”), the motion vector is (+1, +1). )

  Next, processing of the image processing unit 107 will be described with reference to FIGS. FIG. 9 is a flowchart of the motion vector detection process. FIG. 10 is a conceptual diagram for explaining preconditions in the present embodiment. FIG. 10A shows a basic image of the previous frame. FIG. 10B shows a distance map of the previous frame. The distance map in the present embodiment shows the distance from the digital camera 100 to the subject in three states of short distance, medium distance, and long distance, the short distance is black, the medium distance is diagonally hatched, and the long distance is white. Represent.

  FIG. 10C is a division example of the image division unit 310 in the present embodiment. The image dividing unit 310 divides the basic image and the distance map into three horizontal divisions and three vertical divisions, and divides them into basic blocks A to I. Note that the image holding unit 302 and the correlation table holding unit 305 in the image processing unit 107 hold results obtained by processing the basic image in FIG. 10A in advance. FIG. 10D shows the posture of the digital camera 100 in the present embodiment. 10A, the horizontal direction (left-right direction) of the digital camera 100 is taken as the X axis, the vertical direction (up-down direction) is taken as the Y axis, and the optical axis direction (depth direction) is taken as the Z axis.

  Further, a pitch having the X axis as the rotation axis, a yaw having the Y axis as the rotation direction, and a roll having the Z axis as the rotation direction are used. FIG. 11 is an example of a shift of the digital camera 100, and is a conceptual diagram for explaining motion vector detection processing when a shift amount in the Z-axis direction is detected. FIG. 12 is an example in which the digital camera 100 is rotated, and is a conceptual diagram for explaining motion vector detection processing when a rotation angle with respect to the yaw axis is detected.

  Next, a motion vector detection process when the digital camera 100 shifts (a shift amount in the Z-axis direction is detected) will be described with reference to FIGS. When the motion vector detection process is started, the process proceeds to S401, and the basic image is divided into basic blocks of a predetermined size. The control unit 101 inputs the basic image in FIG. 11A to the image dividing unit 310. The image dividing unit 310 divides FIG. 11A into basic blocks A to I in accordance with a request from the control unit 101, and outputs the divided basic blocks to the distance map generating unit 301 and the image holding unit 302.

  In step S402, feature points are extracted for each of the basic blocks A to I. The control unit 101 inputs the basic blocks A to I held in the image holding unit 302 to the feature point extraction unit 303. The feature point extraction unit 303 extracts feature points for each basic block. The feature point extraction unit 303 outputs the extracted feature point coordinates to the template matching unit 304 and the distance acquisition unit 311 according to the request of the control unit 101. FIG. 11B is a conceptual diagram when the feature point coordinates are extracted for the divided basic blocks A to I. The black circles in FIG. 11B indicate the feature point coordinates in each basic block.

  In step S403, a distance map is generated for each of the basic blocks A to I. The control unit 101 inputs the basic blocks A to I divided by the image dividing unit 310 to the distance map generating unit 301. The distance map generation unit 301 generates a distance map for each basic block. The distance map generation unit 301 outputs the generated distance map to the image holding unit 302. FIG. 11C is a conceptual diagram when a distance map is generated for the divided basic blocks A to I.

  In step S404, the distance between feature point coordinates is acquired for each of the basic blocks A to I. The control unit 101 inputs the feature point coordinates of the basic blocks A to I extracted by the feature point extraction unit 303 and the distance maps A to I held in the image holding unit 302 to the distance acquisition unit 311. The distance acquisition unit 311 acquires a distance corresponding to the feature point coordinates for each of the basic blocks A to I from the distance map. The distance acquisition unit 311 outputs the distance corresponding to the acquired feature point coordinates to the priority order determination unit 312. FIG. 11D is a conceptual diagram in which distances corresponding to the feature point coordinates of the divided basic blocks A to I are acquired.

  In step S405, the setting value of the priority order determination unit 312 is set. The control unit 101 outputs the shift amount (cX, cY, cZ) and the rotation angle (cPi, cYa, cRo) of the digital camera 100 held in the RAM 103 to the priority order determination unit 312. Further, the control unit 101 sets the shift amount threshold values (thX, thY, thZ) and the rotation angle threshold values (thPi, thYa, thRo) held in the ROM 102 in the priority order determination unit 312.

  In step S406, the priority determination unit 312 compares the absolute values of whether the set shift amount exceeds the threshold (| thX | <| cX |, | thY | <| cY |, | thZ |). <| CZ |). When the shift amount exceeds the threshold value (S406: YES), the priority determination unit 312 determines that the shift amount result is a shift movement (S407), and proceeds to S409. On the other hand, when the shift amount does not exceed the threshold value (S406: NO), the priority determination unit 312 determines that the shift amount result indicates no shift movement (S408), and proceeds to S409.

  In step S409, the priority determination unit 312 compares the absolute values of whether or not the set rotation angle exceeds the threshold (| thPi | <| cPi |, | thYa | <| cYa |, | thRo |). <| CRo |). When the rotation angle exceeds the threshold (S409: YES), the priority determination unit 312 determines that the rotation angle result is rotated (S410), and proceeds to S412. On the other hand, when the rotation angle does not exceed the threshold (S409: NO), the priority determination unit 312 determines that the rotation angle result indicates no rotation (S411), and proceeds to S412.

  In step S412, the priority determination unit 312 selects a priority based on the shift amount result determined in step S406 and the rotation angle result determined in step S407. When it is determined that “there is a shift amount movement” and “no rotation” (S412: YES), the priority determination unit 312 outputs the distance between the feature point coordinates for each of the basic blocks A to I output from the distance acquisition unit 311. Extract a distant basic block (S413). The priority order determination unit 312 selects a basic block to be executed from the extracted basic blocks in accordance with a request from the control unit 101, and proceeds to S420.

  FIG. 11E is an example in which a basic block with a long distance is extracted based on the distance of the feature point coordinates for each basic block of A to I output from the distance acquisition unit 311 in FIG. 11D. . A, B, C, D, and F for which the determination in FIG.

  Next, when it is determined that “no shift amount movement” and “rotation” (S414: YES), the priority determination unit 312 outputs the feature point coordinates for each of the basic blocks A to I output from the distance acquisition unit 311. A basic block whose distance is near is extracted (S415). The priority order determination unit 312 selects a basic block to be executed from the extracted basic blocks in accordance with a request from the control unit 101, and proceeds to S420.

  In other cases (S416), the priority determination unit 312 extracts the distance between the feature point coordinates for each of the basic blocks A to I output from the distance acquisition unit 311 under a preset condition. The conditions set in advance here are set by the control unit 101 according to the shooting situation, and are not particularly limited, such as a basic block including the main subject, all basic blocks A to I, and a basic block at the center of the screen. The priority order determination unit 312 selects a basic block to be executed from the extracted basic blocks in accordance with a request from the control unit 101, and proceeds to S420.

  In step S420, a correlation table is generated by template matching between the basic block and the reference basic block. In accordance with the basic block to be executed selected by the priority determination unit 312, the control unit 101 uses the basic block held in the image holding unit 302, the reference basic block, and the feature point coordinates held in the feature point extraction unit 303 as templates. Input to the matching unit 304. The template matching unit 304 outputs the generated correlation table to the correlation table holding unit 305 based on the input basic block, reference basic block, and feature point coordinates.

  In step S421, motion vector detection is performed. The motion vector detection unit 307 calculates a motion vector from the correlation table result of the corresponding basic block, outputs the calculated motion vector to the RAM 103, and proceeds to S422.

  In step S422, the control unit 101 determines whether the motion vector detection processing for the extracted basic block has been completed. When the motion vector detection process of the extracted basic block is completed (S422: Yes), the control unit 101 completes the motion vector detection process. On the other hand, when the motion vector detection is not completed in the extracted basic block (S422: No), the control unit 101 sets the next basic block and returns to S420.

  Next, a motion vector detection process when the digital camera 100 is rotated (a rotation angle with respect to the yaw axis is detected) will be described with reference to FIGS. Note that the description of the overlapping motion vector detection process is omitted.

  First, in step S <b> 401, the control unit 101 inputs the basic image in FIG. 12A to the image dividing unit 310. The image dividing unit 310 divides FIG. 12A into basic blocks A to I according to the request of the control unit 101, and outputs the divided basic blocks to the distance map generating unit 301 and the image holding unit 302.

  Next, in S402, the feature point extraction unit 303 extracts feature point coordinates for the divided basic blocks A to I as shown in FIG. Black circles in FIG. 12B indicate feature point coordinates in each basic block.

  In step S403, the distance map generation unit 301 generates a distance map for the divided basic blocks A to I as shown in FIG.

  Next, in S404, the distance acquisition unit 311 acquires a distance corresponding to the feature point coordinates of the divided basic blocks A to I as shown in FIG.

  Next, in S415, as shown in FIG. 12E, based on the distance of the feature point coordinates for each of the basic blocks A to I output from the distance acquisition unit 311 in FIG. It is the example which extracted the block. E, F, H, and I for which the determination in FIG.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  In the present embodiment, the priority order determination unit 312 extracts conditions that satisfy the conditions from all the basic blocks. However, the priority order determination unit 312 may perform determination based on the basic blocks extracted in the previous frame. By extracting from the basic blocks extracted in the previous frame, it is possible to further limit the number of basic blocks from which motion vectors are acquired, and further efficiency can be expected.

[Example 2]
An embodiment of the present invention that improves the efficiency of motion vector detection will be described in detail with reference to the drawings. FIG. 13 is a flowchart of the motion vector detection process for explaining the present embodiment. A description of the same processing as that in FIG. 9 is omitted.

  First, a distance map is used to determine whether a motion vector can be detected. The control unit 101 acquires the depth of field according to the shooting conditions held in the RAM 102, and determines whether or not the distance range of the distance map generated in S403 is included in the depth of field. When the distance range in the distance map is included in the depth of field (S440: Yes), the control unit 101 determines that the distance map is appropriate, and proceeds to S404. On the other hand, when the distance range in the distance map is not included in the depth of field (S440: No), the control unit 101 is improper in the distance map (the state is not suitable for motion vector detection because it is out of focus). The motion vector detection process is completed.

  Next, it is determined from the shift amount whether the motion vector can be detected. The control unit 101 acquires the maximum shift amount held in the RAM 102 and determines whether or not the shift amount of the digital camera 100 held in the RAM 103 exceeds the maximum shift amount. If the shift amount of the digital camera 100 does not exceed the maximum shift amount (S441: Yes), the control unit 101 determines that the shift amount is appropriate, and proceeds to S407. On the other hand, when the shift amount of the digital camera 100 exceeds the maximum shift amount (S441: No), the control unit 101 has an inappropriate shift amount (a state in which the digital camera 100 has a large amount of change in posture and is not suitable for motion vector detection). The motion vector detection process is completed.

  Next, it is determined from the rotation angle whether the motion vector can be detected. The control unit 101 obtains the maximum rotation angle held in the RAM 102 and determines whether or not the rotation angle of the digital camera 100 held in the RAM 103 exceeds the maximum rotation angle. When the rotation angle of the digital camera 100 does not exceed the maximum rotation angle (S442: Yes), the control unit 101 determines that the rotation angle is appropriate, and proceeds to S410. On the other hand, when the rotation angle of the digital camera 100 exceeds the maximum rotation angle (S442: No), the control unit 101 has an inappropriate rotation angle (a state change amount of the digital camera 100 is large and is not suitable for motion vector detection). The motion vector detection process is completed.

  By performing the above processing, it becomes possible to determine in advance a state that is not suitable for motion vector detection, and the motion vector detection can be made more efficient.

100 Digital camera 101 Control unit 102 ROM
103 RAM
DESCRIPTION OF SYMBOLS 104 Optical system 105 Image pick-up part 106 A / D conversion part 107 Image processing part 108 Recording medium 109 Communication part 110 Display part 111 Operation part 112 Angular velocity sensor 201 Color filter 202 Micro lens 204 Optical axis 310 Image division part 301 Distance map generation part 302 Image storage unit 303 Feature point extraction unit 311 Distance acquisition unit 312 Priority control unit 304 Template matching unit 305 Correlation table storage unit 307 Motion vector detection unit 401 Template 402 Search range 403 Template block 500 Unit pixel unit 501 First photoelectric conversion unit 502 Second photoelectric conversion unit

Claims (9)

  Attitude acquisition means for acquiring the attitude of the imaging apparatus, distance acquisition means for acquiring the distance from the imaging apparatus to the subject, image division means for dividing the image, and feature points for calculating feature point coordinates and feature amounts in the image Extraction means; feature point information holding means for holding the feature point coordinates and feature quantities as feature point information; feature point information held by the feature point information holding means; and an image divided by the image dividing means Motion vector detecting means for detecting a motion vector using the image, and the motion vector detecting means is configured to detect the image according to a result of temporal change of the posture acquiring means and a distance to the subject of the distance acquiring means. An image pickup apparatus that selects an image divided by a dividing unit.   The imaging apparatus according to claim 1, wherein the posture acquisition unit detects shift movement and rotation of the imaging apparatus.   The imaging apparatus according to claim 1, wherein the distance acquisition unit acquires a distance from the imaging apparatus to a subject as an image.   4. The apparatus according to claim 1, further comprising distance information acquisition means for calculating a distance from the imaging device acquired by the distance acquisition means to a subject for the feature point coordinates held by the feature point information holding means. The imaging device according to any one of the above.   The motion vector detection unit gives priority to an image including coordinates of a distant distance from the imaging device to the subject acquired by the distance information acquisition unit when the posture acquisition unit determines that the movement is shifted. Item 5. The imaging device according to any one of Items 1 to 4.   The motion vector detecting unit gives priority to an image including coordinates in the vicinity of the distance from the imaging device acquired by the distance information acquiring unit to the subject when it is determined that the posture is acquired by the posture acquiring unit. The imaging device according to any one of claims 1 to 4.   A determination unit configured to determine whether or not the posture acquired by the posture acquisition unit is appropriate as a state for detecting a motion vector, and when the determination unit determines that the state is not appropriate as a state for detecting a motion vector, The imaging device according to claim 1, wherein the imaging device is not detected. -The motion vector detecting means holds information for selecting the image divided by the image dividing means, and selects the image divided by the image dividing means based on the information for selecting the held image. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus.   The motion vector detection means holds a selected area with respect to the immediately preceding image, and selects the held immediately preceding image from the selected area when selecting the current image. Item 9. The imaging device according to Item 8.