JP2020057182A - Image processing device, image processing method, and program - Google Patents

Abstract

To enable obtainment of a natural video even when frames of multiple videos are connected to each other at ends thereof.SOLUTION: An image processing device 100 includes an image acquisition unit 11 that acquires, in units of frame, a first image and a second image that has an end shared with the first image, a conversion matrix calculation unit 15 that calculates a conversion matrix on the basis of the correspondence between features in the respective images, an error calculation unit 17 that calculates an error when conversion using the conversion matrix is performed, a conversion matrix selection unit 18 that selects a conversion matrix in which the error calculated in a prescribed frame by the error calculating means satisfies a prescribed condition, and an image generation unit 19 that generates an image obtained by connecting the first image and the second image to each other using the selected conversion matrix. The conversion matrix selection unit 18 selects the conversion matrix in which the prescribed condition is satisfied, from among a conversion matrix selected in a frame preceding the prescribed frame and a plurality of conversion matrixes calculated in the prescribed frame.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an image processing method, and a program.

  A technique for connecting a plurality of images to generate a large image is known. For example, Patent Literature 1 discloses an image processing technology that enables high-accuracy splicing of images having relatively large distortion amounts, such as those shot with a fisheye lens, by properly using distortion correction parameters. ing.

JP 2016-27744 A

  In the image processing technology disclosed in Patent Literature 1, it is assumed that images to be connected are still images, and it is not assumed to connect moving images. However, since moving images are still images when viewed in frame units, it is possible to connect moving images in frame units by using the image processing technology disclosed in Patent Document 1 by treating moving images in frame units. It is. However, if such processing is performed, the processing is completed for each frame, and the relationship between frames cannot be considered, so that an unnatural moving image may be obtained. For example, even if natural splicing is performed on a frame-by-frame basis, the image may be slightly shifted between frames, and in this case, the moving image becomes an unnatural moving image in which blur occurs.

  The present invention has been made in view of the above-described circumstances, and, for a plurality of moving images, an image that can obtain a natural moving image even when frames of each moving image are connected at their ends. It is an object to provide a processing device, an image processing method, and a program.

In order to achieve the above object, an image processing apparatus according to the present invention includes:
Image acquisition means for acquiring each of the first image and the second image having an end common to the first image in frame units;
Common area specifying means for specifying a common area that is an area common between the first image and the second image obtained by the image obtaining means;
A feature point specifying unit that specifies a pair of corresponding feature points in the first image and the second image in the common region specified by the common region specifying unit;
Based on a pair of feature points specified by the feature point specifying means, a first feature point in the first image is set as a starting point, and a second feature point in the second image corresponding to the first feature point is set. A movement vector calculation unit that calculates a plurality of movement vectors, ending with the feature point of
Conversion matrix calculation means for calculating a conversion matrix based on a selected movement vector selected from the plurality of movement vectors calculated by the movement vector calculation means,
A conversion unit that calculates a position of a post-movement feature point that is a position obtained by converting the position of the first feature point in the first image, using the conversion matrix calculated by the conversion matrix calculation unit;
Error calculation means for calculating an error between the position of the second feature point and the position of the moved feature point calculated by the conversion means;
In a predetermined frame, from among the plurality of conversion matrices calculated by the conversion matrix calculation unit, a conversion matrix selection unit that selects a conversion matrix whose error calculated by the error calculation unit satisfies a predetermined condition,
Image generation means for generating an image obtained by joining the first image and the second image using the conversion matrix selected by the conversion matrix selection means;
With
The conversion matrix selection unit is a conversion matrix that satisfies a predetermined condition from a conversion matrix selected in a frame before the predetermined frame and a plurality of conversion matrices calculated by the conversion matrix calculation unit in the predetermined frame. Select a matrix.

  According to the present invention, a natural moving image can be obtained for a plurality of moving images even when frames of the moving images are connected at their ends.

FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus according to the first embodiment. 3 is a flowchart of image processing according to the first embodiment. It is a figure explaining the common area | region in the hemispherical image of a head-on direction and the hemispherical image of a head-down direction. It is the figure which expanded the common area | region in the semi-celestial sphere image of the top direction in polar coordinates. It is the figure which expanded the common area | region in the semi-celestial sphere image of the downward direction in polar coordinates. It is a figure explaining the common area | region in the semi-celestial sphere image of the state which shifted | deviated 5 degrees from the semi-celestial sphere image of a head-on direction and the head-down direction. It is the figure which expanded the common area | region in the semi-celestial sphere image of the top direction in polar coordinates. It is the figure which expanded the common area | region in the semi-celestial sphere image in the state shifted | deviated from right below by 5 degrees in polar coordinates. It is a figure explaining a movement vector. It is the figure which superimposedly displayed the movement vector and the vector after conversion. FIG. 5 is a diagram illustrating an example in which a first image and a second image are joined. FIG. 7 is a diagram illustrating an example in which a first image and a second image are directly connected. FIG. 9 is a diagram illustrating an example in which a first image is converted and then connected to a second image. 5 is an example of an image generated by joining a first image and a second image in a certain frame. 13 is an example of an image generated by joining a first image and a second image in a next frame without considering a transformation matrix in a previous frame. FIG. 4 is a diagram illustrating an example in which an unnatural moving image is generated. 14 is a flowchart of image processing according to Modification Example 4. 15 is a flowchart of image processing according to Modification Example 5. 15 is a flowchart of image processing according to Modification 6.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts have the same reference characters allotted.

(Embodiment 1)
The image processing apparatus according to the first embodiment of the present invention connects natural image frames of a wide range image such as a celestial sphere image or a panoramic image by connecting frames of each moving image at an end of the plurality of moving images. This is a device that generates a moving image.

  As shown in FIG. 1, the image processing apparatus 100 according to the first embodiment of the present invention includes a control unit 10, a storage unit 20, a first imaging unit 31, and a second imaging unit 32.

  The control unit 10 is configured by a CPU (Central Processing Unit) or the like, and executes programs stored in the storage unit 20 to thereby control each unit (an image acquisition unit 11, a common area identification unit 12, a feature point identification unit 13) described later. , The functions of the motion vector calculation unit 14, the conversion matrix calculation unit 15, the conversion unit 16, the error calculation unit 17, the conversion matrix selection unit 18, and the image generation unit 19).

  The storage unit 20 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The ROM stores a program to be executed by the CPU of the control unit 10 and data necessary for executing the program in advance. The RAM stores data created or changed during the execution of the program.

  The first imaging unit 31 and the second imaging unit 32 each include a monocular imaging device (camera). The first imaging unit 31 captures a first image (a moving image captured in frame units) at, for example, 30 fps (frames per second). The second imaging unit 32 also captures a second image (moving image captured in frame units) at, for example, 30 fps. The first image captured by the first image capturing unit 31 and the second image captured by the second image capturing unit 32 are configured such that the first image capturing unit 31 and the second image The shooting direction of the imaging unit 32 is set.

  For example, both the first imaging unit 31 and the second imaging unit 32 are imaging devices equipped with a fisheye lens that captures a semi-celestial sphere image with an angle of view of 200 degrees, and the first imaging unit 31 and the second imaging unit 32 are opposite to each other. The camera is installed so as to shoot in a direction (for example, the first imaging unit 31 is in a direction directly above, and the second imaging unit 32 is in a direction directly below).

  Next, a functional configuration of the control unit 10 of the image processing apparatus 100 will be described. The control unit 10 includes an image acquisition unit 11, a common region identification unit 12, a feature point identification unit 13, a movement vector calculation unit 14, a conversion matrix calculation unit 15, a conversion unit 16, an error calculation unit 17, a conversion matrix selection unit 18, an image The function of the generation unit 19 is realized.

  The image acquisition unit 11 acquires an image captured by the first imaging unit 31 as a first image in frame units, and acquires an image captured by the second imaging unit 32 as a second image in frame units. Note that the image acquisition unit 11 may or may not acquire the image captured by the first imaging unit 31 or the image captured by the second imaging unit 32 in real time. When the image is not acquired in real time, for example, the image captured by the first imaging unit 31 and the image captured by the second imaging unit 32 are stored in the storage unit 20. The acquisition unit 11 may acquire an image captured by the first imaging unit 31 or an image captured by the second imaging unit 32 from the storage unit 20. The image obtaining unit 11 functions as an image obtaining unit.

  The common area specifying unit 12 is configured to capture an image area common to the first image acquired by the image acquisition unit 11 (taken by the first imaging unit 31) and the second image (taken by the second imaging unit 32). Common area). As described above, the first image and the second image have at least a part of their respective ends in common, and the common end is a common area. The common area specifying unit 12 specifies the common area. The common area specifying unit 12 functions as a common area specifying unit. For example, a geomagnetic sensor or an acceleration sensor is provided in the first imaging unit and the second imaging unit, and the common area specifying unit 12 transmits information on the orientation of the first imaging unit 31 and the second imaging unit 32 at the time of shooting. From the sensor. Then, the common area specifying unit 12 stores the information on the angle of view range of the first imaging unit 31 and the second imaging unit 32 stored in advance and the information on the orientation of the first imaging unit 31 and the second imaging unit 32. The common area may be specified based on the above.

  The feature point identifying unit 13 associates the feature points in the first image with the feature points in the second image in the common area identified by the common area identifying unit 12, and sets the corresponding feature points. Identify the pair. The feature points refer to characteristic portions such as edges, corners, and patterns in the image. The feature point specifying unit 13 searches the second image for a feature point having a feature amount similar to the feature point based on the feature amount of the feature point extracted from the first image, for example. The feature points in the image are associated with the feature points in the second image. The feature point specifying unit 13 functions as a feature point specifying unit.

  The movement vector calculation unit 14 uses the first feature point in the first image as a starting point based on the pair of feature points identified by the feature point identification unit 13, and generates a second image corresponding to the first feature point. A movement vector having the second feature point therein as an end point is calculated. Since the number of motion vectors is equal to the number of pairs of feature points specified by the feature point specifying unit 13, the motion vector calculation unit 14 calculates a plurality of first motion vectors (the number of pairs of specified feature points). I do. The movement vector calculation unit 14 functions as a movement vector calculation unit.

  The conversion matrix calculation unit 15 calculates a specified number (for calculating a conversion matrix in a two-dimensional space, a conversion matrix in a three-dimensional space) from a plurality of movement vectors calculated by the movement vector calculation unit 14. If so, the vector (selected movement vector) of 5) is selected, and a transformation matrix (projection transformation matrix) is calculated based on the selected movement vector. By changing the selection of the selected movement vector, the conversion matrix calculation unit 15 can calculate a plurality of conversion matrices. The transformation matrix calculator 15 functions as a transformation matrix calculator.

  The conversion unit 16 uses the conversion matrix calculated by the conversion matrix calculation unit 15 to calculate the position of the moved feature point, which is the position obtained by converting the position of the first feature point in the first image. The conversion unit 16 functions as a conversion unit.

  The error calculation unit 17 calculates an error between the position of the second feature point in the second image corresponding to the first feature point and the position of the moved feature point calculated by the conversion unit 16. Since a plurality of second feature points exist in a form corresponding to each of the first feature points, and a plurality of post-movement feature points calculated by the conversion unit 16 also exist in a form corresponding to each of the first feature points. , An error is obtained for each feature point. The error calculating unit 17 calculates various evaluation values described later as values for evaluating these errors. The error calculator 17 functions as an error calculator.

  The transformation matrix selection unit 18 selects a transformation matrix whose error calculated by the error calculation unit 17 satisfies a predetermined condition from among the plurality of transformation matrices calculated by the conversion matrix calculation unit 15. The transformation matrix selection unit 18 functions as a transformation matrix selection unit.

  The image generation unit 19 generates an image by joining the first image and the second image using the conversion matrix selected by the conversion matrix selection unit 18. The image generator 19 functions as an image generator.

  The functional configuration of the image processing apparatus 100 has been described above. Next, image processing by the image processing apparatus 100 will be described with reference to FIG. This image processing is started by a user's instruction.

  First, the image acquisition unit 11 acquires the first frame of the moving image captured by the first imaging unit 31 (the next frame when returning from step S113 to be described later) as the first image, and The first frame of the moving image captured by the unit 32 (the next frame when returning from step S113 described later) is acquired as a second image (step S101).

  For example, as shown in FIG. 3, the first imaging unit 31 is an imaging device equipped with a fish-eye lens that captures a directly-upward semispherical image 51 at an angle of view of 200 degrees, and the second imaging unit 32 is a directly-downward semispherical image. It is assumed that the imaging device 52 is equipped with a fisheye lens for photographing at an angle of view of 200 degrees. In this case, the image acquisition unit 11 acquires the semi-celestial sphere image 51 photographed by the first imaging unit 31 as the first image, and acquires the semi-celestial sphere image 52 photographed by the second imaging unit 32 as the second image. . However, as described above, the image acquisition does not need to be performed in real time, and when the images captured by the first imaging unit 31 and the second imaging unit 32 are stored in the storage unit 20 in advance, the image acquisition unit 11 Acquires these images from the storage unit 20.

  Returning to FIG. 2, next, the common area specifying unit 12 specifies a common area that is a common area between the first image and the second image (Step S102). For example, in FIG. 3, the range of the angle from the lens center of each imaging unit in the range of 80 degrees to 100 degrees (the end of the captured image) is an area in which a common subject is captured (a common area). . Therefore, the common region specifying unit 12 specifies a region whose angle from the lens center falls within a range of 80 degrees to 100 degrees as a common region. (In FIG. 3, a solid line at an angle of 100 degrees from the center of the lens is indicated by a solid line 53, and a line at an angle of 80 degrees from the center of the lens is indicated by a dotted line 54. A horizon is indicated by a dotted line 55. In FIG. 3, the horizon coincides with a circumferential line having an angle of 90 degrees from the lens center.)

  Since a common subject (the same object or the like) is photographed in such a common area, a feature point (a characteristic portion such as an edge, a corner, or a pattern of the subject) is extracted from each image in the common area. The feature points indicating the same part of the subject can be extracted from both the first image and the second image. This is called a corresponding feature point. That is, there is a first feature point representing the portion in the first image and a second feature point representing the portion in the second image via the characteristic portion on the subject. Become. At this time, the first feature point and the second feature point correspond to each other, and the first feature point and the second feature point form a pair of corresponding feature points. In practice, first, feature points are extracted from the first image, and a feature point similar to the feature point is searched for in the second image to find a pair of feature points.

  For example, when the common region of the hemisphere image 51 of FIG. 3 is developed in polar coordinates, as shown in FIG. 4, an angle θ from the lens center and a predetermined position of the circumference at the angle θ (for example, the hemisphere of FIG. 3) It can be represented as a rectangular area by the angle φ when the left end of the image is set to 0 degree. Similarly, when the common area of the hemisphere image 52 in FIG. 3 is developed in polar coordinates, as shown in FIG. 5, an angle θ from the lens center and a predetermined position on the circumference at the angle θ (for example, the hemisphere in FIG. 3) It can be represented as a rectangular area by the angle φ when the left end of the image is set to 0 degree.

  If the first imaging unit 31 accurately captures the semi-celestial sphere image 51 in the direction directly above and the second imaging unit 32 accurately captures the semi-celestial sphere image 52 in the downward direction, the dotted line 55 indicating the horizon becomes θ = 90 °, and the same subject is photographed in the rectangular area shown in FIG. 4 and the rectangular area shown in FIG. Therefore, in this case, the positions of the feature points of both captured images match, and if the two images are overlapped and connected so that the positions of these feature points match, a natural spherical image can be generated. Become.

  However, in practice, the first imaging unit 31 and the second imaging unit 32 can hardly be installed in exactly opposite directions, and the characteristics of the cameras of the two imaging units are often slightly different. Therefore, normally, it is not possible to join the two images by overlapping the common area as it is. For example, in the example illustrated in FIG. 6, the first imaging unit 31 accurately captures the semi-celestial sphere image 51 right above, but the second imaging unit 32 shifts the semi-celestial sphere image 5 degrees from directly below. 52 shows a case where an image 52 is being taken. Then, the dotted line 55 indicating the horizon in the semi-celestial sphere image 52 deviates from the dotted line 56 indicating the circumference having an angle of 90 degrees from the lens center.

  In this case, if the common area of the semi-celestial sphere image 51 in FIG. 6 is developed in polar coordinates, it can be represented as a rectangular area shown in FIG. 7, as in FIG. However, when the common area of the hemisphere image 52 in FIG. 6 is developed in polar coordinates, as shown in FIG. 8, a dotted line 55 indicating the horizon intersects a dotted line 56 indicating the circumference of θ = 90 degrees in a wave shape. That is, the rectangular area shown in FIG. 7 and the rectangular area shown in FIG. 8 are common areas, but cannot be overlapped as they are.

  This is the situation under normal circumstances. In this case, first, a feature point is extracted from one rectangular area, and a feature point similar to the feature point is searched for in the other rectangular area to find a pair of feature points (corresponding feature point). If a projective transformation matrix that matches the positions of feature points (pairs of feature points) is calculated, one image is deformed using this matrix, and the other image and the common region are overlapped and connected, A spherical image can be generated. When calculating the projection transformation matrix, a RANSAC (Random Sample Consensus) algorithm can be used to increase the accuracy. In addition, when a moving image is targeted, such two images are joined for each frame.

  Returning to FIG. 2, such processing will be described in order. Although the common area is specified by the common area specifying unit 12 in step S102, if the common area is represented by a two-dimensional image, the first image and the second image corresponding to the common area are respectively set to a predetermined number. (For example, 8 × 8). Then, the feature point specifying unit 13 specifies a pair of feature points corresponding to the first image and the second image in each cell (step S103). For example, a feature point B in a second image corresponding to a feature point A in a certain cell (a, b) in the first image (assuming that coordinates in the cell are (c, d)) is searched for. In this case, first, with respect to the coordinates (c, d) in the cell (a, b) in the second image, a point whose feature amount is similar to the feature amount of the feature point A from the vicinity to the periphery is determined. Search for. In this search, for example, when a point whose similarity is equal to or more than a predetermined threshold value is found, the point may be set as a feature point B in the second image corresponding to the feature point A, or a square ( The similarity to the feature point A may be calculated for all the points in a, b), and the most similar point may be set as the feature point B in the second image corresponding to the feature point A. The feature point specifying unit 13 specifies a pair of corresponding feature points (feature point A and feature point B) in this way, for example.

  Then, for the pair of the feature points identified by the feature point identification unit 13 (the feature point in the first image and the feature point in the second image), the movement vector calculation unit 14 calculates the two features constituting the pair. The deviation of the position of the point is calculated as a vector (movement vector) (step S104). As shown in FIG. 9, the movement vector calculated by the movement vector calculation unit 14 starts from the position of the feature point in the first image in each cell, and corresponds to the position in the second image corresponding to the feature point. A vector indicating where the feature point has moved (end point). In the example of FIG. 9, since the number of cells is 8 × 8 = 64, 64 movement vectors are calculated.

  Returning to FIG. 2, next, the transformation matrix selection unit 18 initializes a “selection transformation matrix” that is a variable for storing an optimal transformation matrix (step S105). Then, based on the plurality of movement vectors calculated by the movement vector calculation unit 14, the conversion matrix calculation unit 15 converts the position of the feature point in the first image to a corresponding feature point in the second image. A matrix is calculated (step S106). This transformation matrix is, for example, a projective transformation matrix. When the first image and the second image are two-dimensional images, the projective transformation matrix can be calculated from four movement vectors.

  Therefore, the transformation matrix calculation unit 15 randomly selects these four movement vectors from the 64 movement vectors, and calculates a projection transformation matrix using the selected movement vectors (selected movement vectors). For example, assuming that four movement vectors surrounded by a circle 61 in FIG. 9 are randomly selected movement vectors, the transformation matrix calculator 15 solves the simultaneous equations from the values of the four movement vectors to solve the simultaneous equations. Calculate the matrix.

  In addition, when the motion vectors extracted from the first image and the second image are three-dimensional, five motion vectors are required to calculate the projective transformation matrix. In the first embodiment, since the first image and the second image are both hemispherical images, the feature point specifying unit 13 determines whether the first feature point in the first image and the second feature point in the second image are the same. After specifying the feature points, the movement vector calculation unit 14 projects the first feature point and the corresponding second feature point onto a spherical surface having a radius of 1, and calculates the 3rd of each feature point obtained by projection. A three-dimensional movement vector is calculated using the dimensional coordinates. Then, the conversion matrix calculation unit 15 calculates a conversion matrix by solving a simultaneous equation using the five three-dimensional movement vectors.

  Returning to FIG. 2, next, the conversion unit 16 converts the position of the feature point in the first image using the conversion matrix calculated by the conversion matrix calculation unit 15, and changes the position after the conversion to the shifted feature point. (Step S107). For example, FIG. 10 illustrates a vector (converted vector) having the position of a feature point in the first image as a start point and the position of the moved feature point obtained by converting the position of the feature point by the conversion unit 16 as an end point, FIG. 10 is a diagram superimposed and displayed with the movement vector (excluding the selected movement vector) shown in FIG. 9. Ideally, the motion vector matches the vector after conversion (at least in the cell in which the selected motion vector is selected), but actually, a shift occurs as shown in FIG.

  Returning to FIG. 2, the error calculation unit 17 calculates this deviation as an error (step S108). Although there are a plurality of movement vectors (and converted vectors) to be evaluated (in the example shown in FIG. 10, 60 movement vectors excluding four selected movement vectors from the 64 movement vectors), the error is evaluated here. As the values, three types of evaluation values are calculated. In any of the evaluation values listed here, the larger the value, the smaller the error (that is, the better the transformation matrix).

  The first evaluation value is a quantization representative class, which is referred to as evaluation value 1. In order to obtain the evaluation value 1, the error calculating unit 17 first obtains the square of the absolute value of the difference between the vectors (the difference between the moving vector and the converted vector), and sets the squared value as the SD (Squared Difference) value. In the example shown in FIG. 10, 60 SD values are obtained. Then, the error calculator 17 quantizes (classifies) the plurality (here, 60) of the SD values into stages (classes) of a predetermined number of classes (for example, 20).

The quantization method is arbitrary, but, for example, a certain support range threshold value A is defined. If the SD value is larger than A, it is classified into class 0; if the SD value is smaller than A and larger than A / 2, it is classified into class 1, a / ^{2 2} classification if the class 2 is greater than, · · ·, a ^{/ 2 17} or less and classified in class 18 is greater than a ^{/ 2 18,} classified into a ^{/ 2 18} or less if the class 19, to the 20 It can be quantized in stages.

Then, the error calculator 17 sets a class including a median value among a plurality (here, 60) of SD values as a representative class. For example, the median of the SD values if A / ^{2 2,} a representative class Class 3. Then, the error calculating unit 17 sets the representative class number (for example, 3 if the median SD value is A / ² 22) as the evaluation value 1.

  The second evaluation value is the number of SD values within the support range, which is referred to as evaluation value 2. In this case, first, the square of the absolute value of the difference between the respective vectors (the difference between the movement vector and the converted vector) is obtained, and is set as the SD value. Then, a certain support range threshold value A is defined, and the number of SD values equal to or smaller than A among a plurality of (60 in this case) SD values is set as the evaluation value 2.

  The third evaluation value is an evaluation value represented by the number of SD values within the support range among evaluation values calculated using the transformation matrix selected in the immediately preceding frame. I do. For example, first, the position of the feature point in the first image of the current frame is set as a starting point, and the position of the feature point is converted by the conversion matrix selected in the immediately preceding frame (the feature point after moving the previous frame). (The position after the previous frame) with the end point of (the position of the previous frame) as the end point. Then, the square of the absolute value of the difference between this vector (the vector after the previous frame conversion) and the movement vector of the current frame is obtained, and is set as the SD value. Then, a certain support range threshold value A is defined, and the number of SD values equal to or smaller than A among a plurality (here, 60) of SD values is set as the evaluation value 3.

  Of the above three evaluation values, evaluation value 1 and evaluation value 2 are values for evaluating an error in the current frame (current frame), and are therefore also referred to as current frame errors. Since the evaluation value 3 is a value for evaluating an error in the previous frame, it is also called a previous frame error.

  Then, the transformation matrix selection unit 18 determines whether or not the error calculated by the error calculation unit 17 satisfies a predetermined condition (Step S109). This determination is made based on the three evaluation values described above. First, if the evaluation value 1 is larger than the previous evaluation value 1 in the same loop (the loop from step S106 to step S111), it is determined that “the error satisfies a predetermined condition”. If the evaluation value 1 is smaller than the previous evaluation value 1, it is determined that "the error does not satisfy the predetermined condition". If the evaluation value 1 is equal to the evaluation value 1 so far, if the value of the evaluation value 2 + the evaluation value 3 is larger than the value of the evaluation value 2 + the evaluation value 3 so far, it is determined that “the error satisfies a predetermined condition”, If the value of the evaluation value 2 + the evaluation value 3 is equal to or less than the value of the evaluation value 2 + the evaluation value 3 so far, it is determined that “the error does not satisfy the predetermined condition”.

  If the error satisfies the predetermined condition (Step S109; Yes), the conversion matrix selection unit 18 updates the variable “selection conversion matrix” with the conversion matrix calculated in this loop (Step S106) (Step S110). Then, the process returns to step S106.

  If the error does not satisfy the predetermined condition (Step S109; No), the conversion matrix selection unit 18 performs the specified number of times (for example, 50) without updating the variable “selection conversion matrix” (without going to Step S110) (Step S106). It is determined whether or not the calculation of the transformation matrix in step (1) is repeated (step S111).

  If it has not been repeated the specified number of times (step S111; No), the process returns to step S106. If it has been repeated the specified number of times (Step S111; Yes), the image generation unit 19 converts the first image using the conversion matrix (the conversion matrix selected by the conversion matrix selection unit 18) stored in the variable “selection conversion matrix”. Then, an image in which the first image and the second image are connected is generated by overlapping the common area with the second image (step S112).

  Then, the control unit 10 determines whether the processing has been completed for all frames of the moving image (Step S113). If all of the processing has not been completed yet (step S113; No), the process returns to step S101 to acquire an image of the next frame and repeat the processing up to this point. When the processing has been completed for all the frames (step S113; Yes), the image processing ends.

  The processing content of the image processing has been described above. Next, an example in which the first image and the second image are joined by the above-described image processing will be described with reference to the image shown in FIG.

  In FIG. 11, the right end of the first image and the left end of the second image are a common area 71 and a common area 72, respectively. Since the first image and the second image are slightly shifted in the vertical direction, if the two images are joined by overlapping the common area 71 and the common area 72 as shown in FIG. In addition, an unnatural image having a step in the joint region 73 is generated. (Here, it is assumed that two images are connected by overwriting the common area 71 with the common area 72.)

  When the above-described image processing is performed, the conversion matrix calculation unit 15 calculates a conversion matrix for converting an image so that such a step does not occur, and the image generation unit 19 converts the first image using the conversion matrix. To generate an image joined with the second image. Then, as shown in FIG. 13, the first image and the second image exactly overlap in the joint region 74, and a natural image is generated.

  If attention is paid only to a certain frame in the moving image, a natural image as shown in FIG. 13 is generated, and it seems that there is no particular problem. However, when the difference in the transformation matrix between frames is not considered at all, for example, the image generation unit 19 generates an image as shown in FIG. 14 in a certain frame, and generates an image as shown in FIG. 15 in the next frame. Can be generated. In this case, a natural image is generated in each frame, but the image shown in FIG. 14 and the image shown in FIG. 15 are vertically displaced. For easier understanding, FIG. 16 shows an image obtained by superimposing the image shown in FIG. 14 and the image shown in FIG. Therefore, when this is reproduced as a moving image, an unnatural moving image in which blur occurs in the vertical direction.

  Therefore, in the above-described image processing, the error calculation unit 17 calculates the evaluation value 3 also using the conversion matrix calculated in the immediately preceding frame, and the conversion matrix selection unit 18 uses the evaluation value 3 to calculate the optimal conversion matrix. Is selected, it is possible to reduce the possibility of selecting a transformation matrix that causes unnaturalness between frames.

  As described above, in the image processing apparatus 100, when selecting the optimal conversion matrix, the evaluation based on the conversion matrix selected in the previous frame is also taken into consideration, so that the conversion matrix selected in the previous frame is considered. A transformation matrix similar to is easily selected. By doing so, the temporal discontinuity of the moving images causes a plurality of moving images to generate a moving image composed of a wide range of images in which frames of each moving image are connected at their ends. The possibility that an unnatural moving image is generated can be reduced.

(Modification 1)
In the first embodiment, in step S109 of the image processing (FIG. 2), if the evaluation value 1 calculated by the error calculation unit 17 is equal to the evaluation value 1 up to that point, the conversion matrix selection unit 18 evaluates the evaluation value 2 + the evaluation value 3 Is larger than the previous evaluation value 2 + evaluation value 3, it is determined that the error satisfies the predetermined condition. If the evaluation value 2 + evaluation value 3 is equal to or less than the previous evaluation value 2 + evaluation value 3, "The error does not satisfy the predetermined condition." However, it is not limited to this. The evaluation value 2 is the number of SD values within the support range in the current frame, and the evaluation value 3 is the number of SD values within the support range among SD values calculated using the transformation matrix selected in the previous frame. As for the number, for example, as a first modified example, these values may be multiplied by a coefficient, weighted, and the added value may be compared.

  In the first modification, the coefficient by which the evaluation value 2 is multiplied is W2, and the coefficient by which the evaluation value 3 is multiplied is W3. Then, in step S109 of the image processing (FIG. 2), if the evaluation value 1 calculated by the error calculation unit 17 is equal to the evaluation value 1 up to that point, the conversion matrix selection unit 18 determines that W2 × evaluation value 2 + W3 × evaluation value 3 Is larger than the previous value of W2 × Evaluation value 2 + W3 × Evaluation value 3, it is determined that the error satisfies a predetermined condition, and the value of W2 × Evaluation value 2 + W3 × Evaluation value 3 is W2 × Evaluation value If it is equal to or less than 2 + W3 × evaluation value 3, it is determined that “the error does not satisfy the predetermined condition”.

  In the first modification, by multiplying the evaluation value 2 and the evaluation value 3 by respective coefficients, it is possible to adjust the degree of influence of the transformation matrix selected in the immediately preceding frame.

(Modification 2)
Further, in the first embodiment, in step S108 of the image processing (FIG. 2), the error calculation unit 17 calculates the above-described three types of evaluation values, but is not limited thereto. For example, as a second modified example, the error calculation unit 17 determines, as a fourth evaluation value, an evaluation value represented by a quantization representative class among evaluation values calculated using a transformation matrix selected in the immediately preceding frame. The value may be calculated. This evaluation value is referred to as evaluation value 4.

  As in the case of calculating the evaluation value 3, the error calculation unit 17 first selects the position of a feature point in the first image of the current frame as a start point and selects the position of the feature point in the immediately preceding frame. A vector (post-frame converted vector) having the position converted by the converted matrix (position of the feature point after moving the previous frame) as the end point is obtained. Then, the error calculator 17 calculates the square of the absolute value of the difference between this vector (the vector after the previous frame conversion) and the movement vector of the current frame, and sets the square as the SD value. As in the case of calculating the evaluation value 1, a plurality of SD values (60 in the example shown in FIG. 10) are obtained.

  Then, the error calculator 17 quantizes (classifies) the plurality (here, 60) of the SD values into stages (classes) of a predetermined number of classes (for example, 20). The quantization method here is also arbitrary. For example, the above-described quantization method (when obtaining the evaluation value 1) can be used. Then, the error calculator 17 sets a class including a median value among a plurality (here, 60) of SD values as a representative class. Then, the error calculating unit 17 sets the representative class number as the evaluation value 4.

  Then, in step S109 of the image processing (FIG. 2), the transformation matrix selection unit 18 determines a predetermined threshold value as a reference class threshold value, and if the evaluation value−the evaluation value 1 ≦ the reference class threshold value, “the error is a predetermined value” The condition is not satisfied "(step S109; No), and the process proceeds to step S111. As a result, the selected movement vector selected when calculating the conversion matrix in the current loop (the loop from step S106 to step S111) and the selected movement vector selected when calculating the selected conversion matrix in the previous frame are calculated. When the difference from the selected movement vector is large, the process can immediately proceed to the next loop. If the selected movement vectors are significantly different, it is assumed that the transformation matrix calculated in the immediately preceding frame and the transformation matrix calculated in the current frame are significantly different, so the transformation matrix calculated in the current loop Is rejected.

  In the second modification, by doing so, when there is a high possibility that a transformation matrix significantly different from the transformation matrix selected in the previous frame is calculated, the transformation matrix is rejected, and This increases the probability that an appropriate transformation matrix will be selected.

(Modification 3)
Further, in the first embodiment, in step S108 of the image processing (FIG. 2), the error calculation unit 17 calculates the above-described three types of evaluation values, but is not limited thereto. For example, as a third modification, the error calculating unit 17 may calculate the following two evaluation values (here, evaluation values 5 and 6) instead of these three types of evaluation values. .

  In the third modification, the error calculation unit 17 obtains the square of the absolute value of the difference between the vectors (the difference between the movement vector and the converted vector), sets the squared value as an SD (Squared Difference) value, and further calculates the sum of the SD values. An SSD (Sum of Squared Difference) is obtained, and this is set as an evaluation value 5.

  When calculating the evaluation value 6, the error calculating unit 17 first selects the position of a feature point in the first image of the current frame as a start point and selects the position of the feature point in the immediately preceding frame. A vector (a vector after the previous frame conversion) having the position converted by the obtained conversion matrix (the position of the feature point after the previous frame movement) as the end point is obtained. Then, the square of the absolute value of the difference between this vector (the vector after the previous frame conversion) and the movement vector of the current frame is obtained, the SD value is obtained, and the SSD which is the sum of the SD values is further obtained. And Note that the evaluation values 5 and 6 are different from the above-described evaluation values, and the smaller the value, the smaller the error (that is, the better the conversion matrix).

  Then, in step S109 of the image processing (FIG. 2), if the evaluation value 5 + the evaluation value 6 is smaller than the previous evaluation value 5 + the evaluation value 6, the conversion matrix selection unit 18 determines that “the error satisfies the predetermined condition”. . If the evaluation value 5 + the evaluation value 6 is larger than the previous evaluation value 5 + the evaluation value 6, it is determined that “the error does not satisfy the predetermined condition”. Note that, similarly to the above-described first modification example, by multiplying each evaluation value by a coefficient (for example, multiplying the evaluation value 5 by W5 and multiplying the evaluation value 6 by W6), the conversion selected in the immediately preceding frame is performed. The degree of influence of the matrix may be adjusted.

  In the third modification, by using the SSD as the error evaluation value, it is not necessary to quantize the SD value into a plurality of classes, or to calculate the number of SD values within the support range. be able to.

(Modification 1 of Modification 3)
Further, as a first modification of the third modification, in step S108 of the image processing (FIG. 2), the error calculating unit 17 determines the evaluation value 5 and the evaluation value 6, and then determines the reference SSD threshold as a predetermined threshold. In step S109, if the evaluation value 6 is equal to or larger than the reference SSD threshold, the conversion matrix selection unit 18 may determine that “the error does not satisfy a predetermined condition” (step S109; No) and proceed to step S111.

  By doing so, the selected movement vector selected when calculating the conversion matrix in the current loop (the loop from step S106 to step S111) and the conversion matrix selected in the previous frame are calculated. When the difference from the selected movement vector is large, the transformation matrix calculated in the current loop is rejected, and the process can immediately proceed to the next loop.

  In the first modification of the third modification, by doing so, if there is a high possibility that a transformation matrix significantly different from the transformation matrix selected in the previous frame is likely to be calculated, the transformation matrix is rejected. This increases the possibility that a more appropriate transformation matrix is selected.

(Modification 2 of Modification 3)
In the first modification of the third modification, in step S109 of the image processing (FIG. 2), if the evaluation value 6 is equal to or larger than the reference SSD threshold, the conversion matrix selecting unit 18 determines that “the error does not satisfy the predetermined condition”. However, the present invention is not limited to this. For example, as a second modification of the third modification, a reference SSD ratio (n%) is set as a predetermined ratio, and in step S109, the conversion matrix selecting unit 18 determines that the evaluation value 6 is the reference SSD ratio of the evaluation value 5 In this case, it may be determined that “the error does not satisfy the predetermined condition” (Step S109; No), and the process may proceed to Step S111.

  By doing so, the selected movement vector selected when calculating the conversion matrix in the current loop (the loop from step S106 to step S111) and the conversion matrix selected in the previous frame are calculated. When the difference from the selected movement vector is large, the transformation matrix calculated in the current loop is rejected, and the process can immediately proceed to the next loop.

  In the second modification of the third modification, by doing so, if there is a high possibility that a transformation matrix significantly different from the transformation matrix selected in the previous frame is likely to be calculated, the transformation matrix is rejected. This increases the possibility that a more appropriate transformation matrix is selected.

(Modification 4)
In the first embodiment, the selected transformation matrix is initialized in step S105 of the image processing (FIG. 2) (that is, immediately before entering a loop in which the transformation matrix is changed to select an optimal transformation matrix). It is not necessary. For example, as a fourth modification, as shown in FIG. 17, instead of the process of step S105, the error calculating unit 17 uses the previous selected conversion matrix without initializing the previous selected conversion matrix, (Evaluation value) may be calculated (step S121). In this way, in the subsequent loop (from step S106 to step S111), the previous selected transformation matrix is treated as an initial value.

  Then, even if a transformation matrix that is too far from the previous selected transformation matrix is calculated in step S106, it is immediately rejected, and the possibility that a more appropriate transformation matrix is selected is increased.

(Modification 5)
Further, as a fifth modification, as shown in FIG. 18, it may be determined whether or not the error (each evaluation value) calculated in step S121 satisfies a predetermined condition (step S122). Specifically, a predetermined reference threshold is set, and if each evaluation value is higher than the reference threshold, the conversion matrix selection unit 18 determines that “the selection conversion matrix in the previous frame can be used in the current frame without any problem. And does not update the selection conversion matrix (leave the value in the previous frame).

  That is, in the fifth modification, when the transformation matrix selection unit 18 determines in step 122 that the error satisfies the predetermined condition (step S122; Yes), the process proceeds to step S112, and the image generation unit 19 performs the processing in the previous frame. An image is generated using the selection transformation matrix of. If it is determined that the error does not satisfy the predetermined condition (Step S122; No), the process goes to Step S106, and enters a loop for selecting a transformation matrix in the current frame. The error satisfies the predetermined condition, for example, when the error is smaller than a predetermined threshold.

  In the fifth modification, when the selected transformation matrix in the previous frame can be used, the process of calculating the transformation matrix in the current frame can be omitted, so that the processing load can be reduced.

(Modification 6)
Further, in the first embodiment, when the image of the frame is obtained in step S101 of the image processing (FIG. 2), the process of calculating the transformation matrix is performed every time. However, this process may not be performed depending on the situation. For example, as a sixth modification, as shown in FIG. 19, after step S101, the control unit 10 may acquire a situation (step S131) and determine whether there is a change in the situation (step S132). Good.

  Here, the situation is a surrounding environment or a movement of the imaging unit. For example, by attaching an acceleration sensor to the imaging unit and checking the value obtained from the acceleration sensor by the control unit 10, it can be determined whether or not the imaging unit has moved. Also, without using an acceleration sensor, a change in the situation can be determined by comparing the captured image with the image captured in the previous frame. Specifically, the control unit 10 calculates the difference between the images themselves between the previous frame and the current frame (for example, calculates the absolute value of the pixel value difference in pixel units, Is calculated, and if the calculated value is smaller than a predetermined threshold, it is determined that there is no change in the situation, and if the calculated value is equal to or greater than the threshold, it is determined that there is a change in the situation.

  When the control unit 10 determines that there is no change in the situation (step S132; No), the process proceeds to step S112, and the image generation unit 19 generates an image using the selection conversion matrix in the previous frame. If it is determined that there is a change in the situation (step S132; Yes), the process proceeds to step S102, and a process for selecting a new transformation matrix in the current frame is started. In steps S131 and S132 described above, the control unit 10 functions as a situation determination unit.

  In the sixth modification, the conversion matrix is calculated only when the situation changes, so that the processing load can be reduced.

(Modification 7)
In the above-described embodiment and the modified example, the transformation matrix selection unit 18 optimizes using the evaluation value in the transformation matrix calculated in the current frame and the evaluation value in the transformation matrix calculated in the immediately preceding frame. Selection of a suitable transformation matrix is performed, but the present invention is not limited to this. For example, as a seventh modification, an optimal conversion matrix may be selected using not only the immediately preceding frame but also the evaluation value of the conversion matrix calculated in the two preceding frames. Furthermore, an optimal conversion matrix may be selected using the evaluation values of the conversion matrix calculated in the frame three or more frames before.

  In Modification 7, the possibility of selecting a more appropriate transformation matrix can be increased by using a plurality of evaluation values in past frames.

  In the above-described embodiment and the modification, the image processing apparatus 100 includes two imaging units (the first imaging unit 31 and the second imaging unit 32). However, for example, when the image acquiring unit 11 can acquire a plurality of moving images by acquiring a moving image from another device or acquiring a moving image stored in the storage unit 20, the image processing apparatus Need not include an imaging unit.

  Further, in the above-described embodiment and the modified example, the image processing apparatus 100 connects the first image and the second image. However, the number of images to be connected is not limited to two. The image processing apparatus may generate an image in which three or more images are connected based on a common area with each of the adjacent images.

  The above-described first embodiment and each of the modifications can be appropriately combined. For example, when Modification Example 4 (FIG. 17) is combined with Modification Example 6 (FIG. 19), if there is no change in the situation, the previous selected transformation matrix becomes the current selected transformation matrix. Since the transformation matrix is treated as the initial value of the current transformation matrix, it is possible to reduce the processing load and increase the possibility of selecting a more appropriate transformation matrix.

  Also, when Modification 6 (FIG. 19) is combined with Modification 5 (FIG. 18), if there is no change in the situation, the previous selected transformation matrix becomes the current selected transformation matrix. If the error in the selected transformation matrix is small, the previous selected transformation matrix will be the current selected transformation matrix, otherwise the previous selected transformation matrix will be treated as the initial value of the current transformation matrix, so the processing load will be further reduced. It is possible to increase the possibility that a more appropriate transformation matrix is selected while reducing the number.

  Note that each function of the image processing apparatus 100 can also be performed by a computer such as a normal PC (Personal Computer). Specifically, in the above-described embodiment, the description has been given assuming that the program of the image processing performed by the image processing apparatus 100 is stored in the ROM of the storage unit 20 in advance. However, the program may be stored in a flexible disk, a compact disc read only memory (CD-ROM), a digital versatile disc (DVD), a magneto-optical disc (MO), a memory card, a USB (Universal Serial Readable Memory) memory such as a computer. A computer capable of realizing the above-described functions by storing and distributing the program in a recording medium, reading the program into the computer, and installing the program may be configured.

  As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to the specific embodiments, and the present invention includes the inventions described in the claims and equivalents thereof. It is. Hereinafter, the inventions described in the claims of the present application will be additionally described.

(Appendix 1)
Image acquisition means for acquiring each of the first image and the second image having an end common to the first image in frame units;
Common area specifying means for specifying a common area that is an area common between the first image and the second image obtained by the image obtaining means;
A feature point specifying unit that specifies a pair of corresponding feature points in the first image and the second image in the common region specified by the common region specifying unit;
Based on a pair of feature points specified by the feature point specifying means, a first feature point in the first image is set as a starting point, and a second feature point in the second image corresponding to the first feature point is set. A movement vector calculation unit that calculates a plurality of movement vectors, ending with the feature point of
Conversion matrix calculation means for calculating a conversion matrix based on a selected movement vector selected from the plurality of movement vectors calculated by the movement vector calculation means,
A conversion unit that calculates a position of a post-movement feature point that is a position obtained by converting the position of the first feature point in the first image, using the conversion matrix calculated by the conversion matrix calculation unit;
Error calculation means for calculating an error between the position of the second feature point and the position of the moved feature point calculated by the conversion means;
In a predetermined frame, from among the plurality of conversion matrices calculated by the conversion matrix calculation unit, a conversion matrix selection unit that selects a conversion matrix whose error calculated by the error calculation unit satisfies a predetermined condition,
Image generation means for generating an image obtained by joining the first image and the second image using the conversion matrix selected by the conversion matrix selection means;
With
The conversion matrix selection unit is a conversion matrix that satisfies a predetermined condition from a conversion matrix selected in a frame before the predetermined frame and a plurality of conversion matrices calculated by the conversion matrix calculation unit in the predetermined frame. Select a matrix,
Image processing device.

(Appendix 2)
Further provided is a situation determination means for determining a situation,
The conversion matrix selection unit, when the situation determination unit determines that there is no change in the situation, selects a conversion matrix selected in a frame before the predetermined frame as a conversion matrix in the predetermined frame,
The image processing device according to supplementary note 1.

(Appendix 3)
The conversion unit uses a conversion matrix selected in a frame before the predetermined frame and a conversion matrix selected by the conversion matrix selection unit in the predetermined frame, and uses the conversion matrix in the first image. The position of the feature point is calculated by converting the position of the first feature point,
The error calculation means is based on the position of the post-movement feature point calculated by the conversion means, a previous frame error which is an error based on a conversion matrix selected in a frame before the predetermined frame, and The current frame error, which is an error based on the selected transformation matrix, is calculated,
The conversion matrix selection unit, from among a plurality of conversion matrices calculated by the conversion matrix calculation unit in the predetermined frame and the conversion matrix selected in the frame before the predetermined frame, the previous frame error and the Selecting a transformation matrix whose current frame error satisfies a predetermined condition,
3. The image processing apparatus according to claim 1 or 2.

(Appendix 4)
When the previous frame error is smaller than a predetermined threshold, the conversion matrix selection unit selects a conversion matrix selected in a frame before the predetermined frame as a conversion matrix in the predetermined frame.
An image processing device according to attachment 3.

(Appendix 5)
The first image and the second image are hemispherical images, and generate a full spherical image by combining them.
The image processing device according to any one of supplementary notes 1 to 4.

(Appendix 6)
Acquiring a first image and a second image having an end common to the first image in frame units,
Specifying a common area that is a common area between the obtained first image and the second image;
In the specified common area, a pair of feature points corresponding to the first image and the second image is specified,
Based on the identified pair of feature points, a first feature point in the first image is set as a start point, and a second feature point in the second image corresponding to the first feature point is set as an end point. , Calculate a plurality of movement vectors,
Calculating a transformation matrix based on the selected movement vector selected from the plurality of calculated movement vectors,
Using the calculated transformation matrix, calculate the position of the moved feature point, which is the position obtained by converting the position of the first feature point in the first image,
Calculating an error between the position of the second feature point and the calculated position of the moved feature point;
In a predetermined frame, from the plurality of calculated conversion matrices, the calculated error selects a conversion matrix that satisfies a predetermined condition,
Using the selected transformation matrix, generate an image obtained by joining the first image and the second image,
An image processing method,
In the selection of the transformation matrix, from a plurality of transformation matrices calculated in the predetermined frame and the transformation matrix selected in the frame before the predetermined frame, to select a conversion matrix that satisfies a predetermined condition,
Image processing method.

(Appendix 7)
On the computer,
Acquiring a first image and a second image having an end common to the first image in frame units,
Specifying a common area that is a common area between the obtained first image and the second image;
In the specified common area, a pair of feature points corresponding to the first image and the second image is specified,
Based on the identified pair of feature points, a first feature point in the first image is set as a start point, and a second feature point in the second image corresponding to the first feature point is set as an end point. , Calculate a plurality of movement vectors,
Calculating a transformation matrix based on the selected movement vector selected from the plurality of calculated movement vectors,
Using the calculated transformation matrix, calculate the position of the moved feature point, which is the position obtained by converting the position of the first feature point in the first image,
Calculating an error between the position of the second feature point and the calculated position of the moved feature point;
In a predetermined frame, from the plurality of calculated conversion matrices, the calculated error selects a conversion matrix that satisfies a predetermined condition,
Using the selected transformation matrix, generate an image obtained by joining the first image and the second image,
A program for executing a process,
In the selection of the transformation matrix, from a plurality of transformation matrices calculated in the predetermined frame and the transformation matrix selected in the frame before the predetermined frame, to select a conversion matrix that satisfies a predetermined condition,
program.

DESCRIPTION OF SYMBOLS 10 ... Control part, 11 ... Image acquisition part, 12 ... Common area specification part, 13 ... Feature point specification part, 14 ... Movement vector calculation part, 15 ... Conversion matrix calculation part, 16 ... Conversion part, 17 ... Error calculation part, 18: transformation matrix selection unit, 19: image generation unit, 20: storage unit, 31: first imaging unit, 32: second imaging unit, 51, 52: semi-celestial sphere image, 53: solid line, 54, 55, 56 ... Dotted line, 61: circle, 71, 72: common area, 73, 74: area, 100: image processing apparatus

Claims (7)

Image acquisition means for acquiring each of the first image and the second image having an end common to the first image in frame units;
Common area specifying means for specifying a common area that is an area common between the first image and the second image obtained by the image obtaining means;
A feature point specifying unit that specifies a pair of corresponding feature points in the first image and the second image in the common region specified by the common region specifying unit;
Based on a pair of feature points specified by the feature point specifying means, a first feature point in the first image is set as a starting point, and a second feature point in the second image corresponding to the first feature point is set. A movement vector calculation unit that calculates a plurality of movement vectors, ending with the feature point of
Conversion matrix calculation means for calculating a conversion matrix based on a selected movement vector selected from the plurality of movement vectors calculated by the movement vector calculation means,
A conversion unit that calculates a position of a post-movement feature point that is a position obtained by converting the position of the first feature point in the first image, using the conversion matrix calculated by the conversion matrix calculation unit;
Error calculation means for calculating an error between the position of the second feature point and the position of the moved feature point calculated by the conversion means;
In a predetermined frame, from among the plurality of conversion matrices calculated by the conversion matrix calculation unit, a conversion matrix selection unit that selects a conversion matrix whose error calculated by the error calculation unit satisfies a predetermined condition,
Image generation means for generating an image obtained by joining the first image and the second image using the conversion matrix selected by the conversion matrix selection means;
With
The conversion matrix selection unit is a conversion matrix that satisfies a predetermined condition from a conversion matrix selected in a frame before the predetermined frame and a plurality of conversion matrices calculated by the conversion matrix calculation unit in the predetermined frame. Select a matrix,
Image processing device. Further provided is a situation determination means for determining a situation,
The conversion matrix selection unit, when the situation determination unit determines that there is no change in the situation, selects a conversion matrix selected in a frame before the predetermined frame as a conversion matrix in the predetermined frame,
The image processing device according to claim 1. The conversion unit uses a conversion matrix selected in a frame before the predetermined frame and a conversion matrix selected by the conversion matrix selection unit in the predetermined frame, and uses the conversion matrix in the first image. The position of the feature point is calculated by converting the position of the first feature point,
The error calculation means is based on the position of the post-movement feature point calculated by the conversion means, a previous frame error which is an error based on a conversion matrix selected in a frame before the predetermined frame, and The current frame error, which is an error based on the selected transformation matrix, is calculated,
The conversion matrix selection unit, from among a plurality of conversion matrices calculated by the conversion matrix calculation unit in the predetermined frame and the conversion matrix selected in the frame before the predetermined frame, the previous frame error and the Selecting a transformation matrix whose current frame error satisfies a predetermined condition,
The image processing device according to claim 1. When the previous frame error is smaller than a predetermined threshold, the conversion matrix selection unit selects a conversion matrix selected in a frame before the predetermined frame as a conversion matrix in the predetermined frame.
The image processing device according to claim 3. The first image and the second image are hemispherical images, and generate a full spherical image by combining them.
The image processing apparatus according to claim 1. Acquiring a first image and a second image having an end common to the first image in frame units,
Specifying a common area that is a common area between the obtained first image and the second image;
In the specified common area, a pair of feature points corresponding to the first image and the second image is specified,
Based on the identified pair of feature points, a first feature point in the first image is set as a start point, and a second feature point in the second image corresponding to the first feature point is set as an end point. , Calculate a plurality of movement vectors,
Calculating a transformation matrix based on the selected movement vector selected from the plurality of calculated movement vectors,
Using the calculated transformation matrix, calculate the position of the moved feature point, which is the position obtained by converting the position of the first feature point in the first image,
Calculating an error between the position of the second feature point and the calculated position of the moved feature point;
In a predetermined frame, from the plurality of calculated conversion matrices, the calculated error selects a conversion matrix that satisfies a predetermined condition,
Using the selected transformation matrix, generate an image obtained by joining the first image and the second image,
An image processing method,
In the selection of the transformation matrix, from a plurality of transformation matrices calculated in the predetermined frame and the transformation matrix selected in the frame before the predetermined frame, to select a conversion matrix that satisfies a predetermined condition,
Image processing method. On the computer,
Acquiring a first image and a second image having an end common to the first image in frame units,
Specifying a common area that is a common area between the obtained first image and the second image;
In the specified common area, a pair of feature points corresponding to the first image and the second image is specified,
Based on the identified pair of feature points, a first feature point in the first image is set as a start point, and a second feature point in the second image corresponding to the first feature point is set as an end point. , Calculate a plurality of movement vectors,
Calculating a transformation matrix based on the selected movement vector selected from the plurality of calculated movement vectors,
Using the calculated transformation matrix, calculate the position of the moved feature point, which is the position obtained by converting the position of the first feature point in the first image,
Calculating an error between the position of the second feature point and the calculated position of the moved feature point;
In a predetermined frame, from the plurality of calculated conversion matrices, the calculated error selects a conversion matrix that satisfies a predetermined condition,
Using the selected transformation matrix, generate an image obtained by joining the first image and the second image,
A program for executing a process,
In the selection of the transformation matrix, from a plurality of transformation matrices calculated in the predetermined frame and the transformation matrix selected in the frame before the predetermined frame, to select a conversion matrix that satisfies a predetermined condition,
program.

Images