JP4809134B2 - Image processing apparatus and processing method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus and a processing method thereof. In detail, when tracking the dynamic image when the imaging device moves relative to the object and measuring the coordinates of the imaging device or the object, an image with a partly high resolution is obtained from a plurality of images. The present invention relates to an image processing apparatus and an image processing method capable of forming and performing highly accurate three-dimensional measurement using an image with high resolution.

  There has been a technique for measuring the position of an imaging apparatus by continuously imaging an object while performing relative movement. However, when an imaging device is actually mounted on a moving body such as an automobile or when a person takes a picture with a hand, a stable image is not necessarily obtained due to the shaking of the automobile or person. It is necessary to correct vertical movement, tilt, etc., and other cars, people, Asuka, fallen leaves etc. enter between the imaging device and the object, and the feature points are hidden behind and disappear, It may be revived again. In addition, it may be difficult to see the movement in the three-dimensional space on the two-dimensional image. Therefore, it is necessary to perform processing for identifying and deleting feature points and corresponding points that are inappropriate for three-dimensional measurement, as well as such processing for swinging the photographing apparatus and processing for feature points that disappear and are reproduced. On the other hand, a technology that forms a wide-field, high-resolution image by laminating low-resolution videos, and a technology that uses a low-resolution image input device to acquire a wide range of high-resolution images without using a special mechanical scanning mechanism Is disclosed. (See Patent Document 1 and Non-Patent Document 1)

JP-A-10-69537 (paragraphs 0032 to 0128, FIGS. 1 to 14 and the like) NEC Technical Review 2005 Vol. 58, no. 3, June 2005

  However, all of the conventional high-resolution techniques target a two-dimensional image, and in order to measure the three-dimensional coordinates of a photographing apparatus or an object from a moving image, they correspond to each other on two or more moving images. A technique for obtaining a feature point, that is, the same point on an object (hereinafter referred to as a corresponding feature point) and tracking the same is required. Further, since the super-resolution processing is performed for a number of corresponding feature points, there is a problem that the processing load is large and a large memory capacity is required.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of accurately measuring the shooting position, posture or three-dimensional coordinates of an object of a shooting device from a shot image that changes in time series such as a moving image. And It is another object of the present invention to provide an image processing apparatus and an image processing method capable of performing super-resolution processing with high speed and small memory capacity.

In order to solve the above problem, the image processing apparatus 100 according to the first aspect of the present invention , for example, as shown in FIG. A captured image acquisition unit 2 that acquires a series of captured images captured in time series so as to be shared, a feature extraction unit 3 that extracts feature points from any captured image captured in time series, and a series of A feature point tracking unit 4 that tracks feature points of a captured image and associates the feature points with each other, and selects an original image group that is an image group for performing super-resolution processing from a series of captured images. The feature point tracking unit 4 extracts main feature points as feature points for performing super-resolution processing from the feature points associated with each other, and sets a partial image in a small area around the main feature points. , Apply the super-resolution processing to the partial image and convert the partial super-resolution image A super-resolution processing unit 8 to be formed; a stereo image selection unit 6 that selects a stereo image that is a paired image from a group of captured images in which a partial super-resolution image is formed by the super-resolution processing unit 8; Using the corresponding feature points (representing the same point on the object) of the stereo images selected by the stereo image selection unit 6 to perform the orientation and three-dimensional measurement, the object A stereo image selection unit 6 for determining the position of the object, the shape of the object, or the position of the imaging device that has imaged the object, and the stereo image selection unit 6 has a baseline length of the stereo image that exceeds a predetermined threshold. Select a stereo image to be longer.

  Here, the relatively moving object is typically shot with either the object or the imaging device moving and the other stationary, but is shot with both moving together. Also good. That is, the image may be taken in a state in which both move relatively. It is possible to track feature points by sharing an overlapping portion between three or more adjacent images. The larger the number of overlapping images, the better because the measurement coordinate accuracy can be improved. For example, 10 or more is preferable, and 50 or more is more preferable. A series of captured images taken in time series is captured images that are sequentially acquired over time, and may be images extracted from frames of moving images continuously captured by a video camera. It may be an image taken sequentially at an appropriate time interval by a camera. Further, it may be acquired from all the frames of the moving image or may be acquired every several frames. Also, the tracking of feature points is performed by searching corresponding points corresponding to the second image with respect to the feature points of the first image. Corresponding corresponding points in the third image are searched as points, and feature points are sequentially associated and tracked. Here, feature points associated with each other in a series of captured images are referred to as corresponding feature points. Also, feature points are newly generated, disappeared, and reproduced with the passage of time. Therefore, in a series of photographed images, the feature points associated with each other may be in at least three or more photographed images. In addition, the selection of the stereo image is selected from a correction image in which the position, magnification, and inclination are corrected by reflecting the tracking process for a series of captured images, or a displacement correction image from which vertical parallax has been removed by the displacement correction processing. Although it is preferable, even if it selects from the acquired original image, since it is matched with each other by the orientation process and the three-dimensional measurement, that may be sufficient. The orientation is a process for calculating the shooting position and inclination of the imaging apparatus, and the three-dimensional measurement is a process for calculating the three-dimensional coordinates of each feature point.

  If comprised in this way, the image processing apparatus which can measure the three-dimensional coordinate of the imaging | photography position and attitude | position of an imaging device or a target object from the captured image which changes in time series, such as a moving image, can be provided. Further, it is possible to provide an image processing apparatus capable of performing super-resolution processing with high speed and small memory capacity using partial images. In addition, since the stereo image is selected such that the base line length of the stereo image is longer than a predetermined threshold, a certain three-dimensional measurement accuracy can be ensured.

Further, according to a second aspect of the present invention, in the image processing apparatus according to the first aspect, for example, as shown in FIG. 2, the super-resolution processing unit 8 performs a super-resolution process from a series of photographed images. An original image group selection unit 81 that selects an original image group that is an image group, and a main point as a feature point for performing super-resolution processing from the feature points associated with each other in the feature point tracking unit 4 in the original image group A main feature point extracting unit 82 that extracts feature points, a partial image setting unit 83 that sets a partial image in a small area around the main feature points extracted by the main feature point extracting unit 82, and a partial image setting unit 83 A partial image enlarging unit 84 for enlarging the set partial image, a plurality of sub feature points are extracted around the main feature points of the partial image, the sub feature points are tracked between the partial images of the captured image group, and the sub features are extracted. Corresponding between the sub-feature point tracking unit 85 that associates points with each other and the sub-feature point tracking unit 85 A partial image deforming unit 86 for deforming the partial image so that the position coordinates of the sub-feature points are well matched between the original images, and a super-resolution partial image from a plurality of partial images deformed by the partial image deforming unit 86 A super-resolution partial image forming unit 87 for forming the image, a partial image reduction unit 88 for reducing the super-resolution partial image formed by the super-resolution partial image forming unit 87 to the dimensions of the original partial image, and partial image reduction A super-resolution image synthesis unit 89 that synthesizes the partial super-resolution image by fitting the partial image reduced by the unit 88 to the position of the original partial image in the captured image.

  In this case, either the partial image enlargement process or the sub-feature point extraction / tracking process can be performed first, but extraction and tracking after enlargement is easier to achieve higher resolution. The partial image may be enlarged by 2 times, 5 times, 10 times, or the like. Also, when a super-resolution partial image is formed from a plurality of partial images, typically the center of gravity of the feature points in the partial image is obtained, but weighting may be performed, and other statistical processing such as the median value is performed. You may do it. In addition, the fitting does not have to be performed so that the end portions coincide with each other, and the position coordinates of the main feature points may be overlapped. If comprised in this way, the super-resolution image which has arrange | positioned the high-resolution partial image everywhere can be formed efficiently.

The third aspect of the present invention, first in the second image processing apparatus according to the embodiment of the stereo image selection section 6, the captured image group partially super resolution image as the base length is approximately equal is formed Select multiple stereo images from. With this configuration, three-dimensional coordinates with uniform accuracy can be obtained.

The fourth aspect of the present invention is an image processing apparatus according to the first to third one aspect, the stereo image selection section 6, partly super-resolution image by super-resolution processing section 8 is formed has been selected stereo images from photographed images, the orientation process, three-dimensional measurement unit 7 performs the matching processing for partial super resolution image of the stereo images selected. With this configuration, the three-dimensional position accuracy can be improved by the rematching process.

Further, according to a fifth aspect of the present invention, in the image processing device according to any one of the first to fourth aspects , the super-resolution processing unit 8 captures the position coordinates of the object or the object measured in advance. The super-resolution processing is performed using the position coordinates of the photographing apparatus, and the orientation processing / three-dimensional measurement unit 7 performs orientation and three-dimensional measurement using the result of the super-resolution processing. Here, the position coordinates measured in advance may be position coordinates measured three-dimensionally after super-resolution processing once, or may be position coordinates measured separately three-dimensionally without super-resolution processing, and are not based on image processing. Alternatively, position coordinates measured three-dimensionally by a GPS or an inertial sensor may be used. If comprised in this way, a suitable stereo image can be selected using the position coordinate obtained by three-dimensional measurement.

In addition, according to a sixth aspect of the present invention, in the image processing device according to the fifth aspect , the stereo image selection unit 6 has a correspondence table that represents the relationship between the baseline length of the stereo image and the measurement accuracy of the three-dimensional coordinates. Then, a stereo image is selected from the correspondence table so that the baseline length satisfies the required accuracy. If comprised in this way, the three-dimensional coordinate suitable for a required precision will be obtained.

Further, in the image processing method according to the seventh aspect of the present invention , for example, as shown in FIG. 3, a relatively moving object is time-sequentially such that three or more adjacent images share an overlapping portion. A captured image acquisition step (S100) for acquiring a series of captured images taken in time, a feature extraction step (S110) for extracting feature points from any of the captured images taken in time series, and a series of captured images A feature point tracking step (S120) for tracking feature points and associating the feature points with each other, and selecting an original image group as an image group for performing super-resolution processing from the series of captured images. Extracting main feature points as feature points for performing super-resolution processing from the feature points associated with each other in the feature point tracking step, setting a partial image in a small area around the main feature points, Perform super-resolution processing on images and partially A super-resolution processing step (S130) for forming a resolution image, and a stereo image for selecting a stereo image that is a paired image from a group of captured images on which a partial super-resolution image is formed in the super-resolution processing step Using the feature points associated with each other of the stereo image selected in the selection step (S140) and the stereo image selection step, orientation and three-dimensional measurement are performed, the position of the object, the shape of the object, or the object An orientation processing / three-dimensional measurement step (S150, S160) for obtaining the position of the photographing device that photographed the object, and in the stereo image selection step (S140), the stereo length is set to be longer than a predetermined threshold Select an image. If comprised in this way, the image processing method which can measure the three-dimensional coordinate of the imaging | photography position and attitude | position of an imaging device or a target object accurately from the imaging | photography image which changes in time series, such as a moving image, can be provided. Further, it is possible to provide an image processing method capable of performing super-resolution processing with high speed and small memory capacity using partial images. In addition, since the stereo image is selected such that the base line length of the stereo image is longer than a predetermined threshold, a certain three-dimensional measurement accuracy can be ensured.

In addition, according to an eighth aspect of the present invention, in the image processing method according to the seventh aspect, for example, as shown in FIG. 8, the super-resolution processing step (S130) performs super-resolution processing from a series of photographed images. An original image group selecting step (S131) for selecting an original image group, which is an image group, and a feature point for performing super-resolution processing from the feature points associated with each other in the feature point tracking step in the original image group A main feature point extracting step (S132) for extracting a main feature point as, a partial image setting step (S133) for setting a partial image in a small area around the main feature point extracted in the main feature point extracting step, A partial image enlarging step (S134) for enlarging the partial image set in the partial image setting step, a plurality of sub feature points are extracted around the main feature point of the partial image, and between the partial images of the captured image group To track the sub-feature point and Sub-feature point tracking step (S135) for associating points with each other, and a partial image deformation step for deforming the partial image so that the position coordinates of the sub-feature points associated in the sub-feature point tracking step are well matched between the original images (S136), a super-resolution partial image formation step (S137) for forming a super-resolution partial image from a plurality of partial images deformed in the partial image deformation step, and a super-resolution image formed in the super-resolution partial image formation step A partial image reduction step (S138) for reducing the resolution partial image to the size of the original partial image, and the partial image reduced in the partial image reduction step is fitted into the position of the original partial image in the captured image, A super-resolution image synthesizing step (S139) for synthesizing a super-resolution image. If comprised in this way, the super-resolution image which has arrange | positioned the high-resolution partial image everywhere can be formed efficiently.

ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing method which can measure the imaging | photography position, attitude | position of an imaging device, or the three-dimensional coordinate of a target object accurately from the imaging | photography image which changes in time series, such as a moving image, can be provided.
In addition, it is possible to provide an image processing apparatus and an image processing method capable of performing super-resolution processing with high speed and small memory capacity.

  Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram for explaining the concept in the present embodiment. This is an example in which a camera is attached to an automobile, time, that is, the position of the automobile is changed little by little, and an urban area as an object is photographed, and the position coordinates of the camera, that is, the locus of the automobile are obtained from the tracking results in these captured images. This makes it possible to continuously display the position of the car in the car navigation, but it is also significant to be used complementarily in a section where GPS radio waves cannot be received. In the first embodiment, an example will be described in which measurement of three-dimensional coordinates is performed with high accuracy by performing partial super-resolution processing on a captured image that changes in time series such as a moving image.

  FIG. 2 shows a configuration example of the image processing apparatus 100 in the present embodiment. In the figure, reference numeral 1 denotes a control unit that controls each unit of the image processing apparatus 100 so as to function as an image processing apparatus. Specifically, the control unit 1 instructs the captured image acquisition unit 2 to acquire a captured image and the feature extraction unit 3. Instructed to execute feature point extraction, instructed to perform tracking to the feature point tracking unit 4, directed to select a stereo image to the arithmetic processing unit 5, directed to execute orientation / three-dimensional measurement, and instructed to execute super-resolution processing.

  Reference numeral 2 denotes a captured image acquisition unit that sequentially acquires captured images that change in time series, such as moving images. In addition to acquiring captured images, the captured image is output to the feature extraction unit 3, stored in the image memory 10, and the like. I do. Note that a captured image may be acquired by communication from another imaging device without performing imaging with the own imaging device. Reference numeral 3 denotes a feature extraction unit that extracts feature points from sequentially acquired captured images. Extraction of feature points from the captured image input from the captured image acquisition unit 2 and extraction of the extracted feature points to the feature point tracking unit 4 are performed. Perform output, etc. 4 is a corresponding point corresponding to the feature point input from the feature extraction unit 3 (strictly speaking, it is a candidate corresponding point until it is determined, but in this embodiment, it is a corresponding point including the candidate corresponding point). It is a feature point tracking unit that searches and tracks feature points. In addition to the tracking process, the tracking result is output to the corresponding point information memory 9A, the arrangement of the corresponding points, and the feature points to the feature extracting unit 3 Instruct new establishment. A series of photographed images in which feature points are associated in the feature point tracking unit 4, that is, to which the corresponding feature points are attached, are corrected images whose positions, magnifications, and inclinations are corrected to reflect the tracking process, or biases described later. The displacement correction image from which the vertical parallax has been removed by the position correction process is stored in the image memory 10 together with the acquired original image.

  An arithmetic processing unit 5 includes a super-resolution processing unit 8, a stereo image selection unit 6, and an orientation processing / three-dimensional measurement unit 7. The super-resolution processing unit 8 selects an original image group that is an image group for performing the super-resolution processing from the series of photographed images, and the feature points associated with each other by the feature point tracking unit 4 in the original image group. Extract main feature points from the points as feature points for super-resolution processing, set a partial image in a small area around the main feature points, perform super-resolution processing on the partial image, and perform partial super-resolution An image is formed. The stereo image selection unit 6 selects a stereo image from a series of images to which corresponding feature points are attached in the feature extraction unit 3 and the feature point tracking unit 4. The orientation processing / three-dimensional measurement unit 7 performs orientation calculation and three-dimensional measurement using the stereo image selected by the stereo image selection unit 6. In addition, the displacement correction processing, the orientation result, and the three-dimensional measurement result Is output to the display unit 11, and the orientation result and the three-dimensional measurement result are output to the outside. One or more stereo images may be used for orientation calculation and three-dimensional measurement, but accuracy can be improved by performing statistical processing such as averaging using a large number of stereo images.

The super-resolution processing unit 8 selects an original image group that is an image group for performing super-resolution processing from a series of captured images, and the original image selected by the original image group selecting unit 81. In the group, a main feature point extracting unit 82 that extracts main feature points as feature points for performing super-resolution processing from the feature points associated with each other by the feature point tracking unit 4, and a main feature point extracting unit 82 A partial image setting unit 83 for setting a partial image in a small area around the main feature point extracted in step (a), a partial image enlargement unit 84 for enlarging the partial image set by the partial image setting unit 83, and a partial image enlargement unit. A sub-feature point tracking unit that extracts a plurality of sub-feature points around the main feature points of the partial image enlarged in 84, tracks the sub-feature points between the partial images of the captured image group, and associates the sub-feature points with each other. 85 and the position of the sub feature point associated with the sub feature point tracking unit 85 A partial image deformation unit 86 that deforms the partial images so that the original images are well matched with each other, and a super-resolution partial image formation that forms a super-resolution partial image from a plurality of partial images deformed by the partial image deformation unit 86 A partial image reduction unit 88 that reduces the super-resolution partial image formed by the super-resolution partial image forming unit 87 to the size of the original partial image, and the partial image reduced by the partial image reduction unit 88 And a super-resolution image composition unit 89 that synthesizes a partial super-resolution image by fitting it at the position of the original partial image in the captured image. The partial super-resolution image formed by the super-resolution processing unit 8 is stored in the image memory 10, and the stereo image is selected from the partial super-resolution image by the stereo image selection unit 6, and the orientation processing / three-dimensional measurement unit 7, orientation calculation and three-dimensional measurement are executed.

11 is a display unit that displays an image of an object subjected to orientation processing or three-dimensional measurement by the arithmetic processing unit 5 and a trajectory of a captured position in a two-dimensional or three-dimensional manner (three-dimensional two-dimensional display), and 9 is a storage unit. Corresponding point information memory 9A for storing information on feature points and their corresponding points (including candidate corresponding points), stereo image information memory 9B for storing information on stereo images, captured images, corrected images, displacement corrected images, and the like An image memory 10 for storing images is included. The corresponding point information memory 9A, the stereo image information memory 9B, and the image memory 10 are referred to as needed at the time of feature point tracking, super-resolution processing, stereo image selection, orientation calculation, three-dimensional measurement, etc. Also written.

  FIG. 3 shows a flow example of the image processing method according to the first embodiment. First, the photographic image acquisition unit 2 acquires a series of photographic images that change in a time series such as a moving image for an object that moves relatively (S100: photographic image acquisition step). A series of captured images are acquired so that three or more adjacent images share an overlapping portion. The image may be obtained by taking an image with its own photographing camera, or an image taken with another photographing apparatus may be obtained via a communication line. The acquired captured image is stored in the image memory 10. The control unit 1 sequentially supplies a photographic image that changes in time series such as a moving image from the photographic image acquisition unit 2 to the feature extraction unit 3. In this embodiment, the camera is mounted on a car and the camera is photographed while moving. Therefore, the photographed image is a photographed image that changes little by little in time or space, and the object is common to most of the neighboring images. is there. The feature extraction unit 3 extracts feature points from one of the captured images taken in time series (S110: feature extraction step), and the extracted feature point data is supplied to the feature point tracking unit 4 to correspond points. It is stored in the information memory 9A.

  It is possible to track feature points by sharing an overlapping portion between three or more adjacent images. The feature point tracking unit 4 searches for a corresponding point corresponding to the feature point in the series of photographed images, and associates and tracks the feature point (S120: feature point tracking step). In the tracking of the feature point, the corresponding point of the second image is searched for the feature point of the first image, and when the corresponding point is matched, the corresponding point of the second image is set as a new feature point. Corresponding points of the image are searched, and the correspondence and feature points are sequentially tracked. Here, feature points associated with each other in a series of captured images are referred to as corresponding feature points. Corresponding feature point data is stored in the corresponding point information memory 9A, and tracking results are also stored as history information. Tracking information regarding each corresponding point is accumulated in the corresponding point information memory 9A, and a candidate corresponding point list is created in the corresponding point information memory 9A. That is, in the candidate corresponding point list, coordinates on the two-dimensional screen are stored for each captured image for each corresponding feature point. In addition, after the three-dimensional coordinate measurement, the three-dimensional coordinates are also stored. In addition, a determination result as to whether or not the corresponding feature point is appropriate is also stored. In addition, the captured image with the corresponding feature points attached to the original image is a corrected image in which the position, magnification, and tilt are corrected by reflecting the tracking process, or a displacement corrected image from which vertical parallax is removed by the displacement correcting process. Is stored in the image memory 10 (S125). Although the corrected image or the deviation corrected image is stored in S125, the stored image data can be used as the correction parameter even when the image is not displayed. If the feature point data extracted by the feature extraction unit 3 and the corresponding feature point data correlated by the feature point tracking unit 4 are sequentially supplied to the arithmetic processing unit 5 in real time, a moving object (such as an automobile) ), It is highly possible that the orientation process and the three-dimensional measurement are performed at an early stage and reflected in the navigator.

Next, super-resolution processing is performed. In the super-resolution processing step (S130), an original image group that is an image group for performing the super-resolution processing is selected from a series of captured images, and the original image group is associated with each other in the feature point tracking step (S120). A main feature point is extracted as a feature point for performing super-resolution processing from the obtained feature point, a partial image is set in a small area around the main feature point, and a portion of the partial image is subjected to super-resolution processing A super-resolution image is formed. Details of the super-resolution processing step (S130) will be described later with reference to FIG.

  Next, a stereo image is selected from a series of photographed images to which corresponding feature points are attached (S140: stereo image selection step). The series of images preferably uses a corrected image in which magnification, fluctuation, and inclination are corrected to reflect the tracking process, or a displacement corrected image from which vertical parallax has been removed by the displacement correcting process. Even if selected from the above, it can be correlated with each other in the next orientation process and three-dimensional measurement, so that may be used. In order to select a stereo image, various combinations are possible as the candidates, but a stereo pair candidate list is created in the stereo image information memory 9B in which information on the combination selected as a stereo image candidate is stored. The stereo pair candidate list stores the captured image numbers constituting the stereo pair, the focal length of each captured image, the capturing direction, and the base line length when the pair is formed, and after the displacement correction processing, the displacement corrected image is stored. After measuring the number and three-dimensional coordinates, the shooting distance is also stored. Therefore, the stereo image selection unit can select an appropriate stereo image with reference to this stereo pair candidate list.

  FIG. 4 shows an example of stereo image selection. In order to perform three-dimensional measurement, it is necessary to select two images (referred to as stereo images) that make a pair from the photographed images, match the feature points, and confirm that they correspond to each other. In this example, an image several images away from the acquired image is selected as a stereo image. If the frame interval between two images that form a pair is made constant, the baseline length can be kept substantially constant, which is preferable. One or more stereo images may be selected. However, since accuracy can be improved by performing statistical processing such as averaging using a large number of stereo images, a large number of tracked images are used here. A stereo image is selected and statistical processing is performed.

  Returning to FIG. 3, using the corresponding feature points of the stereo image selected by the stereo image selection unit 6, the orientation processing and the three-dimensional measurement unit 7 perform orientation processing and three-dimensional measurement. In the orientation process, relative orientation is performed on the selected stereo image using the coordinates of the feature points and the corresponding points, and the photographing position and inclination of the photographing camera are calculated (S150: first orientation step). Connection orientation between stereo images is also performed. Note that, for a captured image for which a stereo image could not be set, the shooting position and tilt of the shooting camera can be calculated by performing single photo orientation. A displacement correction process is performed following the relative orientation process in the orientation process / three-dimensional measurement unit 7 (S155: displacement correction process).

FIG. 5 is a diagram for explaining the deviation correction processing. The deviation correction process is performed in parallel with the subject so that the stereo image geometry is established with respect to the measurement object using the shooting position and inclination of the shooting apparatus obtained by the orientation process. This is a process of correcting the image so that the epipolar line matches the left and right horizontal lines. Therefore, distortions such as magnification and inclination of the left and right images are corrected, the magnification is the same, and vertical parallax is removed.
Returning again to FIG. The three-dimensional coordinates of each feature point are calculated by three-dimensional measurement using the orientation result and the displacement correction image data (S160: first three-dimensional measurement step). The three-dimensional measurement is easily obtained by using the equations (5) to (7) from the principle of the stereo method shown in FIG. Alternatively, the three-dimensional measurement is obtained, for example, by performing bundle adjustment using the position and inclination (external orientation element) of the photographing camera obtained by the orientation process as initial values. As a result, not only the three-dimensional coordinates but also a more accurate position and inclination of the photographing camera can be obtained.

Next, when it is desired to further improve the three-dimensional coordinate accuracy, the process returns to the super-resolution processing step (S130) again, then the stereo image selection step (S140) is performed again, and the orientation is performed again (S150: second). Next, using the orientation result, a displacement correction process (S155: second displacement correction step) and another three-dimensional measurement are performed (S160: second three-dimensional measurement step). . In the second orientation process and the second 3D measurement process, orientation with higher accuracy is performed by using the super-resolution image again, and 3D coordinates are calculated. By repeating this loop processing (S130 to S160), the measurement accuracy can be further improved. For example, the loop processing is repeated until the target accuracy is obtained. Thereby, a three-dimensional coordinate is decided and a measurement is complete | finished.
Below, each process is demonstrated as needed.

[Feature point extraction]
The feature extraction unit 3 extracts feature points from each captured image (S110, see FIG. 6). Typically, the initial frame is extracted from the entire screen, and the next frame is extracted from a new screen area that does not overlap the initial frame. For extracting feature points in the initial frame, for example, the MORAVEC operator (HP Moravec. Towers Automatic Visual Observed Aviation, Proc. 5th International Joint Conference, Artisp. An operator such as can be appropriately employed.

[Tracking process]
FIG. 6 shows a processing flow example of feature point tracking. The feature point tracking unit 4 tracks each feature point selected by the feature extraction process (S120). That is, a candidate corresponding point corresponding to a feature point is obtained, a feature point movement vector and a screen relative movement amount are obtained, and these are connected to obtain a movement locus. The screen relative movement amount is a relative movement amount on the screen between the photographing camera and the photographing object, and the movement vector is a relative movement vector of each feature point on the two-dimensional photographed image. In tracking feature points, first, template matching is performed on adjacent captured images (S13) to obtain candidate corresponding points corresponding to the feature points. Thereby, the movement vector of each feature point is obtained. Further, by performing projective transformation using the adjacent photographed image (S15), the screen relative movement amount with respect to the photographing camera is obtained. That is, since the overall movement between frames is very short in time, it is assumed that it can be approximated by projective transformation, and the screen relative movement amount is estimated by projective transformation. Next, the movement vector of each feature point is compared with the relative movement amount of the screen between frames, and the quality of the movement vector is determined (S14). Then, candidate corresponding points that are considered to be erroneous correspondences indicating abnormal movement are removed (S16). By repeating steps S15 and S16, the accuracy of projective transformation is improved.

  Next, the arrangement of candidate corresponding points is determined (S17). That is, the arrangement of feature points and corresponding points on the captured image is confirmed. If the arrangement of feature points is extremely biased and a blank portion is generated, the feature extraction unit 3 is instructed to newly set a point existing in the newly generated blank portion as a new feature point. Then, the process returns to the feature extraction (S110) again, and the feature extraction (S110) and the tracking process (S120) are sequentially repeated in real time for new adjacent images. If feature extraction has been completed for a series of captured images, the process returns to template matching (S13), and after that, tracking processing (S120) is sequentially performed for new adjacent images.

  For example, template matching is used for tracking feature points (S13). Neighboring images are sequentially selected from the acquired captured images to form a stereo pair, and for example, stereo matching is performed by a technique such as an SSDA (Sequential Similarity Detection Algorithm) method to obtain corresponding points (S13). In the SSDA method, the similarity is determined using the residual, and the position where the partial matrix residual is minimum is obtained as the corresponding point. SSDA template matching is considered to be relatively fast as template matching and easy to implement in hardware. Other methods such as a normalized correlation method can also be employed. For template matching, it is important to optimally select a template size and a search range, and the search range is set optimally based on the frame rate, moving speed, etc. of the video camera.

  In the shooting screen, when a feature point is given to a moving object such as a running car, a person, an asuka, a fallen leaf, or when the camera shakes heavily, an erroneous response point may occur. The camera swing can be corrected by projective transformation. On the other hand, an object that moves differently from the object to be photographed causes a miscorresponding point. Therefore, by removing the miscorresponding points caused by the movement of the object etc., the reliability of the feature points (including corresponding points and candidate corresponding points) is improved, the accuracy of the mismatching determination is improved, It is possible to cope with upsets.

[Super-resolution processing]
Next, super-resolution processing will be described. Super-resolution processing is processing that estimates high-resolution and high-resolution images from multiple images, or reconstructs the current image from multiple degraded images, assuming that the original image was a high-resolution and high-resolution image. Refers to processing. Specifically, a super-resolution image is obtained by repeatedly performing alignment, deformation, and composition between a plurality of images.

  FIG. 7 shows an example of a super-resolution image. FIG. 7A shows an image with high resolution by a conventional bicubic method, and FIG. 7B shows a super-resolution image formed from 16 images. When both are compared, it can be seen that a clearer image is obtained with the super-resolution image.

  In general, when super-resolution image processing is performed, processing is performed on the entire image or a wide area of the image. However, super-resolution image processing requires accurate alignment between images and estimation of deformation parameters. And In contrast, in the present embodiment, partial super-resolution processing is performed. That is, a group of original images for performing super-resolution processing is selected from a series of captured images, main feature points are extracted from the original images, a partial image is set in a small area around the main feature points, and enlarged. A plurality of sub feature points are extracted around the main feature points of the enlarged partial image, tracked and correlated, and the partial images are transformed so that the position coordinates of the sub feature points are well matched between the original images. A super-resolution image is formed from a plurality of deformed partial images, is reduced, and is fitted into the position of the original partial image to form a partial super-resolution image. As described above, the super-resolution processing in the present embodiment improves the resolution in a limited range around the feature point, and if accurate alignment and deformation between images are performed in a limited small area. It is enough.

  For alignment between feature points, SSDA template matching can be used. In addition, projection transformation or affine transformation used for feature point tracking can be applied to deformation between images. In order to create an image with a resolution higher than that of the original image, the images are combined after the image enlargement / reduction processing. Therefore, in general, an enormous memory capacity is required. Since super-resolution processing is performed only in a small area, a large amount of memory is not required.

FIG. 8 shows a flow example of the super-resolution processing step. First, an original image group that is an image group for performing super-resolution processing is selected from a series of photographed images (S131: original image group selecting step). Next, main feature points are extracted as corresponding feature points for performing super-resolution processing from the feature points associated with each other in the feature point tracking unit 4 in the original image group (S132: main feature point extracting step). ), A partial image is set in a small area around the main feature point (S133: partial image setting step). In selecting the original image group, it is preferable to select so that the moving distances on the screen between adjacent images are substantially equal in order to obtain a uniform super-resolution image. In addition, since the number of images in the original image group also affects the coordinate accuracy of the corresponding points of the super-resolution image, it is preferable to make the number of images substantially equal to obtain a homogeneous super-resolution image. In the present embodiment, an original image group is selected so that the moving distances on the screen between adjacent images are substantially equal, and the number of images in the original image group when forming a plurality of partial super-resolution images Are also made equal. Next, the partial image is enlarged (S134: partial image enlargement step), and a plurality of sub feature points are extracted around the main feature point of the enlarged partial image (S135: sub feature point extraction step). For the enlargement process, for example, a Biliner process or a Bicubic process is used.

  FIG. 9 is a diagram for explaining an example of partial image setting. In the figure, F0 is a captured image (including a corrected image and a displacement corrected image), f1 to f5 are partial images set in the captured image F0, P1 to P5 represent main feature points, and the surrounding x is extracted. Represents the position of the sub-feature point. In this way, partial areas are set in a small area in the captured image F0, and super-resolution processing is performed in these partial images f1 to f5. For other original images of the captured image, partial images are set for the same (corresponding to each other) main feature points, and sub feature points are extracted. The partial image is preferably extracted so as to be uniformly distributed in the captured image F0 in order to obtain a uniform accuracy over the entire image F0, and the sub-feature points are extracted so as to be uniformly distributed in the partial image. It is preferable for obtaining uniform accuracy in the partial image. The sub feature points may include corresponding feature points tracked by the feature point tracking unit 4.

  Next, tracking processing is performed on the sub-feature points between the partial images f1 to f5 of the original image group, and the sub-feature points are associated with each other (S135: sub-feature point tracking step). Next, alignment / deformation processing is performed for each of the partial images f1 to f5 so that the position coordinates of the sub-feature points associated with each other are in good agreement (S136: partial image deformation process). Next, a super-resolution partial image is formed from a plurality of deformed partial images (S137: super-resolution partial image forming step).

  FIG. 10 is a diagram for explaining an example of super-resolution partial image formation. In the figure, the position of one arbitrary sub-feature point is indicated by “x” in the partial image of each original image in the original image group. These position coordinates vary, and when forming a super-resolution partial image, typically, the position coordinates of the sub-feature points of the super-resolution partial image are the center of gravity of the positions of the sub-feature points in the original image group. However, the center of gravity may be obtained by weighting according to the proximity of the frame number or the like.

  Next, each formed super-resolution partial image is reduced to the size of the original partial image (S138: partial image reduction process), and the reduced partial images f1 to f5 are the original portions in the captured image F0. The image is inserted into the position of the image, and a partial super-resolution image is synthesized (S139: super-resolution image synthesis step). The super-resolution partial images f1 to f5 are fitted at the positions of the original partial images in FIG. In this embodiment, the super-resolution partial image is inserted only in the center frame of the original image group. However, the same super-resolution partial image may be inserted in other original images. Note that, when it is not necessary to fit the super-resolution partial image, such as when only the coordinate value is required, the fitting can be omitted. Further, it is possible to improve the resolution by forming a repetitive loop returning from the super-resolution image synthesis (S139) to the original image group selection (S131). In this embodiment, the repetitive loop is provided. .

FIG. 11 shows an example of a flow for each frame of the super-resolution processing step. A photographed image (including a corrected image and a displacement-corrected image) that forms a partial super-resolution image is defined as an i-th frame Fi, and a total of 2n + 1 original image groups (frame Fj (j = i) This is an example of super-resolution processing using −n to i + n)). In the feature point tracking unit 4, each captured image is acquired and used as a template image of each frame (S100). Feature points are tracked between these captured images and associated with each other to obtain a movement locus (S120). In the tracking process, conversion between frames is performed, and a conversion parameter (for example, an affine conversion parameter) is obtained (S126). From the feature point tracking unit 4, template image information of 2n + 1 original image groups, feature point movement trajectory information, and deformation information of each inter-frame image are output to the super-resolution processing unit 8.

For the i-th frame Fi, the super-resolution image is initialized. Initialization is performed by setting a partial image as a main feature point (S133) and enlarging (S134). For the enlargement process, for example, a Biliner process is used. On the other hand, for the original image group consisting of the i-th frame Fi and n frames before and after, a total of 2n + 1 frames Fj (F _{i−n to} F _{i + n} ), a partial image is similarly set as the main feature point (S133). Is enlarged (S134). Further, the sub-feature points around the main feature points are tracked and associated (S135). In this process, for example, affine transformation is used to perform alignment / deformation processing (S136). Feature point movement trajectory information and inter-frame deformation information in the feature point tracking process can be used, which facilitates the alignment / deformation process.

Next, a super-resolution partial image is formed from a plurality of modified partial images in 2n + 1 frames Fj (F _{i−n to} F _{i + n} ) (S137). A super-resolution partial image is formed by obtaining the center of gravity of the position of the sub-feature point in the original image group and using this as the position coordinates of the sub-feature point of the super-resolution partial image. The super-resolution partial image is reduced (S138), and is inserted into the position of the original partial image of the i-th frame image Fi and synthesized (S139). As a result, a partial super-resolution image in which only the partial image region is subjected to the super-resolution processing is formed (for example, this is referred to as a j-th partial super-resolution image), and the super-resolution synthesis buffer in the image memory 10 Stored in At this time, if there is an existing partial super-resolution image in the super-resolution synthesis buffer, it is compared with this (S13A), and if there is a change, the existing partial super-resolution image stored in the super-resolution synthesis buffer is changed. It is rewritten to a new partial super-resolution image. This process is repeated until there is no change between the newly synthesized partial super-resolution image and the partial super-resolution image stored in the super-resolution synthesis buffer. This repetition corresponds to the repetition loop in FIG. If there is no change, the repetition is terminated and a new partial super-resolution image is output (S13B). In this case, as another group of original images for forming a new partial super-resolution image, it may be selected from the partial super-resolution image formed by the super-resolution processing. This is preferable for obtaining an image. Alternatively, high accuracy can be achieved by selecting an image group in which the frame interval between original images is increased. In this embodiment, another original image group is selected from partially super-resolution images formed by super-resolution processing.

[Stereo matching]
FIG. 12 shows the relationship between the three-dimensional coordinate position of the corresponding feature point and the camera position. The measurement point P of the object A is stereo-photographed by the stereo cameras C1 and C2 (the camera position is indicated by C1 and C2 in FIG. 12). The captured image (original image) I is acquired by reflecting the tilt and position fluctuation of the camera. However, the deviation correction image M that has been subjected to the deviation correction process described later corrects the tilt and position fluctuation of the camera, and is epipolar. The image is suitable for distance measurement with a line. The base line length B of the camera can be easily obtained by calculation from the two photographing positions obtained by the relative orientation processing, and the photographing distance H can be obtained by calculating the three-dimensional coordinate position of the corresponding feature point calculated by the three-dimensional measurement and the camera. It can be obtained from the position. The pixel resolution (planar resolution, depth resolution) is expressed by equations (8) and (9), which will be described later. To increase the measurement accuracy, that is, to reduce the resolution, the base length B is increased and the shooting distance H is set. It turns out that it is desirable to make it small. In the present embodiment, it is assumed that a stereo image is selected so that the baseline length B is equal to or greater than a predetermined threshold.

  FIG. 13 shows the concept of stereo matching of super-resolution images. A plurality of partial super-resolution images can be obtained by performing the super-resolution processing in the present embodiment on a plurality of image groups. By selecting a stereo image from the plurality of partial super-resolution images and performing template matching, the coordinate position of the corresponding point can be made with higher accuracy. Stereo pairs are sequentially set for a plurality of partial super-resolution images, and a matching process such as a least squares correlation (LSM) method is performed. For example, first, matching processing is performed on the image (a) and the image (b), then the image (b) and the image (c) (not shown) are performed, and the image (z) is sequentially performed. By performing the template matching using the partial image subjected to the super-resolution processing, high-precision coordinates can be obtained with high-speed processing and a reduced memory capacity. For example, it is possible to obtain subpixel accuracy. Since matching is performed at the time of super-resolution processing, it is expressed as rematching in FIG. Further, when selecting a plurality of pairs of stereo images, it is preferable to select the base lines so that the base line lengths are almost equal, because the accuracy can be made uniform.

[Orientation / 3D measurement]
The orientation process / three-dimensional measurement unit 7 performs relative orientation, displacement correction processing, and three-dimensional measurement. For each image selected as a stereo pair, orientation calculation processing is performed using the coordinates of the feature points and the corresponding points. The position and tilt of the photographed camera are obtained by the orientation calculation process. Then, the three-dimensional position of the corresponding point can be obtained. The orientation calculation processing is performed by mutual orientation for associating captured images selected as a stereo pair, and for orientation between a plurality of images or all images, connection orientation or bundle adjustment is performed. When this stereo pair selection is performed, the relationship between the stereo image, the base line length B of the camera, and the shooting distance H (when measured in advance) is stored in the stereo pair candidate list of the stereo image information memory 9B of the storage unit 9. In this case, the orientation processing / three-dimensional measurement unit 7 refers to the stereo image information memory 9B from a plurality of images obtained by the captured image acquisition unit 2, and sets of images estimated to have an appropriate baseline length. By selecting, appropriate orientation processing and three-dimensional measurement can be performed.

[Relative orientation processing]
Next, the orientation calculation will be described. By this calculation, the positions and inclinations of the left and right cameras are obtained.
FIG. 14 is a diagram for explaining relative orientation. The origin of the model coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. Scale takes baseline length (shown as B _x for X-direction) to unit length. The parameters to be obtained at this time are 5 of the left camera Z-axis rotation angle κ1, the Y-axis rotation angle φ1, the right-hand camera Z-axis rotation angle κ2, the Y-axis rotation angle φ2, and the X-axis rotation angle ω2. One rotation angle. In this case, the X-axis rotation angle ω1 of the left camera is 0, so there is no need to consider it.

First, parameters required to determine the positions of the left and right cameras are obtained by the following coplanar conditional expression (1). The screen distance C is equivalent to the focal distance f.

Under the above conditions, the coplanar conditional expression (1) is transformed as shown in Expression (2), and each parameter can be obtained by solving Expression (2).

Here, between the model coordinate system XYZ and the camera coordinate system xyz, the following coordinate transformation relational expressions (3) and (4) hold.

Using these equations, unknown parameters are obtained by the following procedure.
(I) The initial approximate value of the parameters (κ1, φ1, κ2, φ2, ω2) is normally 0.
(Ii) The coplanar conditional expression (2) is Taylor-expanded around the approximate value, and the value of the differential coefficient when linearized is obtained from the expressions (3) and (4), and an observation equation is established.
(Iii) Applying the least square method, a correction amount for the approximate value is obtained.
(Iv) The approximate value is corrected.
(V) The operations from (ii) to (iv) are repeated using the corrected approximate value until convergence.
If it converges, connection orientation is further performed. This is a process for unifying the inclination and scale between the models to make the same coordinate system.

When this processing is performed, a connection range expressed by the following equation is calculated.
ΔXj = (Xjr−Xjl) / (Z0−Zjl)
ΔYj = (Yjr−Yjl) / (Z0−Zjl)
ΔZj = (Zjr−Zjl) / (Z0−Zjl)
ΔDj = √ (ΔXj2 + ΔYj2)
(ΔXjlΔYjlΔZjl): the j-th left model of the unified coordinate system
(ΔXjrΔYjrΔZjr): If the j-th right model ΔZj and ΔDj in the unified coordinate system is 0.0005 (1/2000) or less, it is considered that the connection orientation is normally performed. If not successful, an error is output in the orientation result display to indicate which image is bad. In this case, if there is another orientation point on the image, it is changed and the above calculations from (ii) to (iv) are repeated. If not, change the location of the orientation point.

[Displacement correction processing]
Further, the deviation correction process corrects the image so that the epipolar line matches the left and right horizontal lines, and converts the image into an image for which the stereo method is established. Further, three-dimensional measurement is performed using image data obtained by the orientation process and the displacement correction process.

[Stereo method]
Next, the three-dimensional coordinates of each feature point (candidate corresponding point) are calculated. For example, three-dimensional coordinates are calculated from the stereo method.
FIG. 15 is a diagram for explaining the stereo method. For simplicity, two cameras C1 and C2 having the same specifications are used, the optical axes are parallel, the distance a from the principal point of the camera lens to the CCD surface is equal, and the CCD is placed perpendicular to the optical axis. It shall be. Let B be the distance (base line length) between the optical axes of the two cameras C1 and C2.
The following relationship exists between the coordinates of the points P1 (x1, y1) and P2 (x2, y2) on the object.

x1 = ax / z --- (5)
y1 = y2 = ay / z --- (6)
x2-x1 = aB / z --- (7)

However, the origin of the entire coordinate system (x, y, z) is assumed to be the lens principal point of the camera C1.

Z is obtained from the equation (7), and x and y are obtained from the equations (5) and (6) using this.

As can be seen from the description of the stereo method, if the shooting distance (magnification), shooting direction, and base line length B of the cameras C1 and C2 change, this principle becomes difficult to establish geometrically, resulting in stable accuracy. It becomes difficult to seek a solution.

In addition, the formula for the one-pixel resolution of the stereo method is shown below.
When measuring from two photographed images, a theoretical resolution of one pixel is usually obtained by the following formula.
Planar resolution: Δxy = H × Δp / f (8)
Depth resolution: Δz = H × H × Δp / (B × f) (9)
The error ranges (σ _xy , σ _z ) are represented by σ _xy = k1Δxy, σ _z = k2Δz.
Here, H: photographing distance, Δp: pixel resolution, f: focal length, B: baseline length, k1, k2: coefficients. k1 and k2 are typically set to 1, but are set to be smaller than 1 when it is desired to increase the accuracy, and are set to be larger than 1 when the accuracy is desired to be decreased, for example, 2 or 3.
From these calculation formulas, it is understood that the shooting distance H and the base line length B are important parameters for the resolution of one pixel.
In actual orientation and three-dimensional measurement, the parameters such as the shooting distance, the base line length, and the shooting direction are calculated with corrections and adjustments taking into consideration all of them, and can be calculated without any problem.
However, from the viewpoint of obtaining a stable and accurate solution, from the basic principle, if different stereo parameters having different parameters are used, the accuracy becomes unstable. Therefore, when selecting a stereo image, the shooting distance, the magnification, and the camera direction can be obtained from the shooting position of the camera, and those having substantially the same parameters are selected.

  When a stereo image is selected, a displacement correction process is performed (S155). Next, three-dimensional measurement of each corresponding feature point is performed using the displacement correction image subjected to the displacement correction processing, and three-dimensional coordinates are obtained (S160). When three-dimensional measurement of each corresponding feature point is performed on a plurality of stereo images having different shooting times, a time series change of each corresponding feature point can be obtained. Baseline length B and photographing distance H are obtained from the results of orientation and three-dimensional measurement.

[Another orientation process and 3D measurement]
Next, the process returns to the super-resolution processing step (S130) again, an original image group reflecting the three-dimensional measurement result is selected, stereo image selection is performed again (S140), and orientation is performed again (S150: second). Next, using the orientation result, a displacement correction process (S155: second displacement correction step) and another three-dimensional measurement are performed (S160: second three-dimensional measurement step). . In the second orientation step and the second three-dimensional measurement step, a stereo image that has been subjected to the super-resolution processing again is used, so that orientation with higher accuracy is performed and three-dimensional coordinates are calculated. By repeating this process (S130 to S160), the measurement accuracy can be improved. For example, this process is repeated until the target three-dimensional coordinate accuracy is obtained. Thereby, a three-dimensional coordinate is decided and a measurement is complete | finished.

  As described above, according to the present embodiment, an image processing apparatus and image processing that can accurately measure the shooting position, posture, or three-dimensional coordinates of an object of a shooting apparatus from a shot image that changes in time series such as a moving image. Can provide a method. In addition, it is possible to provide an image processing apparatus and an image processing method capable of performing super-resolution processing with high speed and small memory capacity.

[Second Embodiment]
In the first embodiment, an example of performing orientation and three-dimensional measurement after selecting a stereo image has been described. However, in this embodiment, the stereo image selection unit performs the baseline length and imaging distance obtained by orientation and three-dimensional measurement. An example of selecting a stereo image with reference to FIG. The base line length B of the camera is obtained from the two photographing positions obtained by the relative orientation processing, and the photographing distance H can be obtained from the three-dimensional coordinate position of the corresponding feature point calculated by the three-dimensional measurement and the camera position. . Therefore, if the relationship between the stereo image, the base length B of the camera, and the shooting distance H is stored in the stereo pair candidate list in the stereo image information memory 9B of the storage unit 9, the next time the stereo image is selected, the stereo image A stereo image can be selected with reference to the information memory 9B. For example, a stereo image can be selected so that the base line lengths are substantially equal to make the accuracy uniform, or the shooting distances H are substantially equal. Other points are the same as those of the first embodiment.

[Third Embodiment]
In the third embodiment, the storage unit 9 has a correspondence table that represents the relationship between the baseline length of the stereo image and the measurement accuracy of the three-dimensional coordinates, and the stereo image selection unit 6 satisfies the required accuracy from the correspondence table. An example of selecting a stereo image so as to have a baseline length is shown. Since the measurement accuracy depends on the baseline length, a three-dimensional measurement result that meets the requirements can be provided by determining the baseline length of the stereo image in accordance with the required accuracy. Other points are the same as those of the first embodiment.

[Fourth Embodiment]
In the first embodiment, an example in which the shooting position is measured from the corresponding point tracking process has been described. However, in the fourth embodiment, the shooting position measuring unit including the GPS is provided, and the shooting point is measured by other means. Is to measure.
Compared to the processing flow example of the first embodiment shown in FIG. 11, a photographing position measuring unit (not shown) is added, and the photographing position can be measured without being based on the photographed image. Accordingly, the shooting position can be measured without tracking the captured image or independently of the captured image, which is convenient for real-time processing. In addition, the measurement result of the shooting position by the tracking process of the shot image can be complemented. In addition, a photographing device (not shown) is added, and photographing on site is possible. Other steps are the same as those in the first embodiment.

  A GPS (Global Positioning System) sensor or the like can be employed as the photographing position measuring unit. For example, when the photographing position measuring unit is GPS, the three-dimensional coordinates of the photographing point are output from the GPS. The baseline length B is calculated using the GPS output data, and the shooting distance H is calculated from the three-dimensional coordinate value of the corresponding feature point calculated in the three-dimensional coordinate calculation step (S160) and the shooting position data output from the GPS. . These data are stored in the stereo pair candidate list in the stereo image information memory 9B of the storage unit 9, and can be referred to when selecting a stereo image.

[Fifth Embodiment]
The fifth embodiment is an example in which an inertial sensor is used for the imaging position measurement unit in the fourth embodiment. The baseline length can be obtained from the data obtained by tracking the photographing position with the inertial sensor. For example, a sensor capable of measuring position, velocity, acceleration, angle, or angular velocity can be used as the inertial sensor. It is also possible to improve the coordinate accuracy of the shooting position by comparing the value calculated from the sensor with the value calculated from tracking the shot image. Others are the same as in the fourth embodiment.

[Sixth Embodiment]
In the first embodiment, the example in which the three-dimensional measurement is performed after the super-resolution processing has been described. However, in the present embodiment, the position coordinates of the object measured in advance or the position coordinates of the photographing apparatus that has photographed the object are described. perform super-resolution processing using, the orientation process, three-dimensional measurement unit using the result of super-resolution processing, an example of performing orientation and three-dimensional measurement. By repeating the super-resolution processing, the three-dimensional coordinate accuracy can be further improved. Others are the same as in the first embodiment.

[Seventh Embodiment]
In the first embodiment, the example in which the template matching process is performed using the super-resolution image has been described. However, the template matching process may be further repeated. As a result, the coordinate accuracy of the corresponding feature points can be further increased. At this time, the base line length may be increased along with the number of repetitions to improve accuracy. It is also possible to omit template matching when selecting a stereo image. Still, since the tracking process is performed between the captured images in advance, a sufficiently high coordinate accuracy can be obtained. Others are the same as in the first embodiment.

[Eighth Embodiment]
In the first embodiment, an example in which the photographing apparatus moves while the photographing target is stationary is described. However, the eighth embodiment is an example in which the photographing target moves while the photographing apparatus is stationary. Even in this case, in addition to the original object, a moving object may be interrupted between the image capturing apparatus and the object, or the image capturing apparatus may be shaken. It is possible to obtain the three-dimensional coordinates of the feature points related to the object in the captured image that changes gradually. There is also a significance of improving the position coordinate accuracy by performing super-resolution processing. In addition, if the object itself rotates, the feature point repeatedly disappears and is restored, so that the present invention can be applied. In addition, a plurality of objects may move differently, and the present invention can be applied to each object even in such a case.

  The present invention can also be realized as a program for causing a computer to execute the image processing method described in the above embodiment. The program may be stored and used in a built-in memory of the control unit 1, may be stored in a storage device inside or outside the system, or may be downloaded from the Internet and used. Moreover, it is realizable also as a recording medium which recorded the said program.

  Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and it is obvious that various modifications can be made to the embodiment.

  For example, in the above embodiment, an example has been described in which a captured image is acquired in a state where one of the object or the imaging apparatus is moving and the other is stationary. However, the present invention is applied when both are moving. May be. For example, the present invention is sufficiently applicable when one moving speed and direction are constant. Further, although an example in which the MORAVEC operator is used for feature point extraction and SSDA template matching or LSM template matching is used for template matching has been described, other operators or template matching methods may be used. Further, it is possible to change the process of the super-resolution processing, such as eliminating the repeated loop from the super-resolution image synthesis (S139) to the original image group selection (S131) in FIG. The number of stereo images to be used, the base line length, the number of feature points, and the like can be selected as appropriate.

  The present invention is used for a photographing apparatus using a moving image or measurement of position coordinates of a photographing target.

It is a figure for demonstrating the concept of 1st Embodiment. It is a block diagram which shows the structural example of the image processing apparatus in 1st Embodiment. It is a figure which shows the example of a flow of the image processing method in 1st Embodiment. It is a figure which shows the example of a stereo image selection. It is a figure for demonstrating a deviation correction process. It is a figure which shows the example of a processing flow of feature point tracking. It is a figure which shows the example of a super-resolution image. It is a figure which shows the example of a flow of a super-resolution process process. It is a figure for demonstrating the example of a partial image setting. It is a figure for demonstrating the example of super-resolution partial image formation. It is a figure which shows the example of a flow according to frame of a super-resolution process process. It is a figure which shows the relationship between the three-dimensional coordinate position of a corresponding feature point, and a camera position. It is a figure which shows the concept of the stereo matching of a super-resolution image. It is a figure for demonstrating mutual orientation. It is a figure for demonstrating the stereo method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Control part 2 Captured image acquisition part 3 Feature extraction part 4 Feature point tracking part 5 Arithmetic processing part 6 Stereo image selection part 7 Orientation process and three-dimensional measurement part 8 Super-resolution processing part 9 Storage part 9A Corresponding point information memory 9B Stereo Image information memory 10 Image memory 11 Display unit 81 Original image group selection unit 82 Main feature point extraction unit 83 Partial image setting unit 84 Partial image enlargement unit 85 Sub feature point tracking unit 86 Partial image deformation unit 87 Super-resolution partial image formation unit 88 Partial image reduction unit 89 Super-resolution image composition unit 100 Image processing apparatus A Object B Baseline lengths C1 and C2 First and second imaging positions F0 Captured images f1 to f5 Partial images Fi, Fj i, jth frame H Shooting distance I Shooting image M Deviation corrected image P Measurement points P1 to P5 Main feature points

Claims (8)

A captured image acquisition unit that acquires a series of captured images obtained by capturing a relatively moving object in time series so that three or more adjacent images share an overlapping portion;
A feature extraction unit that extracts feature points from any of the captured images taken in time series;
A feature point tracking unit that tracks the feature points of the series of captured images and associates the feature points with each other;
From the series of captured images, an original image group that is an image group for performing super-resolution processing is selected, and super-resolution processing is performed from feature points that are correlated with each other in the feature point tracking unit in the original image group. Main feature points as feature points to be extracted are extracted, a partial image is set in a small area around the main feature point, and a super-resolution process is performed on the partial image to form a partial super-resolution image A super-resolution processing unit;
The stereo image selection section for selecting stereo images is an image of the pair from the partial super resolution photographic images on which the image is formed by super-resolution processing unit;
Using the mutually correlated feature points of the stereo image selected by the stereo image selection unit, orientation and three-dimensional measurement are performed, and the position of the object, the shape of the object, or the object is photographed. An orientation processing / three-dimensional measurement unit for determining the position of the photographed imaging device;
The super-resolution processing unit extracts a plurality of sub feature points around the main feature points of the partial image, tracks the sub feature points between the partial images of the captured image group, and determines the sub feature points. Have sub-feature point trackers that correspond to each other;
Image processing device. The super-resolution processing unit includes an original image group selecting unit that selects an original image group that is an image group for performing super-resolution processing from the series of captured images, and a feature point tracking unit in the original image group. A main feature point extracting unit that extracts a main feature point as a feature point for performing super-resolution processing from the feature points associated with the sub-feature, and a small area around the main feature point extracted by the main feature point extracting unit Positional coordinates of sub feature points associated with a partial image setting unit for setting a partial image in the region, a partial image enlarging unit for enlarging the partial image set by the partial image setting unit, and the sub feature point tracking unit A partial image deformation unit that deforms the partial image so that the original images closely match each other, and a super-resolution partial image formation that forms a super-resolution partial image from a plurality of partial images deformed by the partial image deformation unit And super-resolution formed by the super-resolution partial image forming unit. A partial image reduction unit that reduces the partial image to the size of the original partial image, and the partial image reduced by the partial image reduction unit is fitted into the position of the original partial image in the photographed image, and a partial super solution It possesses a super-resolution image synthesis unit for synthesizing an image picture;
The stereo image selection unit selects the stereo image so that a baseline length of the stereo image is longer than a predetermined threshold;
The image processing apparatus according to claim 1. The stereo image selection section selects a plurality of stereo images from photographed images in which the partial super resolution image as the base length is approximately equal is formed;
The image processing apparatus according to claim 2. The stereo image selection section, said selects the stereo images from photographed images partially super-resolution image by super-resolution processing section is formed, the orientation processing and the three-dimensional measurement unit, selected the stereo images for partial super resolution image performs the matching process;
The image processing apparatus according to any one of claims 1 to 3. The super-resolution processing unit performs super-resolution processing by using the position coordinate or the position coordinate of the imaging apparatus photographs the object of premeasured the object, the orientation process, the three-dimensional measuring unit the Perform orientation and 3D measurement using the result of super-resolution processing;
The image processing apparatus according to any one of claims 1 to 4. The stereo image selection unit has a correspondence table representing a relationship between the base line length of the stereo image and the measurement accuracy of the three-dimensional coordinates, and selects a stereo image from the correspondence table so as to have a base line length that satisfies the required accuracy. ;
The image processing apparatus according to any one of claims 1 to 5. A captured image acquisition step of acquiring a series of captured images obtained by capturing a relatively moving object in time series so that three or more adjacent images share an overlapping portion;
A feature extraction step of extracting feature points from any of the captured images taken in time series;
A feature point tracking step of tracking the feature points of the series of captured images and associating the feature points with each other;
From the series of captured images, an original image group that is an image group for performing super-resolution processing is selected, and super-resolution processing is performed from the feature points associated with each other in the feature point tracking step in the original image group. Main feature points as feature points to be extracted are extracted, a partial image is set in a small area around the main feature point, and a super-resolution process is performed on the partial image to form a partial super-resolution image A super-resolution processing step;
A stereo image selection step of selecting a stereo image that is a pair of images from the group of captured images in which a partial super resolution image is formed in the super resolution processing step;
Using the mutually corresponding feature points of the stereo image selected in the stereo image selection step, orientation and three-dimensional measurement are performed, and the position of the object, the shape of the object, or the object is photographed. An orientation process and a three-dimensional measurement process for determining the position of the photographed device;
The super-resolution processing step extracts a plurality of sub feature points around the main feature points of the partial image, tracks the sub feature points between the partial images of the captured image group, and determines the sub feature points. Having sub-feature point tracking steps that correspond to each other;
Image processing method. The super-resolution processing step includes an original image group selection step that selects an original image group that is an image group for performing super-resolution processing from the series of captured images, and a feature point tracking step in the original image group. A main feature point extracting step for extracting a main feature point as a feature point for performing super-resolution processing from the feature points associated with the main feature point, and a small area around the main feature point extracted in the main feature point extracting step. Partial image setting step for setting a partial image in a region, partial image enlargement step for enlarging the partial image set in the partial image setting step, and position coordinates of sub feature points associated in the sub feature point tracking step A partial image deformation step for deforming the partial image so that the original images match well, and a super-resolution image formation step for forming a super-resolution partial image from a plurality of partial images deformed in the partial image deformation step And the super-resolution image forming process A partial image reduction step of reducing the super-resolution partial image formed in step 1 to the size of the original partial image, and the partial image reduced in the partial image reduction step, the position of the original partial image in the captured image fitted to, possess a super-resolution image synthesis step of synthesizing a partial super resolution image;
In the stereo image selection step, the stereo image is selected so that a baseline length of the stereo image is longer than a predetermined threshold;
The image processing method according to claim 7.