JP6624880B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to a method for associating points between images of a plurality of viewpoints.

  2. Description of the Related Art Conventionally, in photogrammetry of a construction site or the like, triangulation using images from a plurality of viewpoints having different positions and / or directions of an imaging unit is known as a method of measuring the length of a subject from a photograph. In triangulation, imaging is performed using a plurality of cameras whose camera positions and camera directions (hereinafter, camera parameters) are known in advance. In the image displayed on the display, the operator uses a mouse or the like to specify two or more measurement points on the measurement subject. Then, the operator associates the measurement points while watching the image projected on the display. Alternatively, as in the method disclosed in Patent Literature 1, measurement points are automatically associated with each other based on camera parameters of each viewpoint and image data of each viewpoint. When the coordinates of the corresponding point corresponding to the measurement point are obtained, a predetermined calculation process is executed based on the coordinates of the measurement point and the coordinates of the corresponding point, and the length (physical distance) between the measurement points is calculated. You.

Japanese Patent No. 3898945

  However, in the method in which the worker associates the measurement points while viewing the image, the burden on the worker is large, and the accuracy of the association depends on the skill of each worker. On the other hand, in the method of Patent Document 1, the burden on the worker can be reduced by automatically performing the association of the measurement points, but there is a possibility that an error occurs in the association, and the presence or absence of the error is confirmed. In order to do so, it is necessary for an operator to check each image. Further, when an erroneous response has occurred, it is necessary for the operator to make corrections one by one.

  Accordingly, an object of the present invention is to provide a worker with a composite image obtained by shifting and superimposing another image on the basis of a designated point in one image among images of a plurality of viewpoints, so that the worker can measure the measurement point. An object of the present invention is to provide a method that allows the user to easily confirm the correspondence.

The image processing apparatus of the present invention, viewpoint using images of different multiple, an image processing apparatus for calculating a distance between at least two specified points, a plurality of at least one image of the images designating means for designating specified point, among the plurality of images, either the one image and the reference image, a reference image other than images corresponding to the designated point, respectively, in the reference image, the reference image Searching means for searching for a corresponding point, generating means for generating a composite image obtained by superimposing the reference image on the reference image based on a search result by the searching means, and presenting means for presenting the composite image to a user. It has a, said generating means, for the first designated point and the second designated point of the plurality of specified points, wherein said first specified point in the reference image as a reference A first composite image in which a reference image is superimposed and a second composite image in which the reference image is superimposed on the basis of the second designated point in the reference image; And the second composite image are presented in parallel .

  According to the present invention, it is possible for an operator to easily confirm correspondence of measurement points between images of a plurality of viewpoints.

FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus according to the first embodiment. FIG. 5 is a diagram illustrating a processing flow of an image processing unit according to the first embodiment. FIG. 9 is a diagram illustrating an example of a UI for designating a measurement point. FIG. 4 is a diagram for explaining a composite image generated from an image acquired by a multi-view camera. FIG. 7 is a diagram illustrating an example of a UI for confirming a correspondence between a measurement point and a corresponding point. FIG. 5 is a diagram for explaining a method of calculating a distance between measurement points. FIG. 11 is a diagram illustrating an example of a UI for displaying a calculation result of a distance between measurement points. FIG. 7 is a diagram for explaining an effect of the first embodiment. FIG. 9 is a diagram illustrating a functional configuration of an image processing apparatus according to a second embodiment. FIG. 9 is a diagram illustrating an example of a UI for designating a measurement point.

[Example 1]
FIG. 1A is a diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes an imaging unit 101, an image processing unit 100, and a display unit 108.

  The imaging unit 101 includes a zoom lens, a focus lens, a blur correction lens, an aperture, a shutter, an optical low-pass filter, an IR cut filter, a color filter, and an imaging element (sensor) such as a CMOS or a CCD, and captures an image of a subject. . Note that the imaging unit 101 in the present embodiment may be a multi-view camera capable of simultaneously acquiring a plurality of images from different viewpoints, or may be a plurality of cameras. Further, imaging may be performed a plurality of times using one camera. The present invention is not limited to the above-described camera mode as long as images at a plurality of viewpoints can be acquired by simultaneous or multiple imaging. In the following, a case where an image is acquired by a multi-view camera will be described as an example, and each camera constituting the multi-view camera is referred to as a small camera.

  The image processing unit 100 receives image data of a plurality of viewpoints from the imaging unit 101, and calculates a distance between two or more measurement points specified by the user based on the received image data. The image processing unit 100 according to the present embodiment includes an image receiving unit 102, a measurement point designating unit 103, a corresponding point searching unit 104, a composite image generating unit 105, a corresponding point calculating unit 106, and a distance calculating unit 107.

  The image receiving unit 102 receives image data of a plurality of viewpoints from the imaging unit 101.

  The measurement point designation unit 103 displays one image of the images of the plurality of viewpoints on the display unit 108, and acquires the coordinates of the measurement point (designated point) designated by the user using a mouse or the like.

  The corresponding point search unit 104 automatically obtains a corresponding point in the image of each viewpoint corresponding to the measurement point acquired by the measurement point designating unit 103 using a block matching method or the like.

  The composite image generation unit 105 generates a composite image based on the information on the measurement points and the information on the corresponding points, and displays the composite image on the display unit 108.

  The corresponding point calculation unit 106 calculates the corresponding point of the measurement point in the image of each viewpoint based on the image shift amount at the time of image synthesis controlled by the user, the camera parameters, and the coordinates of the measurement point designated by the measurement point designation unit 103. Calculate the coordinates. Here, the camera parameters (viewpoint parameters) include position information and orientation information of the camera at the time of imaging corresponding to images of a plurality of viewpoints. In the case of a multi-view camera, the orientations of the small cameras constituting the multi-view camera are substantially parallel, and thus the camera parameters are positional information of the small camera.

  The distance calculation unit 107 calculates the physical length between the measurement points specified by the user (the physical distance between the two specified points) based on the coordinates of the measurement points, the coordinates of the corresponding points, and the camera parameters. .

  The display unit 108 (presentation unit) uses, for example, a liquid crystal display and displays an image.

  FIG. 1B is a diagram illustrating the internal configuration of the image processing unit 100 in more detail.

  A central processing unit (CPU) 111 comprehensively controls each unit in the image processing unit 100. The RAM 112 functions as a main memory, a work area, and the like for the CPU 111. The ROM 113 stores a control program executed by the CPU 111 and the like. The bus 114 is a transfer path for various data. For example, image data converted into digital data by the A / D converter is sent to a predetermined processing unit via the bus 114.

  The display control unit 115 controls display of images and characters displayed on the display unit 108. The digital signal processing unit 116 performs various processing such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 114. The encoder unit 117 converts digital data into a file format such as JPEG or MPEG. The external memory control unit 118 is an interface for connecting to a PC or other media (for example, a hard disk, a memory card, a CF card, an SD card, or a USB memory). The input unit 119 is a serial bus interface, such as USB or IEEE1394, for connecting the input device 120 such as a keyboard or a mouse.

  Note that the image processing unit 100 may be configured inside the imaging unit 101.

  FIG. 2 is a diagram illustrating a processing flow of the image processing unit in the present embodiment.

  In step S201, the image receiving unit 102 receives multi-view image data from the imaging unit 101.

  In step S202, the user designates a measurement point on one image, and the measurement point designation unit 103 acquires the coordinates of the measurement point.

  FIG. 3 is a diagram illustrating an example of a UI of the measurement point designation unit 103. In FIG. 3, a display screen 300 is a screen in which a measurement point is designated by the measurement point designation unit 103 on the display unit 108. Here, one of the images of a plurality of viewpoints acquired by the multi-view camera is displayed on the display unit 108.

  When the user drags the input device 120 (for example, a mouse), the position of the mouse cursor 303 moves. When the user clicks on a point (measurement point) on the subject 301 to be measured, a marker 302 of the measurement point appears. The method of specifying the measurement point is not limited to this, and the measurement point may be specified by the arrow 304. When the measurement point is designated by the arrow 304, the mouse is clicked to select the first measurement point, and the mouse is dragged to draw the arrow 304. When the mouse is released at the second measurement point, the arrow 304 is displayed. Is determined.

  In step S203, the corresponding point search unit 104 searches for a corresponding point of the measurement point in the image of each viewpoint.

  In step S204, the composite image generation unit 105 calculates an image shift amount at the time of image composition based on the coordinates of the measurement point specified in step S202 and the coordinates of the corresponding point calculated in step S203. The images are shifted and combined according to the calculated image shift amounts.

  In the present embodiment, the composite image generation unit 105 generates an averaged image by shifting images of a plurality of viewpoints obtained by a multi-view camera in which small cameras are arranged on the same plane.

  FIG. 4A is a diagram illustrating an example of the arrangement of the multi-lens camera 400 according to the present embodiment. In FIG. 4A, a multi-lens camera 400 is configured such that five small and large cameras 401 are arranged in a square lattice in each of the vertical and horizontal directions. The number of the small camera 401 is (0, 0) with the small camera at the center, and (-2, -2), (-1, -2), ... (1, 2) from the upper left to the lower right. , (2,2). Here, the multi-lens cameras are arranged in a square lattice, but the arrangement is not limited to this.

  FIG. 4B is a diagram showing a composite image obtained by using a 2 × 2 multi-lens camera. In FIG. 4B, subjects 402, 403, and 404 are arranged in order of number from the front to the back with respect to the multi-lens camera. Since the distances from the multi-lens camera to the subjects are different, the respective subjects have different parallax amounts. For example, if the images are shifted and overlapped so that the subject 403 is in focus, the subjects 402 and 404 become multiple images.

  Hereinafter, a method of generating a composite image in the case of an N × N multiview camera will be described.

The number of the small camera 401 is (m, n), and the image acquired by the (m, n) th small camera 401 is _{Im, n} (x, y). Further, the coordinates of the marker 302 of the measuring point (0,0) th assume that designated on the acquired image with a small camera, the coordinates of the measurement point _{(x M, y M),} (m, n) th _Let the coordinates of the corresponding point of the measurement point in the image be ( _{xM_m, n} , _{yM_m, n} ). At this time, the composite image generation unit 105 generates a composite image I _syn (x, y) according to Expression (1).

  Equation (1) means that the measurement point and its corresponding point are aligned and the image is averaged. Here, the composite image is obtained by shifting the image of each viewpoint and performing averaging, but may be obtained by shifting the image of each viewpoint and performing weighted addition. Alternatively, the images of the respective viewpoints may be shifted and displayed so as to overlap in a see-through manner.

  Although the above description has been made taking the case of a multi-lens camera as an example, when imaging is performed using a plurality of cameras with known camera parameters, projective transformation is performed on the image of each viewpoint using camera orientation information. Then, the images are converted into a group of images captured in a state where the camera directions are parallel. Next, the magnification of the image is adjusted using the position information of the camera, and the image is converted into a group of images captured in a state where the camera is arranged on the same plane. That is, a so-called parallel stereo conversion process is performed to obtain a parallel stereo image so that the image is captured in a state where the camera direction is parallel and the camera position is in the same plane. The parameters required for the conversion can be obtained by a known calibration method. The calibration method is described, for example, in "Jun Sato, Computer Vision-Visual Geometry-, Corona". The process of synthesizing the converted image is the same as that of the multi-view camera.

  In the case where an image of a plurality of viewpoints is acquired by performing imaging with a single camera a plurality of times, camera parameters are estimated using the images of the plurality of viewpoints. By using the estimated camera parameters, the image is converted in the same manner as in the case of imaging with a plurality of cameras, and the image is synthesized.

  Note that the image processing unit 100 according to the present embodiment determines whether or not images of a plurality of viewpoints are parallel stereo images based on camera parameters, and determines whether to perform parallel stereo conversion based on the determination result. May be provided. That is, when the images of a plurality of viewpoints are parallel stereo images, parallel stereo conversion is not performed. For example, when the image is captured by a multi-lens camera, the acquired image is a parallel stereo image, so that the parallel stereo conversion is not performed. If the images of the multiple viewpoints are not parallel stereo images, any one of the images of the multiple viewpoints is used as a reference image, the other images are used as reference images, and the reference images are parallelized based on the camera parameters of the reference image. Convert to stereo.

  In step S205, the composite image generation unit 105 displays the composite image on the display unit 108. In the displayed composite image, points in the image of each viewpoint shown at the same position as the measurement point are the corresponding points searched in step S203.

  In step S206, the user pays attention to one measurement point and checks whether there is an error in the correspondence between the measurement point and the corresponding point. If there is no error, the process proceeds to step S210, and there is an error. In this case, the process proceeds to step S207.

  FIG. 5 is a diagram illustrating an example of a UI of the composite image generation unit 105. The display screen 500 in FIG. 5A displays the composite image generated by the composite image generation unit 105 on the display unit 108 when there is no mistake in searching for the corresponding point of the measurement point. The display screen 500 includes a window 501 displayed on the left half and a window 502 displayed on the right half. In addition, slide bars 503 and 504 for controlling an image shift amount at the time of image synthesis and markers 505 and 506 of measurement points are shown.

  In the window 501, the subject 301 is focused on the marker 505 at the measurement point, and in the window 502, the subject 301 is focused on the marker 506 at the measurement point. Since the synthesized image generation unit 105 synthesizes an image by aligning the measurement point and the corresponding point, the fact that the subject 301 is in focus at the position of the measurement point indicates that the corresponding point is correctly searched. Means. The user confirms that the subject 301 is in focus on the measurement point markers 505 and 506, and clicks the button 507. When the button 507 is clicked, the coordinates of the measurement point and the coordinates of the corresponding point corresponding to the measurement point are sent to the corresponding point calculation unit 106.

  In step S207, the user adjusts the slide bars 503 and 504 to control the amount of image shift during image synthesis. The composite image generation unit acquires a scalar value corresponding to the status of the slide bars 503 and 504, and calculates an image shift amount at the time of image composition based on the acquired scalar value and camera parameters.

  Here, the shift amount of each image can be controlled by a common scalar value. The cause is as follows. The image to be shifted is a parallel stereo image, and the shift amount has a ratio relationship between the respective images determined by the camera parameters. For example, taking as an example a multi-viewpoint image acquired by a multi-view camera configured such that small cameras are arranged at equal intervals, each image based on the image of the (0,0) th small camera Has the following relationship. That is, the shift amount of the image of the (m, n) th small camera is m times in the x direction and n times in the y direction, compared to the shift amount of the image of the (1,1) th small camera. I have. Such a relationship is calculated from the camera parameters by a composite image generation unit or the like. Therefore, the user can optimally control the respective image shift amounts by adjusting the common scalar value to the optimal value.

  In step S208, the composite image generation unit 105 generates a composite image by shifting and averaging the images based on the image shift amount controlled by the user.

  In step S209, the composite image generation unit 105 displays the composite image on the display unit 108. Here, when displaying on the display unit 108, the screen is divided into left and right like windows 501 and 502, but may be switched and displayed one by one.

  The display screen 510 in FIG. 5B displays the image generated by the composite image generation unit 105 on the display unit 108 when there is a search error at the corresponding point of the measurement point. The display screen 510 includes a window 511 displayed on the left half and a window 512 displayed on the right half.

  In the window 512, the subject 301 is out of focus on the marker 506 of the measurement point. This means that the corresponding point search unit 104 did not correctly search for the corresponding point of the measurement point. In this case, the user adjusts the slide bar 504 in the window 512 so that the subject 301 is focused on the marker at the measurement point (that is, the state shown in the window 502).

  In step S210, the corresponding point calculation unit 106 calculates the coordinates of the corresponding point corresponding to the measurement point based on the image shift amount, the coordinates of the measurement point, and the camera parameters.

In this embodiment, the coordinates of the corresponding point corresponding to the measurement point are calculated based on the image shift amount controlled by the user with the slide bars 503 and 504, the coordinates of the measurement point designated by the measurement point designation unit 103, and the camera parameters. In the case of an image of a plurality of viewpoints acquired by a multi-view camera configured such that small cameras are arranged at equal intervals, the corresponding point calculation unit 106 calculates the coordinates of the corresponding point according to Expression (2). Controlled by a slide bar 503 and 504 (1, 1) th shift amount of the miniature camera images the s [pixel], the coordinates of the measurement points when the _{(x M, y M) [} pixel], (m, The coordinates of the corresponding point of the measurement point in the image of the (n) th small camera are represented by Expression (2).

  If the small cameras are not arranged at regular intervals, the coordinates of the corresponding points can be calculated by a similar method by setting a numerical value that reflects the distance between the small cameras instead of m and n in Equation (2). Can be.

  In the case where a plurality of cameras are used or the case where a plurality of images are taken, first, the coordinates of the corresponding points in the image subjected to the parallel / stereo conversion by a similar method are calculated. Then, the coordinates of the corresponding point in the image before conversion are calculated from the calculated coordinates of the corresponding point and the camera parameters. That is, the coordinates of the corresponding points in the image before the conversion are calculated by performing the inverse conversion before the parallel stereo conversion.

  In step S211, it is determined whether the user has finished checking the corresponding points for all the measurement points. If there is a measurement point whose corresponding point has not been confirmed, the process returns to step S206. If the confirmation of the corresponding points has been completed for all the measurement points, when the user clicks the button 507 in FIG. 5, the process proceeds to step S212.

  In step S212, the distance calculation unit 107 calculates the distance between the measurement points based on the coordinates of the measurement points and the corresponding points corresponding thereto, displays the distance on the display unit 108, and ends the operation of the image processing unit 100.

  Hereinafter, the operation of the distance calculation unit 107 will be described in more detail. The distance calculation unit 107 calculates the distance between the measurement points basically using a triangulation method.

FIG. 6 is a diagram for explaining a method of calculating the distance between measurement points. FIG. 6 shows world coordinates 601, camera coordinates 602, a subject 603, and a subject image 605. The image plane 604 is a plane corresponding to the image sensor when the small camera 401 is regarded as a pinhole camera. The origin of the camera coordinates 602 is the center position of the small camera 401, and the subject image 605 is the intersection of a straight line connecting the subject 603 and the center position of the small camera and the image plane 604. It is assumed that the position vector of the subject 603 is X _obj , the unit vector extending the camera coordinates 602 is i, j, k, the unit vector extending the world coordinates is X, Y, Z, and the vector representing the center position of the camera is _Xc . Also, the position vector of the subject image on the image plane 604 viewed from the center position of the small camera 401 is X _screen , the position of the subject image 605 on the image plane 604 is (u, v), and the focal length of the small camera is f. I do. (u, v) corresponds to the position of the subject on the image.

  From FIG. 6, equations (3) and (4) hold.

Here, s is an appropriate scalar value and represents scale conversion. _Assuming that the vector X _obj is expressed by Expression (5) and the vector X _c is expressed by Expression (6), Expression (3) is substituted into Expression (4), and the inner product of the vectors X, Y, and Z is obtained. (7).

here,

And R represents a rotation matrix of the camera direction. Equation (7) coordinates on the image of the subject (u, v), the center position of the small camera _{(x c, y c, z} c), represents the relationship of the object coordinates (x, y, z). If a suffix (m, n) indicating the position number of the small camera is added to equation (7), equation (9) holds for each camera.

In equation (9), (x, y, z) is a common value irrespective of (m, n), and therefore _{sm, n} is determined here to minimize the following function.

here,

And

When V is partially differentiated with respect to s _{m, n} and set to 0, Equation (12) is obtained.

Where N ² is the number of small cameras,

Equation (12) can be transformed to Equation (14).

Since M and v can be obtained by knowing the position of the subject on the image, the rotation matrix (direction) of the camera, and the position of the camera, solving equation (14) for sm and _n gives the scale in each viewpoint image. A conversion value can be obtained.

  Finally, by using the scale conversion value s of a certain viewpoint image and Expression (7), the distance between the measurement points is calculated.

You can ask. Here, s ₁ and s ₂ are scale conversion values of the measurement points 1 and 2, and X _{obj, 1} and X _{obj, 2} are spatial coordinates of the measurement points 1 _{and 2} . The distance calculation unit 107 calculates the distance between two measurement points on the image according to Expressions (14) and (15).

  The above is the operation principle of the distance calculation unit 107. FIG. 7 is a diagram illustrating an image in which the distance calculated by the distance calculation unit 107 is displayed on the display unit 108. In FIG. 7, a display screen 700 displays an image of the (0,0) th small camera, and shows an arrow 703 connecting measurement points 701 and 702 specified by the user, and a distance between the measurement points 701 and 702. Window 704 is displayed. By looking at the display screen 700, the user can know the distance between the designated measurement points.

  FIG. 8 is a diagram for explaining an effect when the image processing according to the present embodiment is performed.

  FIG. 8A shows a screen for confirming whether there is an error in the correspondence between the measurement points and the corresponding points by the conventional method. The display screen 800 displays an image of nine viewpoints of 3 × 3, and the user sees each of the nine divided screens and checks whether there is an error in the correspondence. In the case of FIG. 8A, since the corresponding point has an erroneous correspondence in the lower right image, the user corrects the coordinates of the corresponding point on the lower right screen.

  FIG. 8B illustrates a screen for confirming whether or not there is an error in the correspondence between the measurement points and the corresponding points on the composite image by applying the present embodiment. The display screen 810 is obtained by synthesizing images of 3 × 3 nine viewpoints so that the measurement points and the corresponding points are aligned. The user looks at the two images on the left and right, and checks the respective measurement points and the corresponding points. Check if there is an error in the response. In the window 801, the corresponding point of the measurement point 803 is correctly obtained, but in the window 802, since the subject 301 is not focused on the measurement point 804, an erroneous correspondence has occurred in the corresponding point. The user adjusts the slide bar 504 in the window 802 so that the subject 301 is focused on the measurement point 804.

  In the case of the conventional method, each viewpoint image becomes small, and it is difficult to confirm whether there is an erroneous correspondence. In addition, a method of switching and displaying each image one by one to check it is conceivable, but it is necessary to check images for the number of viewpoints. Further, when erroneous correspondences occur at a plurality of viewpoints, the user needs to correct the corresponding points one by one.

  On the other hand, in the case of this embodiment, it is possible to check at a glance whether there is an erroneous response on a large screen as shown by the display screen 810. Further, in the present embodiment, when the images of the plurality of viewpoints are not parallel stereo images, the images of the plurality of viewpoints are subjected to parallel stereo conversion, and a composite image is generated based on the converted images. The composite image generated in this manner makes it possible to easily and accurately confirm the presence or absence of an incorrect correspondence. Also, even when an erroneous correspondence has occurred, it is not necessary to correct one by one as in the conventional method, and the erroneous correspondence can be corrected by adjusting a common scalar value on the composite image. .

  As described above, according to the present embodiment, it is possible to determine at a glance whether there is an erroneous correspondence between the corresponding points of the measurement points, and even if there is an erroneous correspondence, the erroneous correspondence can be corrected by a simple operation. Will be possible.

[Example 2]
In the first embodiment, the user specifies the markers 505 and 506 of the measurement points on the display screen 500. In the present embodiment, the case where the vicinity of the specified area of the marker 506 of the measurement points is enlarged and displayed on the display unit 108 is displayed. Will be described.

  FIG. 9 is a diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment.

  The image processing unit 900 receives image data of a plurality of viewpoints from the imaging unit 101, and calculates a distance between two or more measurement points specified by the user based on the received image data. Here, only the differences from FIG. 1 will be described.

  The area enlargement unit 901 acquires the coordinates of the mouse pointer, enlarges the area around the mouse pointer, and displays the enlarged area on the display unit 108. In the present embodiment, the method of enlarging the peripheral area is based on interpolation such as the bicubic method. However, the present invention is not limited to this, and another method such as super-resolution may be used as long as the image can be enlarged.

  The measurement point designation unit 902 receives the information of the area enlargement unit 901 and enlarges the area around the mouse pointer to display the enlarged area on the display unit 108.

  FIG. 10 is a diagram illustrating an example of a UI of the measurement point designation unit 902. The UI of the measurement point designation unit 902 is basically the same as the display screen 500, and the display screen 1000 is obtained by cutting out the right half window.

  The window 1001 is obtained by enlarging an area around the mouse coordinates in the composite image by the area enlarging unit 901 and displaying the measurement point specifying unit 902 on the display unit 108.

  As described above, according to the present embodiment, by enlarging and displaying the vicinity of the area where the measurement point is to be specified, the user can specify the measurement point with higher accuracy and check in more detail whether there is an erroneous response. Will be possible.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Claims (14)

Viewpoint using images of different multiple, an image processing apparatus for calculating a distance between at least two specific points,
Designation means for designating a plurality of designated points in at least one of the plurality of images;
Among the plurality of images, either the one image and the reference image, a reference image other than images corresponding to the designated point, respectively, in the reference image, a search means for searching the corresponding point in the reference image ,
Generating means for generating a composite image in which the reference image is superimposed on the reference image based on a search result by the search means;
Possess a presenting means for presenting the combined image to the user,
A generating unit configured to superimpose the reference image on the first specified point and the second specified point of the plurality of specified points based on the first specified point in the reference image; And generating a second composite image in which the reference images are superimposed on the basis of the second designated point in the reference image,
The image processing apparatus according to claim 1, wherein the presenting unit presents the first combined image and the second combined image in parallel . The search means searches for a corresponding point in the reference image corresponding to the designated point in the reference image by a block matching method,
The generation unit generates the composite image by shifting the reference image so that the searched corresponding point is aligned with the designated point, and weighting and adding the shifted reference image and the reference image. The image processing apparatus according to claim 1, wherein: Based on the viewpoint parameters corresponding to each of the plurality of images, it is determined whether the plurality of images are parallel stereo images, and when it is determined that the plurality of images are not parallel stereo images, Further comprising a conversion means for performing parallel stereo conversion of the reference image based on viewpoint parameters,
The image according to claim 1, wherein when the reference image is subjected to the parallel / stereo conversion by the conversion unit, the generation unit generates the composite image using the converted reference image. Processing equipment. A receiving unit that receives a common scalar value that controls a shift amount of each of the reference images, which is input by the user to adjust a shift amount of the reference image in the composite image,
In response to the receiving unit receiving the scalar value, the generating unit calculates and calculates the shift amount of each of the reference images based on the scalar value and the viewpoint parameters of the reference image and the reference image. By weighting and adding the reference image and the reference image shifted according to the shift amount, an adjusted composite image is generated,
The image processing apparatus according to claim 3, wherein the presenting unit presents the adjusted composite image to the user. The presenting means presents a slide bar for adjusting the scalar value to the user together with the adjusted composite image,
The image processing apparatus according to claim 4, wherein the receiving unit acquires the scalar value according to a status of the slide bar adjusted by the user.   The shift amount of the reference image controlled by the scalar value, the viewpoint parameters of the reference image and the reference image, and the reference image corresponding to the designated point based on the coordinates of the designated point in the reference image. The image processing apparatus according to claim 4, further comprising a corresponding point calculation unit configured to calculate coordinates of the corresponding point in the image processing apparatus.   The coordinates of the corresponding points in the reference image corresponding to the two designated points in the reference image are calculated by the corresponding point calculating means, and the calculated coordinates of the corresponding points, the coordinates of the two specified points, and the reference image The image processing apparatus according to claim 6, further comprising a distance calculation unit configured to calculate a physical distance between the two designated points based on a viewpoint parameter of the reference image. 8. The apparatus according to claim 1, wherein the designation unit receives a designated point in the reference image from a user, and enlarges and displays a peripheral area of the coordinate based on the coordinate input by the user. Item 2. The image processing device according to item 1. Among images of a plurality of viewpoints having different viewpoint parameters including the position and orientation of the imaging unit, one of the images is set as a reference image, the other images are set as reference images, and the reference image is set based on a designated point in the reference image. Generating means for generating a composite image superimposed on the reference image by shifting
  Presentation means for presenting the composite image to a user;
  Based on the viewpoint parameters of the images of the multiple viewpoints, it is determined whether or not the images of the multiple viewpoints are parallel stereo images, and when it is determined that the images of the multiple viewpoints are not parallel stereo images, Conversion means for performing parallel stereo conversion of the reference image based on viewpoint parameters,
  Receiving means for receiving a common scalar value controlling the shift amount of each of the reference images, input by the user to adjust the shift amount of the reference image in the composite image,
  The shift amount of the reference image controlled by the scalar value, the viewpoint parameters of the reference image and the reference image, and the reference image corresponding to the designated point based on the coordinates of the designated point in the reference image. Corresponding point calculating means for calculating coordinates of corresponding points in
Has,
  When the reference image is subjected to the parallel-stereo conversion by the conversion unit, the generation unit generates the composite image using the converted reference image,
  In response to the receiving unit receiving the scalar value, the generating unit calculates and calculates the shift amount of each of the reference images based on the scalar value and the viewpoint parameters of the reference image and the reference image. By weighting and adding the reference image and the reference image shifted according to the shift amount, an adjusted composite image is generated,
  The presenting means presents the adjusted composite image to the user
An image processing apparatus comprising: Among images of a plurality of viewpoints having different viewpoint parameters including the position and orientation of the imaging unit, one of the images is set as a reference image, the other images are set as reference images, and the reference image is set based on a designated point in the reference image. Generating means for generating a composite image superimposed on the reference image by shifting
  Presentation means for presenting the composite image to a user;
  Based on the viewpoint parameters of the images of the multiple viewpoints, it is determined whether or not the images of the multiple viewpoints are parallel stereo images, and when it is determined that the images of the multiple viewpoints are not parallel stereo images, Conversion means for performing parallel stereo conversion of the reference image based on viewpoint parameters,
  Receiving means for receiving a common scalar value that controls the shift amount of each of the reference images, input by the user to adjust the shift amount of the reference image in the composite image,
  The shift amount of the reference image controlled by the scalar value, the viewpoint parameters of the reference image and the reference image, and the reference image corresponding to the designated point based on the coordinates of the designated point in the reference image. Corresponding point calculating means for calculating coordinates of corresponding points in
Has,
  When the reference image is subjected to the parallel-stereo conversion by the conversion unit, the generation unit generates the composite image using the converted reference image,
  In response to the receiving unit receiving the scalar value, the generating unit calculates and calculates the shift amount of each of the reference images based on the scalar value and the viewpoint parameters of the reference image and the reference image. By weighting and adding the reference image and the reference image shifted according to the shift amount, an adjusted composite image is generated,
  The presenting means presents to the user both the adjusted composite image and a slide bar for adjusting the scalar value,
  The receiving means obtains the scalar value according to the status of the slide bar adjusted by the user.
An image processing apparatus comprising: Viewpoint using images of different multiple, an image processing method for calculating a distance between at least two specific points,
A designation step of designating a plurality of designated points in at least one of the plurality of images;
Among the plurality of images, either the one image and the reference image, a search step of the reference image to the other image, corresponding to the designated point, respectively, in the reference image to search the corresponding point in the reference image ,
A generation step of generating a composite image superimposed on the reference image based on a search result obtained by the search step;
Including a presentation step of presenting the composite image to a user,
The generating step includes, for a first designated point and a second designated point of the plurality of designated points, a first composite image in which the reference image is superimposed on the basis of the first designated point in the reference image. And generating a second composite image in which the reference images are superimposed on the basis of the second designated point in the reference image,
The image processing method , wherein the presenting step presents the first combined image and the second combined image in parallel . Among images of a plurality of viewpoints having different viewpoint parameters including the position and orientation of the imaging unit, one of the images is set as a reference image, the other images are set as reference images, and the reference image is set based on a designated point in the reference image. Generating a composite image superimposed on the reference image by shifting
  A presentation step of presenting the composite image to a user;
  Based on the viewpoint parameters of the images of the multiple viewpoints, it is determined whether or not the images of the multiple viewpoints are parallel stereo images, and when it is determined that the images of the multiple viewpoints are not parallel stereo images, A conversion step of performing parallel-stereo conversion of the reference image based on viewpoint parameters,
  A receiving step of receiving a common scalar value that controls the shift amount of each of the reference images, input by the user to adjust the shift amount of the reference image in the composite image;
  The shift amount of the reference image controlled by the scalar value, the viewpoint parameters of the reference image and the reference image, and the reference image corresponding to the designated point based on the coordinates of the designated point in the reference image. A corresponding point calculation step of calculating the coordinates of the corresponding point in
Has,
  When the reference image is subjected to the parallel-stereo conversion by the conversion step, the generation step generates the composite image using the converted reference image,
  In response to the receiving step receiving the scalar value, the generating step calculates and calculates the shift amount of each of the reference images based on the scalar value and the viewpoint parameters of the reference image and the reference image. By weighting and adding the reference image and the reference image shifted according to the shift amount, an adjusted composite image is generated,
  The presenting step presents the adjusted composite image to the user
An image processing method comprising: Among images of a plurality of viewpoints having different viewpoint parameters including the position and orientation of the imaging unit, one of the images is set as a reference image, the other images are set as reference images, and the reference image is set based on a designated point in the reference image. Generating a composite image superimposed on the reference image by shifting
  A presentation step of presenting the composite image to a user;
  Based on the viewpoint parameters of the images of the multiple viewpoints, it is determined whether or not the images of the multiple viewpoints are parallel stereo images, and when it is determined that the images of the multiple viewpoints are not parallel stereo images, A conversion step of performing parallel-stereo conversion of the reference image based on viewpoint parameters,
  A receiving step of receiving a common scalar value that controls the shift amount of each of the reference images, input by the user to adjust the shift amount of the reference image in the composite image;
  The shift amount of the reference image controlled by the scalar value, the viewpoint parameters of the reference image and the reference image, and the reference image corresponding to the designated point based on the coordinates of the designated point in the reference image. A corresponding point calculation step of calculating the coordinates of the corresponding point in
Has,
  When the reference image is subjected to the parallel-stereo conversion by the conversion step, the generation step generates the composite image using the converted reference image,
  In response to the receiving step receiving the scalar value, the generating step calculates and calculates the shift amount of each of the reference images based on the scalar value and the viewpoint parameters of the reference image and the reference image. By weighting and adding the reference image and the reference image shifted according to the shift amount, an adjusted composite image is generated,
  The presenting step presents both the adjusted composite image and the slide bar for adjusting the scalar value to the user,
  The receiving step obtains the scalar value according to the status of the slide bar adjusted by the user.
An image processing method comprising: A program for causing a computer to function as the image processing device according to any one of claims 1 to 10 .