JP2008109481A - Image generating apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain an image generating apparatus capable of generating a clear image having presence because sizes, positions and the like of objects coincide without executing highly accurate camera calibration. <P>SOLUTION: The objects of multiview targets and a floor position of a vertical reference bar are used to arrange the positions of the objects of the multiview targets in an original image n (1≤n≤8), an image scale factor for making the sizes of the objects of the multiview targets the same is calculated for each original image n, and the original image n is converted according to the image scale factor. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image generation apparatus that generates images useful for applications such as sports appreciation and monitoring from multi-viewpoint images captured by a plurality of cameras or the like.

In a conventional image generation device, when generating a multi-viewpoint image centered on a gazing point designated by a user, camera parameters of each camera (world coordinates in a three-dimensional space) are obtained for images taken by a plurality of cameras. Are used to convert the world coordinates of the subject photographed by the camera into the video coordinates of each camera (see, for example, Patent Document 1).
Therefore, in order for the image generation apparatus to generate a multi-viewpoint image, it is necessary to perform high-precision camera calibration on each camera in advance in order to obtain camera parameters of each camera.

JP 2006-115298 A (paragraph numbers [0015] to [0017], FIG. 1)

  Since the conventional image generating apparatus is configured as described above, if camera parameters for converting world coordinates to video coordinates are obtained in advance, a multi-viewpoint centered on a gaze point designated by the user An image can be generated. However, in order to obtain camera parameters, there has been a problem that high-precision camera calibration has to be performed for each camera.

  The present invention has been made to solve the above-described problems, and generates an easy-to-see and realistic image in which the size and position of the subject match without performing high-accuracy camera calibration. An object of the present invention is to obtain an image generation apparatus capable of performing the above.

  The image generating apparatus according to the present invention includes a multi-view target object existing in a plurality of images from the position of the multi-view target object specified by the position specifying unit and the plane projective transformation matrix calculated by the plane projective transformation matrix calculating unit. Position calculating means for calculating the position of the vertical reference bar present in a plurality of images from the floor position of the vertical reference bar specified by the position specifying means and the plane projective transformation matrix, and position calculation Using the position of the object of the multi-view target calculated by the means and the floor position of the vertical reference bar, the positions of the objects of the multi-view target in the plurality of images are aligned, and the size of the object of the multi-view target is the same Image magnification calculation means for calculating the image magnification for each of a plurality of images, and the image conversion means according to the image magnification calculated by the image magnification calculation means. It is obtained so as to convert the plurality of images.

  According to the present invention, the position of the multi-view target object existing in the plurality of images is determined from the position of the multi-view target object specified by the position specifying means and the plane projection transformation matrix calculated by the plane projection transformation matrix calculation means. The position calculating means calculates the floor position of the vertical reference bar existing in a plurality of images from the floor position of the vertical reference bar specified by the position specifying means and the plane projective transformation matrix, and the position calculating means An image in which the positions of the multi-view target objects in the plurality of images are aligned and the sizes of the multi-view target objects are the same using the position of the object of the multi-view target and the floor position of the vertical reference bar. An image magnification calculating means for calculating the magnification for each of the plurality of images, and the image converting means calculates the plurality of images according to the image magnification calculated by the image magnification calculating means; And then, is converted, without performing a highly accurate camera calibration, there is an effect that can produce an image with easy to see realistic that the size and position of the object match.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an image generation apparatus according to Embodiment 1 of the present invention.
In the case of the image generating apparatus of FIG. 1, for a plurality of images obtained by photographing N-dimensional images from different directions using N cameras, an object existing in the region is all images. Each image is enlarged or reduced so as to have the same size, moved so as to be positioned at the same coordinates in the image, and further, distortion in the horizontal direction or the vertical direction is corrected.
In addition, after generating the above images using N images taken by N cameras, when these images are displayed in the order of camera arrangement, an equidistant viewpoint centered on an object It is possible to obtain an image effect such that the images are sequentially moved along the camera. Hereinafter, such an effect is referred to as a multi-view effect or simply a multi-view.
In the following description, it is assumed that an area in which several objects are placed on the floor surface, which is a horizontal plane, is captured by N cameras, and an image having a multi-view effect is generated for the area. To do.

In the figure, the cameras 1-1 to 1-N photograph the same three-dimensional area from different directions, and output an original image n (1 ≦ n ≦ N) that is a photographing result of the three-dimensional area.
The image data temporary storage unit 2 is a memory for temporarily storing an original image n (1 ≦ n ≦ N) output from the camera 1-n (1 ≦ n ≦ N). The image data temporary storage unit 2 constitutes image storage means.
In the first embodiment, however, it is assumed that N = 8 for the sake of simplicity.

The multi-view position designating unit 3 accepts the selection of an arbitrary original image n ₁ (1 ≦ n ₁ ≦ 8) from the eight original images n stored in the image data temporary storage unit 2, and the original image n _A process of accepting designation of image coordinates indicating the position on the floor surface of the multi-view target object (object that the user desires for multi-view) existing in ₁ is performed.
The image coordinates are read by the multi-view position designating unit 3 while the user views the original image n ₁ and, for example, operates the mouse to designate the position in the image, thereby reading the image coordinates at that position. Alternatively, it may be automatically specified by some method.

The reference position specifying unit 4 accepts the selection of an arbitrary original image n ₂ (1 ≦ n ₂ ≦ 8) from the eight original images n stored in the image data temporary storage unit 2, and the original image n _2. The process of accepting designation of image coordinates indicating the position on the floor surface where the vertical reference bar present in FIG.
The reading of the image coordinates in the reference position specifying unit 4 may be performed by reading the image coordinates of the position by, for example, operating the mouse and specifying the position in the image while the user views the original image n _2. Alternatively, it may be automatically specified by some method.
The multi-view position specifying unit 3 and the reference position specifying unit 4 constitute position specifying means.

The floor surface coordinate reading unit 5 is the vertex (1) of the marking (see FIG. 2) that exists in common on the floor surface of the original image n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2. ), Vertex (2), vertex (3) and vertex (4) are read. The floor coordinate reading unit 5 constitutes a marking position acquisition unit.
For each original image n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2, the horizontal vanishing point calculation unit 6 uses the horizontal vanishing point of the original image n (the optical axis of the camera 1-n as the floor surface). A process of calculating the image coordinates of a vanishing point obtained by projecting a straight infinity point on the floor surface projected in the vertical direction onto the captured image is performed.
For each original image n (1 ≦ n ≦ 8) stored in the image data temporary storage unit 2, the vertical vanishing point calculation unit 7 calculates a vertical vanishing point of the original image n (a straight line infinity point perpendicular to the floor surface). The process of calculating the image coordinates of the vanishing points projected on the image is performed.
The vanishing point calculating means is composed of the horizontal vanishing point calculating unit 6 and the vertical vanishing point calculating unit 7.

The planar projective transformation matrix calculation unit 8 images the marking vertex (1), vertex (2), vertex (3), and vertex (4) in the original image n ₁ and other original images n (excluding the original image n ₁ ). Using the coordinates, a plane projective transformation matrix H _{n1-1 to} H _n1-8 (1 ≦ 1) of the plane constituting the floor surface between the original image n ₁ and another original image n (excluding the original image n ₁ ). The process of calculating 7 pieces of n ₁ ≦ 8 is performed.
The plane projective transformation matrix calculation unit 8 also performs marking vertex (1), vertex (2), vertex (3), vertex (4) in the original image n ₂ and other original images n (excluding the original image n ₂ ). The plane projection transformation matrices H _{n2-1 to} H _n2-8 of the plane constituting the floor surface between the original image n ₂ and another original image n (excluding the original image n ₂ ) A process of calculating 7 of 1 ≦ n ₂ ≦ 8 is performed.
The calculation method of the planar projective transformation matrix is described in, for example, a non-patent document “Koichiro Deguchi“ Basics of Robot Vision ”p47 (described as a homography matrix in this document)”.
The plane projection transformation matrix calculation unit 8 constitutes plane projection transformation matrix calculation means.

Multi-view position calculating unit 9 using the homography matrix _{H n1-1} ~H n1-8 calculated by planar projective transformation matrix calculation unit 8, present in the original image n (excluding original image n ₁₎ Multi A process of calculating image coordinates indicating the position of the object to be viewed on the floor surface is performed. That is, using the plane projection transformation matrices H _{n1-1 to} H _n1-8 , the image coordinates indicating the position on the floor surface of the object of the multi-view target accepted by the multi-view position designating unit 3 (or the multi-view By converting the image coordinates converted by the view image coordinate conversion unit 22 into the image coordinates of the original image n (excluding the original image n ₁ ), the multi-image existing in the original image n (excluding the original image n ₁ ) is converted. A process of calculating image coordinates indicating the position of the object to be viewed on the floor surface is performed.

The reference position calculation unit 10 uses the plane projection transformation matrices H _{n2-1 to} H _n2-8 calculated by the plane projection transformation matrix calculation unit 8 to use the vertical reference existing in the original image n (excluding the original image n ₂ ). A process of calculating image coordinates indicating the position of the bar on the floor is performed. That is, using the plane projection transformation matrices H _{n2-1 to} H _n2-8 , the image coordinates indicating the floor position of the vertical reference bar in the original image n ₂ that has been designated by the reference position designating unit 4 are converted into the original coordinates. by carrying out the processing of converting the image coordinates of the image n (excluding the original image n _2), image coordinates indicating the position on the floor surface of the vertical reference rod present in the original image n (excluding the original image n ₂₎ The process of calculating is performed.
The multi-view position calculation unit 9 and the reference position calculation unit 10 constitute position calculation means.

  The in-image magnification calculation unit 11 is present in the original image n calculated by the multi-view position calculation unit 9 and in the original image n calculated by the reference position calculation unit 10. The object of the multi-view target is the same as the vertical reference bar using the image coordinates indicating the floor position of the vertical reference bar and the horizontal vanishing point image coordinates of the original image n calculated by the horizontal vanishing point calculation unit 6. A process of calculating an in-image magnification (change magnification of how the size of the object of the multi-view target is viewed) when the object is moved to the floor position of the depth is performed.

The camera horizontal axis distortion correction parameter calculation unit 12 calculates the horizontal axis distortion caused by the tilt of the camera 1-n when shooting the original image n from the vertical vanishing point of the original image n calculated by the vertical vanishing point calculation unit 7. A process for calculating a correction parameter is performed.
The perspective projection distortion correction parameter calculation unit 13 is based on perspective projection of a three-dimensional space onto a two-dimensional plane from the vertical vanishing point of the original image n calculated by the vertical vanishing point calculation unit 7 when the camera 1-n is photographed. Processing for calculating a correction parameter for perspective projection distortion of an image is performed.

The primary correction image generation unit 14 uses the horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12 and the perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13. Then, the horizontal axis distortion and the perspective projection distortion of the original image n are corrected, and the corrected image is output as a primary corrected image n (1 ≦ n ≦ 8).
The reference length reading unit 15 performs a process of reading the apparent length of the vertical reference bar in the primary correction image n output from the primary correction image generation unit 14.

The inter-image magnification calculator 16 calculates the in-image magnification of the object to be viewed by the multi-view target calculated by the in-image magnification calculator 11 and the apparent vertical reference bar in the primary correction image n read by the reference length reader 15. Using the length, a process of calculating an image-to-image magnification for each primary correction image n in a case where an object to be multiviewed in the primary correction image n (1 ≦ n ≦ 8) is displayed with the same size is performed. .
The in-image magnification calculation unit 11 and the inter-image magnification calculation unit 16 constitute image magnification calculation means. At this time, the size may be corrected by enlarging or reducing the primary correction image using the inter-image magnification.

The secondary correction image generation unit 17 includes a horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12, a perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13, and an image. By performing secondary perspective transformation using the inter-image magnification calculated by the inter-image magnification calculation unit 16, the distortion and size of the original image n (1 ≦ n ≦ 8) are corrected, and the corrected image Is output as a secondary corrected image n (1 ≦ n ≦ 8).
The camera horizontal axis distortion correction parameter calculation unit 12, the perspective projection distortion correction parameter calculation unit 13, and the secondary correction image generation unit 17 constitute distortion correction means.

The secondary perspective transformation coordinate calculation unit 18 is calculated by the secondary correction image generation unit 17 by the horizontal axis distortion correction parameter calculated by the camera horizontal axis distortion correction parameter calculation unit 12 and the perspective projection distortion correction parameter calculation unit 13. The image coordinates of the multi-view target object when the second perspective transformation is performed using the perspective projection distortion correction parameter and the inter-image magnification calculated by the inter-image magnification calculating unit 16 (the multi-view target object image coordinates). A process of calculating the image coordinates of the movement destination is performed.
The movement parameter calculation unit 19 performs a process of calculating, as a movement parameter, a shift between the image coordinates of the movement destination of the multi-view target object calculated by the secondary perspective transformation coordinate calculation unit 18 and the image center.

The multi-view image generation unit 20 moves the secondary correction image n (1 ≦ n ≦ 8) output from the secondary correction image generation unit 17 in accordance with the movement parameter calculated by the movement parameter calculation unit 19. A process of generating a multi-view image n (1 ≦ n ≦ 8) in which a multi-view target object exists at the center of the image is performed.
The secondary perspective transformation coordinate calculation unit 18, the movement parameter calculation unit 19, and the multi-view image generation unit 20 constitute an image conversion unit.

The multi-view position designation unit 21 performs a process of receiving designation of a new multi-view position for the multi-view image n (1 ≦ n ≦ 8) generated by the multi-view image generation unit 20.
The multi-view image coordinate conversion unit 22 is a reverse movement parameter of the movement parameter calculated by the movement parameter calculation unit 19, an inverse conversion of the secondary perspective conversion calculated by the secondary perspective conversion coordinate calculation unit 18, and a planar projection conversion matrix. Using the inverse transformation matrix of the planar projective transformation matrix calculated by the calculation unit 8, the coordinate position on the multi-view image n (1 ≦ n ≦ 8) received by the multi-view position specification unit 21 is converted into the original A process of converting to a coordinate position on the image n (1 ≦ n ≦ 8) is performed.

  FIG. 2 is an explanatory diagram showing a situation where images are taken using eight cameras 1-n (1 ≦ n ≦ 8), and FIG. 3 shows cameras 1-1, 1-3, and 1-5. , 1-7.

Next, the operation will be described.
In the first embodiment, as shown in FIG. 2, a floor surface composed of a horizontal plane is considered, and a “stuffed animal” or “vase” is placed on the floor surface, and the surroundings of the “stuffed animal” or “vase” are arranged. It is assumed that eight cameras 1-n (1 ≦ n ≦ 8) fixed on a tripod of substantially the same size are arranged so as to surround.
However, it is assumed that all of the eight cameras 1-n (1 ≦ n ≦ 8) are installed at positions where the “floor surface area within the field of view of all cameras” shown in FIG.
In FIG. 2, the installation form of the eight cameras 1-n and the “floor area that falls within the field of view of all cameras” are indicated by a circular or elliptical dotted line. However, this dotted line does not actually exist. Good.

Further, in the example of FIG. 2, a square marking that fits the frame of “a floor area that falls within the field of view of all cameras” is provided, and all eight cameras 1-n (1 ≦ n ≦ 8) It is assumed that the four corners of the marking can be photographed without being disturbed by “stuffed animals” or “vases”.
In the example of FIG. 2, a case using a quadrangular marking is shown, but it is not limited to a quadrangular marking as long as four or more points can be photographed.
In addition, although eight cameras 1-n are supposed to photograph four points simultaneously, it is sufficient that at least four or more of the same points can be photographed simultaneously by two different cameras, and it is not necessary to be a quadrangular quadrangular.

In FIG. 2, the vertical reference bar is a bar that stands vertically near the center of the “floor area that enters the field of view of all cameras”, and is marked at a position that is half the total length. .
The topmost vertex of the vertical reference bar, the center marking, and the contact point in contact with the floor surface are all the eight cameras 1-n (1 ≦ n ≦ 8) are used as “stuffed animals” and “vases”. It is assumed that the camera can be photographed without interruption.
The vertical reference rod is sufficiently thin and can be regarded as a straight line when photographed by the camera 1-n (1 ≦ n ≦ 8).

In the example of FIG. 2, the vertical reference rod is used for the purpose of reading the size when taken in different directions and calculating the vanishing point obtained by projecting the infinity point of a straight line perpendicular to the plane onto the image. It is what is done.
The vertical reference bar is shaped as described above to improve accuracy, but the vertical reference bar is used only for reading the size, and the vanishing point is calculated by another method as described later. You may make it calculate. In this case, the vertical reference rod does not have to have the form as described above, and may be substituted by, for example, an actual person.

The vertical reference bar in FIG. 2 is marked at a position of a half of the entire length, but may not be a position of a half, and it is sufficient that the ratio is known.
Further, the vertical reference bar in FIG. 2 is set near the center of the “floor surface area that enters the field of view of all cameras”, but it may not be near the center.
In addition, the vertical reference bar in FIG. 2 can be photographed from all the cameras 1-n (1 ≦ n ≦ 8), with the topmost vertex, the contact point in contact with the floor surface, and the central marking, It is not always necessary to photograph the same vertical reference rod. For example, the vertical reference bar A photographed in common by the camera 1-1 and the camera 1-2 and the vertical reference stick B photographed in common by the camera 1-1 and the camera 1-3 may not be the same. . In this case, it is only necessary to know the size of the vertical reference rod A and the size of the vertical reference rod B in the camera 1-1. In this embodiment, a vertical reference bar, which is a bar, is used as a reference for the vertical direction and size. In this case, once installed, there is an advantage that the shape is not easily lost and it is easy to measure a straight line. On the other hand, a string or the like may be used for this purpose instead of a stick. In this case, a weight or the like is attached to the tip, and it is hung vertically so that the tip contacts the floor. In this case, the vertical system is high and the length can be easily adjusted.

In FIG. 2, a horizontal reference bar for the camera 1-7 is drawn.
The horizontal reference bar is, for example, a floor surface so that the optical axis of the target camera (camera 1-7 in the example of FIG. 3) is parallel to a straight line projected onto a plane along a direction perpendicular to the floor surface. Is installed.
In addition, marking is provided at a half of the length of the horizontal reference bar. It is assumed that the markings at both ends and the center of the horizontal reference bar are ready to be photographed without being disturbed by the “stuffed toy” or “vase” from the camera 1-7.
The horizontal reference rod is sufficiently thin and can be regarded as a straight line when photographed by the camera 1-7.
In addition, the horizontal reference bar is marked at a position that is a half of the entire length, but it may not be a position that is a half, and it is sufficient that the ratio is known. In addition, the horizontal reference rod is, for example, “installed on the floor so that the optical axis of the camera is parallel to a straight line projected onto the floor along a direction perpendicular to the floor”. There is an effect that the accuracy is easily improved. In addition, the horizontal reference bar may be in either direction, and a vanishing point obtained by projecting a straight infinity point on the floor surface including the horizontal reference bar onto the image may be used. In this case, there is an effect that it is not necessary to move the horizontal reference bar for each camera. Furthermore, as shown in FIG. 2, “a square side marked on the floor” may be used. Furthermore, this quadrangle may be a square, a rectangle, or a parallelogram, and an infinite point of a straight line on the floor surface may be obtained using the two sides facing each other, and a vanishing point obtained by projecting it on the image may be used. .
In this embodiment, a horizontal reference bar is used as a horizontal reference. However, as described above, an example using a string instead of a vertical reference bar may be used instead of a string. . In that case, it is necessary to apply tension so that the string becomes linear, but on the other hand, there is an advantage similar to the case where the vertical reference bar is roughly made of a string.

In the first embodiment, the cameras 1-1 to 1-8 release the shutter at the same synchronization timing from the outside, and shoot an image with a frame at the same synchronization timing.
Shooting with such synchronized timing is essential when the subject moves. For example, if the stuffed animal, vase, horizontal reference bar / vertical reference bar, floor marking, etc. do not move, It does not have to be synchronized. In that case, it is possible to sequentially shoot with one camera without preparing eight cameras 1-1 to 1-8.
In the first embodiment, a subject to be multi-viewed such as “stuffed toy” or “vase”, a horizontal reference bar / vertical reference bar for obtaining parameters necessary for multi-viewing, floor surface marking, etc. After taking a picture, they may be removed without moving the camera, and a subject to be multi-viewed such as “stuffed toy” may be placed and photographed. In this case, care must be taken not to move the camera, but on the other hand, there is an advantage that the reference bar and the marking on the floor surface can be taken without being obstructed by the subject to be multiviewed.

Hereinafter, the processing content of the image generation apparatus will be specifically described.
When the camera 1-n (1 ≦ n ≦ 8) captures the same three-dimensional area from different directions, the image data temporary storage unit 2 is an original image captured by the camera 1-n (1 ≦ n ≦ 8). n (1 ≦ n ≦ 8) is temporarily stored.

When the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the multi-view position specifying unit 3 stores an arbitrary original image n ₁ from the original image n (1 ≦ n ≦ 8). A process of accepting selection and accepting designation of image coordinates indicating a position on the floor surface of an object to be viewed in the original image n ₁ (an object for which the user desires multi-view) is performed.
The image coordinates are read by the multi-view position designating unit 3 while the user views the original image n ₁ and, for example, operates the mouse to designate the position in the image, thereby reading the image coordinates at that position. Alternatively, it may be automatically specified by some method.

Further, when the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the reference position specifying unit 4 selects an arbitrary original image n ₂ from the original image n (1 ≦ n ≦ 8). And a process of receiving designation of image coordinates indicating the position on the floor surface where the vertical reference bar existing in the original image n ₂ is in contact with the floor surface.
The reading of the image coordinates in the reference position specifying unit 4 may be performed by reading the image coordinates of the position by, for example, operating the mouse and specifying the position in the image while the user views the original image n _2. Alternatively, it may be automatically specified by some method.
When the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the floor surface coordinate reading unit 5 exists in common on the floor surface of the original image n (1 ≦ n ≦ 8). Processing for reading the image coordinates of the vertex (1), vertex (2), vertex (3), and vertex (4) of the marking is performed.
The reading of the image coordinates in the floor surface coordinate reading unit 5 may be performed by, for example, operating the mouse to specify the position in the image while the user views the original image n, thereby reading the image coordinates at that position. Alternatively, it may be automatically specified by some method.

In addition, when the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the horizontal vanishing point calculation unit 6 performs horizontal processing of the original image n for each original image n (1 ≦ n ≦ 8). The vanishing point, for example, the image coordinates of the vanishing point obtained by projecting the infinity point of a straight line on the floor surface obtained by projecting the optical axis of the camera 1-n in the direction perpendicular to the floor surface is calculated.
As a method of calculating the horizontal vanishing point, for example, it can be obtained by using a horizontal reference bar placed on a straight line that satisfies the above-mentioned conditions of the floor surface.
That is, the horizontal reference bar is marked, and the horizontal vanishing point can be obtained from the actual position of the marking and the apparent position on the image. Since this is the same as the method for obtaining the vanishing point, see the method for obtaining the vertical vanishing point described later).
In addition, it is also possible to set two or more straight lines parallel to the straight line on the floor and use the intersection point when these straight lines are projected on the image as vanishing points.

When the image data temporary storage unit 2 stores the original image n (1 ≦ n ≦ 8), the vertical vanishing point calculation unit 7 calculates the vertical vanishing point of the original image n for each original image n (1 ≦ n ≦ 8). That is, the image coordinates of the vanishing point obtained by projecting a straight line infinity point perpendicular to the floor surface to the image are calculated.
Hereinafter, an example of a method for calculating the vertical vanishing point will be described.
In the first embodiment, a vertical reference bar exists in all original images n (1 ≦ n ≦ 8), and the vertical reference bar has a marking at the center in the length direction as shown in FIG. It has been subjected.
When the vertical reference bar is projected on the image as shown in FIG. 6, the three points X1, X2, and X3 on the vertical reference bar and the infinity point X _∞ on the straight line passing through the three points X1, X2, and X3. but in the image, respectively x1, x2, x3, and those projected to x _∞.

式（１）において、Ｘ_∞は無限に遠い点であるから｜Ｘ_∞−Ｘ₂｜／｜Ｘ_∞−Ｘ₃｜＝１であり、また、垂直基準棒の実距離の比は｜Ｘ₁−Ｘ₂｜／｜Ｘ₁−Ｘ₃｜＝２である。
また、｜ｘ₁−ｘ₂｜／｜ｘ₁−ｘ₃｜は、画像座標から得ることができるので、垂直消失点ｘ_∞の座標（ｘ_vin，ｙ_vin）を算出することができる。
詳細は非特許文献「小島他 “消失点を用いた多視点カメラキャリブレーション”ＦＩＴ２００４」に開示されている。
この他、上記床面に垂直な直線（棒などで代用する）を２つ以上設定して、それらの直線を画像に投影した場合の２つの直線の交点を垂直消失点として利用することもできる。 At this time, since the cross ratio calculated from the four points on the straight line is unchanged before and after the projection, the following equation (1) is established.

In the formula (1), X _∞ infinitely distant point because it _{| X ∞ -X 2 | / |} X ∞ -X 3 | a = 1, also, the ratio of the actual distance of the vertical reference rod | X _{1 -X 2 | / | X 1 -X} 3 | a = 2.
Since | x ₁ −x ₂ | / | x ₁ −x ₃ | can be obtained from the image coordinates, the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ can be calculated.
Details are disclosed in the non-patent document “Kojima et al.“ Multi-viewpoint camera calibration using vanishing points ”FIT 2004”.
In addition, it is possible to set two or more straight lines (substitute with a bar or the like) perpendicular to the floor surface and use the intersection of the two straight lines as a vertical vanishing point when these straight lines are projected on an image. .

The planar projective transformation matrix calculation unit 8 includes a marking vertex (1), a vertex (2), and a vertex that are commonly present on the floor surface of the original image n (1 ≦ n ≦ 8). (3) When the image coordinates of the vertex (4) are read, the marking vertex (1), vertex (2), vertex (3) in the original image n ₁ and other original images n (excluding the original image n ₁ ) , Using the image coordinates of the vertex (4), between the original image n ₁ and another original image n (excluding the original image n ₁ ), a plane projection transformation matrix H _n1-1 of the plane constituting the floor surface H _n1-8 (7 pieces of 1 ≦ n ₁ ≦ 8) is calculated.
Further, the plane projective transformation matrix calculation unit 8 performs marking vertex (1), vertex (2), vertex (3), vertex (4) in the original image n ₂ and other original images n (excluding the original image n ₂ ). ), The plane projection transformation matrices H _{n2-1 to} H _n2-8 of the plane constituting the floor surface between the original image n ₂ and another original image n (excluding the original image n ₂ ) are _used. (7 of 1 ≦ n ₂ ≦ 8) is calculated.
The calculation method of the planar projective transformation matrix is described in, for example, the non-patent document “Koichiro Deguchi“ Basics of Robot Vision ”p47 (described as a homography matrix in this document)”.

When the plane projection transformation matrix calculation unit 8 calculates the plane projection transformation matrices H _{n1-1 to} H _n1-8 , the multi-view position calculation unit 9 uses the plane projection transformation matrices H _{n1-1 to} H _n1-8. Then, the image coordinates indicating the position on the floor surface of the object of the multi-view target existing in the original image n (excluding the original image n ₁ ) are calculated.
That is, the multi-view position calculation unit 9 uses the plane projection transformation matrices H _{n1-1 to} H _n1-8 to determine the position on the floor of the object to be multi-viewed that has been designated by the multi-view position designation unit 3. Or the image coordinates converted by the multi-view image coordinate conversion unit 22 is converted into the image coordinates of the original image n (excluding the original image n ₁ ), thereby obtaining the original image n (original image n _1). Image coordinates indicating the position on the floor surface of the object of the multi-view target existing in (1) is calculated.
Note that the image coordinates indicating the position on the floor surface of the object of the multi-view target that has been designated by the multi-view position designating unit 3 are converted, or the image coordinates converted by the multi-view image coordinate converting unit 22 are converted. For conversion, for example, the former may be selected only when the system is activated, or both may be monitored constantly and the one that has changed may be selected.

When the plane projection transformation matrix calculation unit 8 calculates the plane projection transformation matrices H _{n2-1 to} H _n2-8 , the reference position calculation unit 10 uses the plane projection transformation matrices H _{n2-1 to} H _n2-8 . Image coordinates indicating the position of the vertical reference bar on the floor surface existing in the original image n (excluding the original image n ₂ ) are calculated.
That is, the reference position calculation unit 10 uses the plane projection transformation matrices H _{n2-1 to} H _n2-8 to specify the floor position of the vertical reference bar in the original image n ₂ that has been specified by the reference position specification unit 4. Is converted into the image coordinates of the original image n (excluding the original image n ₂ ), so that the position on the floor surface of the vertical reference bar existing in the original image n (excluding the original image n ₂ ) is converted. The image coordinates shown are calculated.

Up to this point, the original image whose selection has been received by the multi-view position specifying unit 3 is the original image n ₁ , and the original image whose selection has been received by the reference position specifying unit 4 is the original image n ₂ . However, the original image n ₁ and the original image n ₂ may be separate images or the same image.
If the original image n ₁ and the original image n ₂ are the same original image (n ₁ = n ₂ ), there are _seven plane projection matrices H _{n1-1 to} H _n1-8 (1 ≦ n ₁ ≦ 8). Only need.
Hereinafter, a case where n ₁ = n ₂ will be described.

The in-image magnification calculation unit 11 calculates image coordinates indicating the floor surface position of the multi-view target existing in the original image n (1 ≦ n ≦ 8) by the multi-view position calculation unit 9, and the reference position calculation unit 10 performs the original position calculation. When the image coordinates indicating the position on the floor surface of the vertical reference bar existing in the image n are calculated, the image coordinates and the image coordinates of the horizontal vanishing point of the original image n calculated by the horizontal vanishing point calculating unit 6 are obtained. Use this to move the multi-view target object to the floor position at the same depth as the vertical reference bar. In-image magnification m _{n of the} multi-view target object (change in the appearance of the multi-view target object Magnification) is calculated.

Hereinafter, a method for calculating the in-image magnification will be described.
FIG. 4 shows an example in which the horizontal vanishing point is at the center of the image, and the camera is installed so that the optical axis of the camera is parallel to the floor and the horizontal axis of the camera image is substantially parallel to the floor. It is explanatory drawing which shows.
FIG. 5 shows a case where the horizontal vanishing point is above the center of the image, and the horizontal axis of the camera image is substantially parallel to the floor surface, but the optical axis of the camera is slightly downward with respect to the horizontal floor surface. It is explanatory drawing which shows the example currently image | photographed.

また、これらの垂直基準棒の位置や大きさ、および水平消失点の位置を図４や図５に示すように利用することで、被写体である物体の大きさや人物の身長などを精度よく推定することができる。またそれらの物体や人物の大きさがわかっている場合、その物体や人物が存在する床面上の位置を容易に知ることができる。 4 and 5, the coordinates of the horizontal vanishing point are (x _hin , y _hin ), the position coordinates of the multi-view target object are (x _mv , y _mv ), and the position coordinates of the same depth as the vertical reference bar are (x Assuming that _b , y _b ), the size of the multi-view target object can be approximated by the following equation (2) when the multi-view target object is moved to the position of the vertical reference bar. Therefore, the in-image magnification m _n can be calculated from the following equation (2).

Further, by using the position and size of these vertical reference bars and the position of the horizontal vanishing point as shown in FIG. 4 and FIG. 5, the size of the object that is the subject, the height of the person, etc. can be accurately estimated. be able to. In addition, when the sizes of these objects and persons are known, it is possible to easily know the position on the floor where the objects and persons exist.

When the vertical vanishing point calculating unit 7 calculates the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ of the original image n (1 ≦ n ≦ 8), the camera horizontal axis distortion correction parameter calculating unit 12 calculates the original image n. Using the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ , a correction parameter for the horizontal axis distortion caused by the tilt of the camera 1-n when the original image n is captured is calculated.
That is, for example, the camera horizontal axis distortion correction parameter calculation unit 12 performs the image center point B (x _size / 2, y _size / 2) and the vertical vanishing point x with respect to the vertical axis of the image as shown in FIG. _Considering that the inclination (angle rz) of the straight line connecting _∞ can be approximated to the camera horizontal axis distortion, the angle rz is calculated.
For example, the horizontal size of the original image is x _size , the vertical size is y _size , the coordinates of the vertical vanishing point x _∞ are (x _vin , y _vin ), and y _vin >> x _size , y _vin >> y _size If there is, the following equation (3) is established.

Here, with respect to the vertical axis of the image, the inclination (angle rz) of the straight line connecting the center point B (x _size / 2, y _size / 2) of the image and the vertical vanishing point x _∞ is approximated to the camera horizontal axis distortion. Assuming that it is possible to calculate the angle rz, the center point of the upper end of the image (x _size / 2, 0) and the center point of the lower end of the image (x _{size are shown).} / 2, y _size ) may be regarded as being able to approximate the camera horizontal axis distortion.
Furthermore, instead of approximation, an accurate angle may be calculated and used.
Also, the other image of the upper end of the middle point (x _size / 2,0), the midpoint of the lower end of the image _{(x size / 2, y size} ) may be used a point on a line connecting the, other A point inside the image may be used. Furthermore, you may utilize the point on the plane containing an image.

The perspective projection distortion correction parameter calculation unit 13 calculates the vertical vanishing point x when the vertical vanishing point calculation unit 7 calculates the coordinates (x _vin , y _vin ) of the vertical vanishing point x _∞ of the original image n (1 ≦ n ≦ 8). _Using the coordinates (x _vin , y _vin ) of _∞ , a correction parameter for the perspective projection distortion of the image resulting from the perspective projection of the three-dimensional space onto the two-dimensional plane at the time of photographing by the camera 1-n is calculated.
The perspective projection distortion is a distortion that is considered to be caused by photographing with a camera, that is, a distortion that is considered to be generated by projective transformation of an object in a three-dimensional space to a plane. This is a phenomenon in which when there are a plurality of straight lines perpendicular to the floor surface and the straight lines are projected on the image, the straight lines are not parallel to each other on the image.

As shown in FIG. 6, the perspective projection distortion correction parameter calculation unit 13 includes an image including the center point B (x _size / 2, y _size / 2) of the image with respect to the image whose camera horizontal axis distortion has been corrected previously. Perspective projection distortion at an angle (angle rx) between the straight line connecting the first pixel (0, y _size / 2) of the horizontal line and the vertical vanishing point and the vertical axis of the image after correcting the camera horizontal axis distortion Is approximated and the angle rx is calculated.
For example, the horizontal size of the original image is x _size , the vertical size is y _size , the coordinates of the vertical vanishing point x _∞ are (x _vin , y _vin ), and y _vin >> x _size , y _vin >> y _size If there is, the following equation (4) is established.

Here, assuming that the horizontal axis distortion correction angle rz is sufficiently small, the perspective projection distortion correction angle rx is defined as “the first horizontal line of the image including the center point B (x _size / 2, y _size / 2) of the image”. It is assumed that the perspective projection distortion can be approximated by the inclination (angle rx) of the vertical axis at one pixel (0, y _size / 2). However, as shown in FIG. 6, the center point of the image perpendicular to the straight line BX _∞ The coordinates of a point on a straight line passing through may be used.
Further, “the perspective projection distortion is approximated by the inclination of the vertical axis at the x _size pixel (x _size , y _size / 2) of the horizontal line of the image including the center point B (x _size / 2, y _size / 2) of the image. Can be used ", a straight line connecting points on the left and right ends of another image may be used, or a straight line connecting points in the other image may be used.
Furthermore, instead of approximation, an accurate angle may be calculated and used.

When the camera horizontal axis distortion correction parameter calculation unit 12 calculates a correction parameter for horizontal axis distortion and the perspective projection distortion correction parameter calculation unit 13 calculates a correction parameter for perspective projection distortion, the primary correction image generation unit 14 calculates the horizontal correction. Using the correction parameter for axial distortion and the correction parameter for perspective projection distortion, the horizontal axis distortion and perspective projection distortion of the original image n are corrected, and the corrected image is defined as a primary corrected image n (1 ≦ n ≦ 8). Output.
That is, as shown in FIG. 7, the primary corrected image generation unit 14 sets the three-dimensional coordinate axis so that the origin of the three-dimensional coordinate is the image center and the image is included in the XY plane, and performs perspective projection. Consider a case where the screen is set on a plane perpendicular to the Z axis and the viewpoint is set on the Z axis.
At this time, the image is rotated by rz around the Z axis, and a perspective projection image on the screen when the image is rotated by rx around the X axis is calculated and output as a primary correction image.
At this time, rz is a horizontal axis distortion correction parameter calculated by the camera horizontal distortion correction parameter calculation unit 12, and rx is a perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13.

FIG. 8 is an explanatory view of the three-dimensional coordinates of FIG. 7 viewed from the positive direction of the X axis.
In the example of FIG. 8, the camera horizontal axis distortion correction parameter calculation unit 12 calculates a horizontal axis distortion correction parameter at the image center, and the perspective projection distortion correction parameter calculation unit 13 at a point on the horizontal axis of the image including the image center. The perspective transformation distortion correction parameter is calculated, and the primary correction image generation unit 14 performs the perspective projection transformation with the image center as the origin. The horizontal axis distortion correction parameter and the perspective transformation distortion correction parameter of the point that is not the image center are set. It may be calculated and perspective projection conversion with the point that is not the center of the image as the origin may be performed. Further, the perspective transformation may be performed without matching the two.

When the primary correction image generation unit 14 outputs the primary correction image n (1 ≦ n ≦ 8), the reference length reading unit 15 determines the apparent length L _n of the vertical reference bar in the primary correction image n. read. At this time, the user may read while viewing the primary correction image n, or may automatically read by some method.
Here, the reading of the apparent length L _n of the vertical reference bar in the primary correction image n is shown, but the apparent length L _n of the vertical reference bar in the original image n may be read. .
In this case, since the process becomes simple, there is an effect that the processing speed can be improved.
When the reference length reading unit 15 reads the apparent length L _n of the vertical reference bar in the primary correction image n, the inter-image magnification calculation unit 16 apparent length of the vertical reference bar in the primary correction image n The same multi-view target object in the primary correction image n (1 ≦ n ≦ 8) is used by using L _n and the intra-image magnification m _n of the multi-view target object calculated by the intra-image magnification calculation unit 11. An inter-image magnification ratio m _o1n when displaying in size is calculated for each primary correction image n.

ただし、Ｌ₁はオリジナル画像ｎ₁の１次補正画像ｎ₁から基準長読取部１５により読み取られ垂直基準棒の長さである。
式（５）のＬ₁は、倍率ｍ_olnを適当な範囲に収めるために使用される定数であるため、実際の基準棒の見かけの長さそのものでなくてもよく、全てのオリジナル画像ｎで同じであればよい。

However, L ₁ is the length of the vertical reference rod read by reference length reading unit 15 from the primary corrected image n ₁ original image n _1.
Since L _{1 in} equation (5) is a constant used to keep the magnification _moln within an appropriate range, it does not have to be the apparent length of the actual reference bar itself, and in all original images n If it is the same.

When the inter-image magnification calculation unit 16 calculates the inter-image magnification m _o1n for each primary correction image n, the secondary correction image generation unit 17 uses the inter-image magnification m _o1n and the camera horizontal axis distortion correction parameter calculation unit 12. By performing secondary perspective transformation using the calculated horizontal axis distortion correction parameter and the perspective projection distortion correction parameter calculated by the perspective projection distortion correction parameter calculation unit 13, the original image n (1 ≦ 1) is obtained. n ≦ 8) is corrected, and the corrected image is output as a secondary corrected image n (1 ≦ n ≦ 8).
That is, like the primary correction image generation unit 14, the secondary correction image generation unit 17 sets a three-dimensional coordinate axis, generates a perspective transformation image on the screen, and outputs it as a secondary correction image.
At this time, the primary correction image generation unit 14 has an image magnification (magnification of the image perspective-transformed on the screen) which is a parameter other than the Z-axis rotation rz and the X-axis rotation rx being 1 ×. As shown, the secondary correction image generation unit 17 _sets the magnification on the screen to m _o1n .

The secondary perspective transformation coordinate calculation unit 18 is calculated by the correction parameter for the horizontal axis distortion calculated by the camera horizontal axis distortion correction parameter calculation unit 12 by the secondary correction image generation unit 17 and the perspective projection distortion correction parameter calculation unit 13. The image coordinates of the multi-view target object when the second perspective transformation is performed using the perspective projection distortion correction parameter and the inter-image magnification calculated by the inter-image magnification calculation unit 16 (multi-view target object). Image coordinates of the movement destination).
When the secondary perspective transformation coordinate calculation unit 18 calculates the image coordinates of the movement destination of the multi-view target object, the movement parameter calculation unit 19 and the image coordinates of the movement destination and the image center (for example, 640 × 480 image) are obtained. In this case, the deviation from the center of the image (320, 240) is calculated as a movement parameter.
In this example, the deviation from the center of the image is calculated so that the multi-view target object moves to the center of the image. The deviation from the position may be calculated.

When the movement parameter calculation unit 19 calculates the movement parameter, the multi-view image generation unit 20 outputs the secondary correction image n (1 ≦ n ≦ 8) output from the secondary correction image generation unit 17 according to the movement parameter. By moving the object, a multi-view image n (1 ≦ n ≦ 8) in which the object to be multi-viewed exists at the center of the image is generated.
That is, the multi-view image generation unit 20 moves the secondary correction image n (1 ≦ n ≦ 8) output from the secondary correction image generation unit 17 by the movement amount indicated by the movement parameter, thereby correcting the secondary correction image. A multi-view image n (1 ≦ n ≦ 8) is generated from n (1 ≦ n ≦ 8).

FIG. 9 is an explanatory diagram illustrating an example in which the image of FIG. 3 is converted so that the size of the “stuffed animal” is the same in each image and the “stuffed animal” is positioned at the center of each image.
However, for ease of viewing, the horizontal reference bar and the vertical reference bar that are shown in each image are omitted.
For “vase”, the image of the camera 1-1 in FIG. 9 remains in the range of the image even after the deformation, but the image of the camera 1-3, the image of the camera 1-5, and the image of the camera 1-7. The image is not visible because it has moved out of the image range. As for the “rectangle marked on the floor”, the portion outside the range of the image is indicated by a dotted line.

The multi-view position designation unit 21 performs a process of receiving designation of a new multi-view position for the multi-view image n (1 ≦ n ≦ 8) generated by the multi-view image generation unit 20.
The new multi-view position may be specified while the user is viewing the multi-view image, or may be automatically specified by some method.

When the multi-view position specifying unit 21 receives the designation of a new multi-view position, the multi-view image coordinate converting unit 22 receives the reverse moving parameter of the moving parameter calculated by the moving parameter calculating unit 19 and the secondary perspective conversion coordinate. Using the inverse transformation of the secondary perspective transformation calculated by the calculation unit 18 and the inverse transformation matrix of the plane projection transformation matrix calculated by the plane projection transformation matrix calculation unit 8, designation is accepted by the multiview position designation unit 21. The coordinate position (new multi-view position) on the obtained multi-view image n (1 ≦ n ≦ 8) is converted to the coordinate position on the original image n (1 ≦ n ≦ 8).
Here, using the inverse transformation matrix of the planar projection transformation matrix calculated by the planar projection transformation matrix calculation unit 8, the coordinate position on the multi-view image n (1 ≦ n ≦ 8) is converted to the original image n (1 ≦ n ≦ 8). 8) Although shown about what is converted to the upper coordinate position, the coordinate position at the stage of being converted to the coordinate position on the original image n (1 ≦ n ≦ 8) by not using the inverse transformation matrix of the planar projective transformation matrix May be substituted for the coordinate position corresponding to the multi-view position for each original image n calculated by the multi-view position calculation unit 9.

  As is apparent from the above, according to the first embodiment, the position of the object to be multiviewed calculated by the multiview position calculation unit 9 and the floor position of the vertical reference bar calculated by the reference position calculation unit 10 Using the size of the vertical reference bar calculated by the reference length reading unit 15, the positions of the objects of the multi-view target in the original image n (1 ≦ n ≦ 8) are aligned, and Since the image magnification with the same size is calculated for each original image n and the original image n is converted according to the image magnification, the size of the subject can be determined without performing high-precision camera calibration. There is an effect that it is possible to generate an easy-to-see image with matching positions and the like.

In other words, according to the first embodiment, an object placed in contact with a certain plane and a multi-view image after conversion of the image A photographed from a certain direction and an image obtained by photographing the object from another direction. In the multi-view image after the conversion of B, the position of the object is the same and the size is the same.
In addition, by specifying the position of another object placed at a different position on the plane while viewing the multi-view image, a new multi-view with the same position and the same size of the other object An image can be easily generated from images A and B.

For example, the position of a vanishing point (horizontal vanishing point) that is a point obtained by projecting the infinity point of a straight line on a plane, which is a straight line obtained by projecting the optical axis of the camera perpendicular to the floor surface, and an image A And the position and size of the same object S appearing in the image B and the position of the object T, a magnification is obtained so that the size of the object T is the same in the converted image. The magnification for making the same size can be obtained with high precision anywhere in the plane.
A point obtained by projecting a distortion caused by the horizontal tilt of the camera of images A and B and a distortion caused by the principle of perspective projection at the time of photographing onto an image at a point at a straight line at infinity perpendicular to the plane. Using the perspective projection transformation after rotating the angle formed by the straight line connecting the vanishing point (vertical vanishing point) and the point inside the image with the vertical axis of the image as the rotation angle of the image placed in the three-dimensional space, Since image correction is performed, it is possible to perform image correction with less distortion of an object caused by image correction. In addition, parameters necessary for correction can be easily determined.

  When the coordinates of the position a where the object in the image A is placed are known, the coordinates of the position b where the object in the image B is placed are calculated using the plane projective transformation matrix for the plane. Thus, the movement of the image such that the position a and the position b are at the same position in the image can be accurately and efficiently performed for all points on the plane.

In the first embodiment, the camera is assumed to be an “ideal pinhole camera model”. In this “ideal pinhole camera model”, the camera optical center, which is an internal parameter of the camera, coincides with the image center of the captured image, and the XYZ axes of the camera coordinates are also orthogonal to each other for the internal parameters of the camera. In addition, it is assumed that the camera coordinates coincide with the image coordinates photographed by the camera.
When these assumptions do not hold, it is necessary to improve the algorithm in consideration of errors in the assumptions. However, in many cases, there are few problems even if the processing content of the first embodiment is performed using an image captured by a normal camera.
The “ideal pinhole camera model” and “internal parameters of the camera” are disclosed in non-patent documents such as Koichiro Deguchi Robot Vision Basic Corona.

In the first embodiment, lens distortion (distortion aberration) is ignored with respect to the image captured by the camera.
This “lens distortion” includes distortion called barrel distortion and pincushion distortion, but this “lens distortion” is negligible as long as a good lens is used under good conditions. However, it may not always be ignored depending on the performance of the lens used, the distance between the camera and the subject, and image processing performed by the camera after shooting.
If this “lens distortion” cannot be ignored, the lens distortion is corrected for the photographed image and then used as the original image of the first embodiment, and the subsequent algorithm does not need to be improved (image) The data is not changed after the temporary data storage unit 2).
The correction of lens aberration distortion is disclosed in a non-patent document “Koichiro Deguchi Robot Vision Basic Corona”.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows the image generation apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows a mode that the image is image | photographed using eight cameras 1-n (1 <= n <= 8). It is explanatory drawing which shows the image image | photographed with the camera 1-1, 1-3, 1-5, 1-7. It is explanatory drawing which shows the example installed so that the horizontal vanishing point is in the center of the image, the optical axis of the camera is parallel to the floor surface, and the horizontal axis of the camera image is substantially parallel to the floor surface. is there. It is explanatory drawing which shows the example which is making the horizontal axis of a camera image substantially parallel to a floor surface, and makes the optical axis of a camera face down slightly with respect to a horizontal floor surface. It is explanatory drawing explaining calculation of the correction parameter of the horizontal axis distortion of a camera, and perspective projection distortion. It is explanatory drawing explaining correction | amendment of the horizontal axis distortion and perspective projection distortion of the original image n. It is explanatory drawing which looked at the three-dimensional coordinate of FIG. 7 from the positive direction of the X-axis. It is explanatory drawing which shows the example of a conversion of the image of FIG.

Explanation of symbols

  1-1 to 1-N camera, 2 image data temporary storage unit (image storage unit), 3 multi-view position designation unit (position designation unit), 4 reference position designation unit (position designation unit), 5 floor coordinate reading unit (Marking position acquisition means), 6 horizontal vanishing point calculation section (vanishing point calculation means), 7 vertical vanishing point calculation section (vanishing point calculation means), 8 plane projection transformation matrix calculation section (plane projection transformation matrix calculation means), 9 Multi-view position calculation unit (position calculation unit), 10 reference position calculation unit (position calculation unit), 11 intra-image magnification calculation unit (image magnification calculation unit), 12 camera horizontal axis distortion correction parameter calculation unit (distortion correction unit), 13 perspective projection distortion correction parameter calculation unit (distortion correction unit), 14 primary correction image generation unit, 15 reference length reading unit, 16 inter-image magnification calculation unit (image magnification calculation unit), 17 secondary correction image generation Unit (distortion correction unit), 18 secondary perspective transformation coordinate calculation unit (image conversion unit), 19 movement parameter calculation unit (image conversion unit), 20 multi-view image generation unit (image conversion unit), 21 multi-view position designation unit 22 Multi-view image coordinate conversion unit.

Claims (5)

  An image storage means for storing a plurality of images in which the same three-dimensional region is photographed from different directions, a position of a multi-view target object existing in an arbitrary image among the plurality of images, and Position specifying means for specifying a floor surface position where a vertical reference bar existing in an arbitrary image is in contact with the floor surface, marking position acquiring means for acquiring a marking position on the floor surface in the plurality of images, and the marking A plane projection transformation matrix calculating means for calculating a plane projection transformation matrix between the plurality of images from the marking position obtained by the position obtaining means; a position of the object to be multi-view designated by the position designation means; and the plane. While calculating the position of the object of the multi view object which exists in the above-mentioned plurality of images from the plane projection transformation matrix computed by the projection transformation matrix calculation means Position calculating means for calculating the floor position of the vertical reference bar present in the plurality of images from the floor position of the vertical reference bar specified by the position specifying means and the planar projective transformation matrix; and calculating by the position calculating means The position of the multi-view target object in the plurality of images is aligned and the size of the multi-view target object is the same using the position of the object of the multi-view target and the floor position of the vertical reference bar. An image generation apparatus comprising: an image magnification calculation unit that calculates an image magnification for each of the plurality of images; and an image conversion unit that converts the plurality of images according to the image magnification calculated by the image magnification calculation unit.   The image magnification calculation means moves the multi-view target object to the floor position of the vertical reference bar using the position of the object of the multi-view target calculated by the position calculation means and the floor position of the vertical reference bar. In this case, the in-image magnification calculating unit that calculates the in-image magnification of the object of the multi-view target for each of a plurality of images, the in-image magnification calculated by the in-image magnification calculating unit and the length of the vertical reference bar, 2. The inter-image magnification calculating unit that calculates an inter-image magnification for each of the plurality of images when the multi-view target objects in the plurality of images are displayed with the same size. The image generating apparatus described.   A vanishing point calculating unit that calculates vanishing points in a plurality of images, a vanishing point calculated by the vanishing point calculating unit, and an inter-image magnification calculated by the image magnification calculating unit, correct distortion of the plurality of images. The image generation apparatus according to claim 2, further comprising a distortion correction unit.   The distortion correction unit is calculated by the camera horizontal axis distortion correction parameter calculation unit that calculates the correction parameter of the horizontal axis distortion caused by the tilt of the camera from the vanishing point calculated by the vanishing point calculation unit and the vanishing point calculation unit. Calculated by a perspective projection distortion correction parameter calculation unit that calculates a correction parameter for perspective projection distortion of an image resulting from perspective projection from a vanishing point, the camera horizontal axis distortion correction parameter calculation unit, and the perspective projection distortion correction parameter calculation unit 4. The image generation apparatus according to claim 3, further comprising: a correction image generation unit that corrects distortion of a plurality of images in accordance with the correction parameter and the inter-image magnification calculated by the image magnification calculation means.   The image generation apparatus according to claim 4, wherein the correction image generation unit corrects vertical or horizontal shifts of the plurality of images using perspective transformation.

Images