JP6344050B2 - Image processing system, image processing apparatus, and program - Google Patents

Description

  The present invention relates to an image processing system that reduces distortion on a projection surface of a projected image.

  A projection method is known in which an image projected by a projector is corrected according to the shape of a screen or a three-dimensional object serving as a projection surface, and a new visual effect or effect is produced by the projector. Such a projection technique is sometimes called projection mapping. For example, it is considered to use the visual effects and effects of projection mapping also in events and digital signage.

  Also, multi-projection is known in which images are projected by a plurality of projectors to project a large screen image that cannot be obtained by a single projector. In the multi-projection, the brightness of the overlapping region is adjusted so that the boundary is not conspicuous by partially overlapping the projection ranges of a plurality of projectors, or the overlapping portion can be seen with a sense of incongruity with the surroundings.

  Even in this multi-projection, an image may be projected onto a three-dimensional object such as a curved surface (see, for example, Patent Document 1). In Patent Document 1, a structured pattern is projected on a display surface by a projector, the coordinates of the structured pattern are determined from an image captured by a camera having a fixed physical relationship with the projector, and distortion of the projected image is calculated. A method of correcting is disclosed.

  By the way, the distortion correction performed for correcting the distortion of the three-dimensional shape of the projection surface is possible without particularly acquiring the distance information to the projection surface. However, since information on the shape of the projection plane can be obtained by using the distance information, it is expected that more accurate distortion correction can be performed.

  However, the conventional distortion correction method has a problem that distortion correction using the shape of the projection surface is not performed.

  In view of the above problems, an object of the present invention is to provide an image processing system capable of correcting distortion with higher accuracy using the shape of a projection surface.

In view of the above problems, the present invention is an image processing system for reducing distortion of a projected image on a projection plane, and is obtained from a plurality of different locations, and is a three-dimensional projection image projected on the projection plane. 3D information acquisition means for acquiring information; a shape model determination means for determining a shape model of the projection plane by applying a shape model to the 3D information; and 3D information as a 2D coordinate on the surface of the shape model. A coordinate conversion means for projecting three-dimensional information onto the surface of the shape model, and a distortion correction by comparing the projection image converted to the two-dimensional coordinate with the original image projected as the projection image It includes a distortion correction parameter generating means for generating a parameter, wherein the coordinate conversion means, when converting the three-dimensional information to two-dimensional coordinates of the surface of the shape model, the three-dimensional information three-dimensional information Go to the distance direction of the resulting unit, characterized in that projection.

  An image processing system in which each projector can be easily installed can be provided.

It is an example of the figure explaining the outline of the image processing system of this embodiment. It is an example of a block diagram of an image processing system. It is an example of the hardware block diagram of information processing apparatus. It is an example of a functional block diagram of an information processor. It is an example of the flowchart figure which shows the whole operation | movement procedure of an image processing system. It is an example of the schematic diagram when a camera divides and images the projection image of a projection target. It is an example of the figure which illustrates acquisition of a three-dimensional coordinate typically. It is an example of the figure explaining the image of the corresponding point of the camera imaged in the overlapping imaging area, and the image of the corresponding point of the camera. FIG. 9 is an example of a diagram illustrating rotation, translation, and scale conversion of five corresponding points. It is an example of a diagram for schematically explaining alignment of three-dimensional coordinates acquired from N alignment patterns. It is an example of the flowchart figure which shows the procedure in which the conversion information preparation part calculates | requires the rotation transformation Q, the translation transformation T, and the scale transformation S by the three-dimensional coordinate acquired from the N alignment patterns. It is an example of the figure which illustrates typically the alignment of the three-dimensional coordinate of the distortion correction pattern acquired from N projection images. It is an example of the flowchart figure which shows the procedure in which the position alignment part aligns the three-dimensional coordinate acquired from the projection image of the N distortion correction patterns. It is an example of the figure which illustrates the position alignment of the point cloud of a projection target typically. It is an example of the flowchart figure which shows the procedure in which an alignment part aligns N point clouds. It is an example of the figure explaining conversion of the three-dimensional coordinate to the two-dimensional projection object surface coordinate system of the surface of a shape model. It is a schematic diagram of two methods which implement | achieve the projection to the surface of the shape model of a three-dimensional coordinate. It is an example of the figure explaining the corresponding point in a two-dimensional image and a projector image. It is an example of the figure explaining the calculation process of a distortion correction parameter. It is an example of the figure for demonstrating calculation of the projective transformation matrix H. FIG. It is an example of the figure explaining correction | amendment of a projector image. It is an example of the figure which illustrates edge blending typically. It is an example of the figure which shows acquisition of the surface information which a cylindrical projection object has typically. It is an example of the expanded view of several shape models.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
FIG. 1 is an example of a diagram illustrating an outline of an image processing system 200 according to the present embodiment. First, the overall processing of the image processing system 200 will be described with reference to FIG.

  A. In the present embodiment, a cylindrical outer wall or inner wall will be described as the projection target 11. In FIG. 1, the distortion correction pattern 32 is projected from the two projectors 13 on the inner wall. When the projectors 13 are distinguished, they are referred to as projectors 1, 2,... M (m: natural number). When the so-called ultra-single-focus projector 13 projects the distortion correction pattern 32, the image is projected from the side of the projection object 11 (downward in this example), so the rectangular distortion correction pattern 32 is a cylindrical projection. As shown in the drawing, the object 11 has a semicircular shape.

  B. The camera 12 images the distortion correction pattern 32 from almost the front of the distortion correction pattern 32 (for example, from the normal direction of the center of the distortion correction pattern 32). When distinguishing the cameras 12, they are referred to as cameras 1, 2,... M (m: natural number). The camera 12 may be mounted on the projector 13 or may be picked up by moving one camera 12. The camera 12 is a camera that can acquire distance information, such as a stereo camera.

  Since the projection target 11 is three-dimensional, the camera 12 cannot capture the entire image from one imaging location. For this reason, the projection object 11 is imaged from several imaging locations, and it is necessary to align the three-dimensional coordinates of the plurality of projection objects 11 obtained by imaging.

  C. Image data 1 and image data 2 including three-dimensional coordinates based on the position of the camera 12 at the time of imaging are obtained. Here, the three-dimensional coordinates are not obtained for each element of the distortion correction pattern 32 (in the figure, the boundary between the black region and the white region), but are obtained according to the resolution of the camera 12. For example, three-dimensional coordinates can be obtained with pixel-level resolution such as 640 × 480 and 1920 × 1080. Such dense three-dimensional coordinates may be called a point cloud (a set of three-dimensional coordinates). In FIG. 1, the point cloud 33 is acquired from the distortion correction pattern.

  D. The information processing apparatus 50 aligns the corresponding points of the two three-dimensional coordinates using the distortion correction pattern 32 (or the alignment pattern) imaged in the overlapping region. In order to align the corresponding points of the two three-dimensional coordinates, a translational transformation T, a rotational transformation Q, and a scale transformation S are calculated. By converting the three-dimensional coordinates imaged by the camera 2 by the translation conversion T, the rotation conversion Q, and the scale conversion S, the three-dimensional coordinates imaged by the camera 2 can be converted into the three-dimensional coordinates of the coordinate system of the camera 1.

  As described above, the image processing system 200 according to this embodiment can perform alignment in a relatively short time and with high accuracy by projecting the alignment pattern or the distortion correction pattern by the projector 13 during alignment. . Further, since the alignment is performed using the three-dimensional coordinates, it is not necessary to fix the projector 13 in advance with high accuracy.

  E. Next, the information processing apparatus 50 fits the shape model (cylindrical in the example of the drawing) 35 of the projection target 11 using the three-dimensional coordinates converted into the coordinate system of the camera 1. Thereby, the parameters of the shape model 35 (for example, the straight line and radius of the central axis of the cylinder) are obtained.

  F. The information processing apparatus 50 estimates the shape model 35 from the three-dimensional coordinates of the projection target 11, thereby converting the three-dimensional coordinates into the two-dimensional projection target in the coordinate system of the projection target surface (projection target surface coordinates of the two-dimensional coordinate system). It can be converted to a surface coordinate system. At the time of conversion to the projection target surface coordinate system, three-dimensional coordinates are arranged (projected) on the surface of the shape model 35. Since the three-dimensional coordinates are supposed to be on the surface of the projection target 11, the error of the distance information included in the three-dimensional coordinates can be reduced by this process, and the distortion in the projection target 11 can be reduced.

  G. The information processing apparatus 50 calculates a distortion correction parameter from the distortion of the distortion correction pattern 32. Then, the original projection image is converted so as to eliminate distortion in the projection target 11 (for example, projective conversion).

  As described above, the image processing system 200 of the present embodiment can align the three-dimensional coordinates captured by the camera 12 at each location with the common coordinates by using the acquisition of the three-dimensional coordinates. Further, by fitting the three-dimensional coordinates to the shape model 35 and arranging (projecting) the three-dimensional coordinates on the surface of the shape model 35, it is possible to reduce the error of distance information and reduce the distortion in the projection target 11. it can.

<System configuration>
FIG. 2 shows an example of a configuration diagram of the image processing system 200. The image processing system 200 has a configuration in which a plurality of projectors 13 (numbers identifying the projectors in parentheses) are connected to the information processing apparatus 50. The camera 12 (the number identifying the camera in parentheses) may be connected to the information processing apparatus 50, or the image data may be stored in a memory card or the like and read by the information processing apparatus 50 without being connected. . The connection may be wired or wireless.

  The projectors 13 are arranged at substantially equal intervals so as to surround the projection target 11. In the figure, for example, it is assumed that there are six projectors 13, but at least two projectors 13 are sufficient.

The image processing system 200 is a system that displays a single large screen image by tiling display on a screen (surface of a projection target) by combining projection images from a plurality of projectors 13.
The information processing apparatus 50 divides an image that is a source of a large screen image projected as multi-projection by the image processing system 200 into partial images corresponding to the number of projectors 13 (for example, six), and each projector 13 projects the image. Partial image data to be generated is generated. The information processing apparatus 50 is an example of an image processing apparatus.

  The partial image data is generated so that the end portions overlap in the adjacent projector 13, and the brightness of the overlapping region is adjusted (edge blending). The information processing apparatus 50 transmits the generated partial image data to the projector 13 arranged at a corresponding position.

  By simply projecting image data input from the information processing apparatus 50 by the projectors 1 and 2, projection by tiling can be performed as a multi-projection system.

  If the arrangement of each projector 13 is not strictly set, partial image data projected by each projector 13 may be shifted. For example, the deviation occurs depending on the distance of the projector 13 from the projection target 11, the angle of the optical axis, and the like. Therefore, the information processing apparatus 50 causes each projector 13 to project an alignment pattern and aligns the three-dimensional coordinates of the projection target 11 using image data captured by the camera 12. Thereby, the shift | offset | difference of a superimposition part can be suppressed. Further, since the coordinate system is common, fitting to the shape model 35 is facilitated.

  The information processing apparatus 50 is, for example, a computer such as a PC (Personal Computer), but may be any apparatus capable of projecting image data such as a tablet terminal, a smartphone, a PDA (Personal Digital Assistant), or an electronic book terminal. Moreover, what is necessary is just to exist on a network, such as a server. Furthermore, the camera 12 illustrated as a separate body in FIG. 1 may have the function of the information processing apparatus 50 of the present embodiment. Further, two or more information processing devices 50 may be arranged, and for example, a device that performs distortion correction and a device that transmits partial image data to the projector 13 may exist separately.

  The camera 12 may be mounted on the projector 13, but may be disposed in front of the projection image so that the correction can be performed correctly when the projection image is viewed from the user's viewpoint. Further, the camera 12 may move and image every time the distortion correction pattern 32 or the alignment pattern is imaged (that is, there may be only one camera 12). In the present embodiment, it is assumed that there are as many cameras 12 as the number of projectors 13 for ease of explanation.

  Further, the camera 12 acquires three-dimensional coordinates (such image data may be referred to as a distance image) together with image data of the distortion correction pattern 32 and the alignment pattern by capturing the projection image. As a method for acquiring three-dimensional coordinates, for example, a method for measuring the time (TOF: Time Of Fright) until irradiation with reflected light and the reflected light arrives, and a stereo camera method using parallax are known. . Kinect (registered trademark) is known as the camera 12 of the former method.

  The projector 13 includes a DLP (Digital Light Processing) projector, a liquid crystal projector, and the like, but the projection method is not limited in this embodiment. Further, a projector that projects from a distance may be used instead of the ultra-short focus projector.

  FIG. 3 shows an example of a hardware configuration diagram of the information processing apparatus 50. The information processing apparatus 50 includes a CPU 101, a RAM 102, a ROM 103, a storage medium mounting unit 104, a communication device 105, an input device 106, a drawing control unit 107, and an HDD 108 that are connected to each other via a bus. The CPU 101 reads out an OS (Operating System) and a program 111 from the HDD 108 and executes them to provide various functions and generate a tone conversion parameter.

  The RAM 102 is a working memory (main storage memory) that temporarily stores data necessary when the CPU 101 executes the program 111, and the ROM 103 is a program (static input data) for starting up a BIOS (Basic Input Output System) or OS. Is remembered.

  A storage medium 110 is detachably attached to the storage medium mounting unit 104, and data recorded in the storage medium 110 is read and stored in the HDD 108. Further, the storage medium mounting unit 104 can also write data stored in the HDD 108 into the storage medium 110. The storage medium 110 is, for example, a USD memory or an SD card. The program 111 is distributed by being downloaded from a state stored in the storage medium 110 or a server (not shown).

  The input device 106 is a keyboard, a mouse, a trackball, or the like, and receives various user operation instructions to the information processing device 50. The input device 106 may be a touch panel or a voice input device.

  The HDD 108 may be a nonvolatile memory such as an SSD, and stores various data such as an OS and application software.

  The communication device 105 is a NIC (Network Interface Card) for connecting to a network, for example, an Ethernet (registered trademark) card.

  The drawing control unit 107 interprets a drawing command written in the graphic memory by the CPU 101 executing the program 111, generates a screen, and draws it on the display 109. In FIG. 3, the display 109 is externally attached to the information processing apparatus 50, but the information processing apparatus 50 may incorporate the display 109.

  FIG. 4 shows an example of a functional block diagram of the information processing apparatus 50. The information processing apparatus 50 includes a projected image three-dimensional position acquisition unit 21, an alignment unit 22, a model calculation unit 23, a coordinate conversion unit 24, a distortion correction parameter calculation unit 25, and an image deformation unit 26. These are realized by the CPU 101 executing the program 111 and cooperating with the hardware shown in FIG.

  The information processing apparatus 50 also includes an image data storage unit 27 that stores projected image data, a distortion correction pattern 32, and an alignment pattern, and a distortion correction parameter storage unit 28 that stores distortion correction parameters. doing. These may be in a location where the information processing apparatus 50 can access (for example, on a network such as a LAN or the Internet).

  The projection image three-dimensional position acquisition unit 21 acquires the three-dimensional coordinates of the projection image on the projection target. Specifically, a distance image (including distance information for each pixel together with luminance information or color pixel values) captured by the camera 12 is acquired via a cable or a storage medium. In the present embodiment, image data including distance information is simply referred to as three-dimensional coordinates.

  The alignment unit 22 includes a conversion information creation unit 29. The conversion information creation unit 29 calculates conversion information based on the three-dimensional coordinates of feature points of a plurality of alignment patterns (or distortion correction patterns 33). Then, the alignment unit 22 aligns the three-dimensional coordinate positions of the distortion correction pattern 32 or the point cloud 33 with this conversion information. Specifically, translation conversion T, rotation conversion Q, and scale conversion S are obtained as conversion information between the three-dimensional coordinates of the alignment pattern projected by the adjacent projector 13. Then, the three-dimensional coordinates or the point cloud 33 of the feature points obtained by imaging the distortion correction pattern 32 by the translational transformation T, the rotational transformation Q, and the scale transformation S are coordinate systems common to the cameras 1 to m. Convert to 3D coordinates.

  The model calculation unit 23 fits the aligned three-dimensional coordinate group of the distortion correction pattern 32 or the point cloud 33 to the shape model 35 of the projection target 11. Thereby, the linear equation and radius of the axis of the shape model 35 (for example, a cylinder) are obtained.

  The coordinate conversion unit 24 converts the three-dimensional coordinates of the feature points of the distortion correction pattern 32 or the point cloud 33 into the coordinate system of the shape model 35 of the projection target 11. Since this coordinate system is, for example, a coordinate system of the surface of the cylindrical shape model 35, the three-dimensional coordinates of the feature points of the distortion correction pattern 32 or the three-dimensional coordinates of the point cloud 33 are the two-dimensional projection target surface coordinate system. It is expressed by

  The distortion correction parameter calculation unit 25 includes an original image of the distortion correction pattern 32 projected by the projector 13 (hereinafter referred to as a projector image) and an image of the distortion correction pattern 32 expressed in a two-dimensional projection target surface coordinate system ( Hereinafter, a projective transformation matrix H is calculated from the two-dimensional image). A distortion correction parameter is obtained from the projective transformation matrix H.

  The image deformation unit 26 deforms the projector image projected by the projector 13 using the distortion correction parameter and creates a corrected image. Thereby, a projection image with little distortion is projected onto the cylindrical projection target 11.

<Overall operation procedure>
FIG. 5 shows an example of a flowchart showing an overall operation procedure of the image processing system 200.
S10: The projection image three-dimensional position acquisition unit 21 initializes a variable n (sets n to 1). n is a variable to be compared with the number N of projectors 13.
S20: The projection image three-dimensional position acquisition unit 21 acquires the three-dimensional coordinates of the projection image. This three-dimensional coordinate may be any one of the three-dimensional coordinates of the feature point of the alignment pattern projected by each projector 13, the feature point of the distortion correction pattern 32, or the point cloud 33. When alignment is performed using the distortion correction pattern 32, the three-dimensional coordinates of the alignment pattern are not necessary.
S30: The projected image three-dimensional position acquisition unit 21 determines whether n matches N. That is, it is determined whether or not three-dimensional coordinates have been acquired from all the projectors 13.
S40: When the determination in step S30 is No, the projected image three-dimensional position acquisition unit 21 increases the variable n by one.
S50: If the determination in step S30 is Yes, the alignment unit 22 aligns the N three-dimensional coordinates captured by each camera 12 with one camera coordinate. In other words, a plurality of three-dimensional coordinates captured respectively are represented by one camera coordinate.
S60: The model calculation unit 23 fits the three-dimensional coordinates in step S50 to the cylindrical shape model 35.
S70: The coordinate conversion unit 24 converts the three-dimensional coordinates in step S50 into the two-dimensional projection target surface coordinates on the surface of the cylindrical shape model 35.
S80: The distortion correction parameter calculation unit 25 calculates, for example, a projection transformation matrix H from the projector image and the two-dimensional image expressed by the two-dimensional projection target surface coordinates, and calculates a distortion correction parameter from the projection transformation matrix H. This is stored in the distortion correction parameter storage unit 28.
S90: The image deforming unit 26 deforms the image projected by the projector 13 using the distortion correction parameter.

  Below, step S20 and S50-S90 are demonstrated in detail for every step.

<S20 Acquisition of 3D coordinates of projected image>
FIG. 6 is an example of a schematic diagram when the camera 12 divides and captures the projection image of the projection target 11. In FIG. 6, the camera 1 and the camera 2 divide and image the projection images on the concave and convex surfaces of the cylindrical projection object 11. In the case where all the projected images cannot be captured by imaging from one place, in the present embodiment, an overlapping imaging area 36 is provided, and the images are divided and imaged from different locations. It should be noted that the procedure up to the deformation of the image by the distortion correction parameter is the same except that the distortion is different between the concave and convex surfaces.

  FIG. 7 is an example of a diagram schematically illustrating acquisition of three-dimensional coordinates. In FIG. 7A, the projector 1 projects the alignment pattern 31. Each of the cameras 1 and 2 images an alignment pattern. Thus, only the left projector 13 among the adjacent projectors 13 projects the alignment pattern 31. The cameras 1 and 2 can acquire distance information in units of pixels, but in the alignment pattern 31, for example, three-dimensional coordinates of feature points (circles) are acquired.

  In FIGS. 7B and 7C, the projectors 1 and 2 project the distortion correction pattern 32, respectively. As the distortion correction pattern 32, a circular pattern is projected in FIG. 7B, and a lattice pattern is projected in FIG. 7C. Each of the cameras 1 and 2 images the distortion correction pattern 32. The cameras 1 and 2 can acquire distance information in units of pixels. However, in the circular pattern distortion correction pattern 32, three-dimensional coordinates of feature points (for example, circles) are acquired, and the distortion correction pattern 32 of the lattice pattern is acquired. Then, the three-dimensional coordinates of the feature point (for example, the vertex of the black region) are acquired.

  When the projected image three-dimensional position acquisition unit 21 extracts feature points from the images of the alignment pattern 31 and the distortion correction pattern 32, for example, the following processing is performed. In the case of a circular pattern, for example, the center of gravity of black pixels that are binarized and continuous is detected as a feature point. In the case of a lattice-shaped distortion correction pattern, for example, a corner detection method such as a Harris method is used to detect the vertex of the lattice as a feature point. In addition, intersections of straight lines extracted by detecting edges may be used as feature points.

  In addition to the known images such as the alignment pattern 31 and the distortion correction pattern 32, an unknown image can be used as the alignment image. An unknown image is projected and, for example, a SIFT (Scale Invariant Feature Transform) feature quantity is detected. The SIFT feature amount is a feature amount calculated from a feature point having a large luminance change in the image, a luminance gradient around the feature point, and a luminance direction. Since the same SIFT feature value is detected even in an unknown image, a corresponding feature point is obtained. As described above, if the alignment unit 22 described below can correctly determine the correspondence between the feature points of the projected images captured from a plurality of locations, an arbitrary image can be used for alignment.

  Hereinafter, for the sake of explanation, the description will be made assuming that the alignment is performed by the alignment pattern 31. Of course, the distortion correction pattern 32 is also imaged for distortion correction, and a point cloud 33 is also obtained at that time.

<S50 alignment>
The alignment unit 22 associates the feature points of the alignment pattern imaged at a plurality of locations, and aligns the coordinate systems of the cameras 1 and 2 using these feature points. That is, the three-dimensional coordinates captured by the camera 2 are converted into coordinates in the coordinate system of the camera 1. The corresponding feature points of the two alignment patterns imaged at different places are referred to as corresponding points.

  First, the alignment unit 22 projects the alignment pattern 31 only on the projector 1 onto the overlapping imaging region 36 of the cameras 1 and 2. By acquiring the three-dimensional coordinates of the feature point group that is robust to changes in the viewpoint, it is possible to stably acquire corresponding points corresponding to adjacent imaging locations.

  FIG. 8A is an example for explaining the image of the corresponding point of the camera 1 and the image of the corresponding point of the camera 2 captured in the overlapping imaging region 36. In FIG. 8A, a circular pattern is imaged as the alignment pattern. Although this circular pattern is projected separately from the distortion correction pattern 32, the distortion correction pattern 32 can be used as the alignment pattern 31 as described above.

  The circular pattern 1 is captured in the overlapping imaging area 36 captured by the camera 1, and the circular pattern 2 is captured in the overlapping imaging area 36 captured by the camera 2. A circle pattern 1 is projected by the projector 1. As shown in FIG. 8B, in the image 1 captured by the camera 1, the circular pattern 1 is captured at the right end, and in the image 2 captured by the camera 2, the circular pattern 2 is captured at the left end.

  Since the circular patterns 1 and 2 are projected at the same place with different shooting locations, assuming that the cameras 1 and 2 are taken from the same position, the circular patterns 1 and 2 must be taken at the same position in the image. I must. In order to capture images at the same position in the image, the camera 2 may be moved to the position of the camera 1, but the amount of movement is different between the locations where the circular patterns 1 and 2 are captured in the images 1 and 2. It is none other than. Therefore, the three-dimensional coordinates acquired by the camera 2 are positioned in the coordinate system of the camera 1 by aligning the images 1 and 2 so that the positions of the circular pattern 1 in the image 1 and the circular pattern 2 in the image 2 match. Can be combined.

  In addition, an index is given to the corresponding point (each circle) detected as a feature point from the images 1 and 2. In image 1, A1-1 to A1-5 are given to the circle in the first circle pattern from the right end, and A2-1 to A2-5 are given to the circle in the second circle pattern. In image 2, B1-1 to B1-5 are given to the circle of the first circular pattern from the left end, and B2-1 to B2-5 are given to the circle of the second circular pattern. The alignment unit 22 treats A1-1 to A1-5 and A2-1 to A2-5, and B1-1 to B1-5 and B2-1 to B2-5 as corresponding points.

  For alignment, the three-dimensional coordinates of corresponding points are used. That is, the three-dimensional coordinates of each circle of the circle pattern 1 and the three-dimensional coordinates of each circle of the circle pattern 2 are used. The conversion for alignment is divided into a translational transformation T, a rotational transformation Q, and a scale transformation S. In the present embodiment, the translational transformation T, the rotational transformation Q, and the scale transformation S are calculated by a method called Absolute Orientation.

Calculation of translational transformation T, rotational transformation Q, and scale transformation S will be described with reference to FIG. FIG. 9 is an example of a diagram illustrating rotation, translation, and scale conversion of five corresponding points. _Assume that the corresponding points of the image 2 are P _{11 to} P ₁₅ , and the corresponding points of the image 1 are P _{r1 to} P _r5 . The corresponding point group I is obtained by rotating P _{11 to} P ₁₅ , the corresponding point group II is obtained by scaling the corresponding point group I, and the corresponding point group III is obtained by translating the corresponding point group II. Shall be obtained. Therefore, the rotation transformation R, the translation transformation T, and the scale transformation S in which the corresponding point group III is transformed into P _{r1 to} P _r5 are obtained.

First, the scale conversion S is obtained as follows.
1. The centroid G1 of the corresponding points P _{11 to} P _{15 and} the centroid G2 of the corresponding points P _{r1 to} P _r5 are obtained and converted to coordinates having the centroid as the origin.
2. Square sum S1 of distance from the center of gravity G1 of the corresponding point _P l1 _{~P l5,} obtains the square sum S2, the distance from the center of gravity G2 of the corresponding point _P r1 _{to P r5.}
3. Let the ratio S2 / S1 of the two sums of squares be the scale conversion S. That is, the scale conversion S is obtained independently of the rotation conversion R.

As for the rotation transformation R, the sum of the inner products of the corresponding points P _{r1 to} P _r5 when the corresponding points P _{l1 to} P ₁₅ having the center of gravity G1 as the origin are rotated by a certain angle around a certain axis is maximized. Required as rotation.

The relative coordinates of the corresponding points P _{11 to} P _{15 from} the centroid G1 are rl _i (i = 1 to 5), the relative coordinates of the corresponding points P _{r1 to} P _{r5 from} the centroid G2 are rr _i (i = 1 to 5), And

Ｍの要素から４×４対象行列Ｎを求める。

A 4 × 4 target matrix N is obtained from the elements of M.

この４×４対象行列Ｎの最大固有値に対応する単位固有ベクトルを求めると、最適な回転を表す単位４元数が得られる。４元数とは「a + i b + j c + k d」のように実部aと虚部ｂ、ｃ、ｄにより表される（i、j、kは虚数）。i、j、kは三次元のそれぞれの軸方向の回転を表している。

When a unit eigenvector corresponding to the maximum eigenvalue of the 4 × 4 target matrix N is obtained, a unit quaternion representing an optimum rotation is obtained. A quaternion is represented by a real part a and imaginary parts b, c, and d, such as “a + ib + jc + kd” (i, j, k are imaginary numbers). i, j, and k represent the three-dimensional axial rotations.

The unit quaternion is a quaternion in which the square root of the square sum of the real part and the imaginary part is 1. The unit quaternion is represented by Σ = (cos θ / 2; sin (θ / 2) p) (p is a three-dimensional vector of an imaginary part). When Σ * qΣ is calculated by the unit quaternion (Σ * is a conjugate of Σ), the point represented by the vector q can be rotated θ around the rotation axis p. Therefore, if the vector q is a coordinate vector from the centroid G1 of the corresponding point, the corresponding points P _{11 to} P ₁₅ can be rotated.

Finally, translational transform T rotates the corresponding point _P l1 _{to P l5} in rotational transform R, to those scaled by scaling S, is determined as the difference of the corresponding point _P r1 _{to P r5.}

  For the specific method of Absolute Orientation, refer to Closed-form solution of absolute orientation using unit quaternions by Berthold K. P. Horn.

  FIG. 10 is an example of a diagram for schematically explaining the alignment of three-dimensional coordinates acquired from N alignment patterns. As shown in FIG. 10A, the cameras 1 to 2 -N respectively image the alignment pattern 31 in the overlapping imaging region 36. A dotted square indicates an imaging range. As described above, corresponding points between the alignment pattern 31 captured by the camera 2 and the alignment pattern 31 captured by the camera 1 are obtained. From these corresponding points, rotation conversion Q1, translation conversion T1, and scale conversion S1 are obtained.

  Next, as illustrated in FIG. 10B, the cameras 2 and 3 respectively capture the alignment pattern 31 in the overlapping imaging region 36. Corresponding points between the alignment pattern 31 captured by the camera 3 and the alignment pattern 31 captured by the camera 2 are obtained. From these corresponding points, rotation conversion Q2, translation conversion T2, and scale conversion S2 are obtained.

  Finally, the cameras N-1 and N respectively image the alignment pattern 31 in the overlapping imaging region 36. Similarly, rotation conversion Q (N-1), translational conversion T (N-1), and scale conversion S (N-1) are obtained from the corresponding points.

  When rotation transformation Q, translation transformation T, and scale transformation S are obtained between adjacent imaging locations, as shown in FIG. 10C, an arbitrary point cloud 33 imaged at each imaging location is converted to the Nth point cloud. Positioning can be performed in order from the imaging location.

FIG. 11 is an example of a flowchart illustrating a procedure in which the conversion information creation unit 29 obtains the rotation transformation Q, translation transformation T, and scale transformation S from the three-dimensional coordinates acquired from the N alignment patterns 31.
S101: The conversion information creation unit 29 initializes a variable n to 1.
S102: The projected image three-dimensional position acquisition unit 21 acquires three-dimensional coordinates from the feature points of the alignment pattern 31 captured from the overlapping imaging region 36 by the cameras n and n + 1.
S103: The conversion information creation unit 29 obtains scale conversion S (n).
S104: The conversion information creation part 29 calculates | requires rotation conversion R (n).
S105: The conversion information creation unit 29 obtains a translation conversion T (n).
S106: The conversion information creation unit 29 determines whether n matches N-1.
S107: If the determination in step S106 is No, the conversion information creation unit 29 increases n by one.

  When the determination in step S106 is Yes, rotation transformation Q, translation transformation T, and scale transformation S at all adjacent imaging positions are obtained. As a result, distortion correction using the distortion correction pattern 32 becomes possible. Therefore, next, each projector 13 projects the distortion correction pattern 32 and each camera captures an image. Then, the alignment unit 22 aligns the feature points of the distortion correction pattern 32.

  FIG. 12 is an example of a diagram schematically illustrating the alignment of the three-dimensional coordinates of the feature points of the distortion correction pattern 32 acquired from N projection images. As shown in FIG. 12A, the camera 1 images a distortion correction pattern 32 projected by the projector 1. As illustrated in FIG. 12B, the camera 2 captures the distortion correction pattern 32 projected by the projector 2.

  Finally, when the camera N captures the distortion correction pattern 32 projected by the projector N, as shown in FIG. 12C, the tertiary feature points of the distortion correction pattern 32 captured by the cameras 1 to N are obtained. The original coordinates are aligned.

FIG. 13 is an example of a flowchart illustrating a procedure in which the alignment unit 22 performs alignment of the three-dimensional coordinates acquired from the projection images of the N distortion correction patterns 32.
S201: The alignment unit 22 initializes a variable n to 1.
S202: The projected image three-dimensional position acquisition unit 21 acquires the three-dimensional coordinates of the feature points of the distortion correction pattern 32 captured by the camera n.
S203: The alignment unit 22 determines whether n = N.
S204: When the determination in step S203 is No, the alignment unit 22 increases n by one.
S205: If the determination in step S203 is Yes, the alignment unit 22 initializes n to N.
S206: The alignment unit 22 uses the translation transformation T (n-1), the rotation transformation Q (n-1), and the scale transformation S (n-1) for the three-dimensional coordinates of the feature points of the distortion correction pattern 32. To convert.

Thereby, the three-dimensional coordinates of the feature points of the distortion correction pattern 32 imaged by the camera N are converted into the coordinate system of the camera N-1. In the next cycle, the three-dimensional coordinates of the feature points of the distortion correction pattern 32 imaged by the camera N-1 are converted into the coordinate system of the camera N-2. Hereinafter, the three-dimensional coordinates of the feature points of the distortion correction pattern 32 are sequentially converted up to the coordinate system of the camera 1.
Camera N → Camera N-1
Camera N-1 → Camera N-2
Camera N-2 → Camera N-3
:
Camera 2 → Camera 1
Therefore, finally, all the three-dimensional coordinates of the feature points of the distortion correction pattern 32 imaged by the cameras N to 2 are converted into the coordinate system of the camera 1.
S207: The alignment unit 22 determines whether n = 2. That is, it is determined whether or not the conversion of the camera 1 to the coordinate system has been completed.
S208: When the determination in step S207 is No, the alignment unit 22 decreases n by one.

  If the determination in step S207 is Yes, the three-dimensional coordinates of the cameras 2 to N are aligned with the coordinate system of the camera 1.

  By the way, in this embodiment, model fitting is performed in the next step S60. However, in order to perform model fitting, the point interval is too large at the feature points of the distortion correction pattern 32, and there is a possibility that an error may occur in the model fitting. Therefore, as described below, not the feature points of the distortion correction pattern 32 but the point cloud 33 may be aligned. When distance information is acquired by the TOF method like Kinect (registered trademark), distance information can be obtained for each pixel by imaging the projection object 11 as it is. In addition, since the surface of the projection object 11 has a uniform brightness, when the accuracy of the distance information is reduced in the stereo camera, an image for model fitting (an image with feature points denser than the distortion correction pattern 32). May be projected.

  FIG. 14 is an example of a diagram schematically illustrating the alignment of the point cloud 33 of the projection target 11. The processing procedure is the same as in FIG.

  As shown in FIG. 14A, the camera 1 images the projection object 11-1. For example, the projector 1 may project a content image, or the projector 1 may project nothing but the design of the projection target 11 as it is. As shown in FIG. 14B, the camera 2 images the projection target 11-2.

  Finally, when the camera N captures the projection target 11-N, as shown in FIG. 14C, the alignment unit 22 aligns the point cloud 33 of the projection target 11 captured by the cameras 1 to N. Done.

FIG. 15 is an example of a flowchart illustrating a procedure in which the alignment unit 22 aligns the three-dimensional coordinates acquired from the N point clouds 33. The processing procedure is the same as in FIG.
S201: The alignment unit 22 initializes a variable n to 1.
S202: The projection image three-dimensional position acquisition unit 21 acquires the point cloud 33 from the projection target 11 captured by the camera n.
S203: The alignment unit 22 determines whether n = N.
S204: When the determination in step S203 is No, the alignment unit 22 increases n by one.
S205: If the determination in step S203 is Yes, the alignment unit 22 initializes n to N.
S206: The alignment unit 22 converts the point cloud 33 of the projection target 11 using translational transformation T (n-1), rotational transformation Q (n-1), and scale transformation S (n-1). Thereby, the point cloud 33 of the projection target 11 imaged by the camera N is converted into the coordinate system of the camera N-1. In the next cycle, the point cloud 33 of the projection object 11 captured by the camera N-1 is converted into the coordinate system of the camera N-2. Therefore, finally, all the point clouds 33 of the projection object 11 captured by the cameras N to 2 are converted into the coordinate system of the camera 1.
S207: The alignment unit 22 determines whether n = 2. That is, it is determined whether or not the conversion of the camera 1 to the coordinate system has been completed.
S208: When the determination in step S207 is No, the alignment unit 22 decreases n by one.

  By acquiring dense three-dimensional coordinates (point cloud 33) in this way, the approximation accuracy of model fitting can be improved.

<S60 model calculation>
The model calculation unit 23 fits the three-dimensional coordinates of the feature points of the distortion correction pattern 32 or the point cloud 33 to the shape model 35 of the projection target 11, and estimates the optimal model parameters of the shape model 35.

  The model calculation unit 23 may perform removal of the three-dimensional coordinates in advance as preprocessing for model fitting. For example, assuming that the three-dimensional coordinates are uniformly distributed, the distance between the points is obtained, and the points that are more than the threshold from the average distance are removed from the model fitting target.

  The model calculation unit 23 applies a predetermined three-dimensional shape model 35 to the aligned three-dimensional coordinates to calculate a projection model representing the projection target 11. More specifically, the model parameter of the three-dimensional shape model 35 that matches the set of three-dimensional coordinates is calculated. Here, in this embodiment, since the shape model 35 of the projection target 11 is a cylindrical model, in the model fitting calculation, at least one parameter of the linear equation of the axis of the cylindrical model and the radius is assumed to be indefinite, The most probable parameter or combination of parameters with the smallest error is estimated.

  All the parameters of the cylindrical model may be estimated by calculation, but are not particularly limited. For example, when the size (radius) of the cylinder of the projection target 11 is known or when the direction and position of the axis can be regarded as known, the input value or default value from the user of one or more parameters is a constraint condition. The remaining parameters can be calculated as indefinite.

  Even if only a part of the projection target 11 can be measured by fitting the shape model 35 prepared in advance or an object other than the projection target is reflected, it is also included in the three-dimensional measurement. Even when there is a lot of noise, the three-dimensional shape of the projection object 11 can be suitably estimated.

  The method of model fitting is not particularly limited. For example, other three-dimensional shape models (any three-dimensional model having a smooth surface such as a sphere, a cube, a plane, an elliptical curved surface, etc.) may be used, and several different shape models 35 are prepared in advance. The most probable model among the plurality of shape models 35 may be determined.

<S70 coordinate transformation>
The coordinate conversion unit 24 converts the three-dimensional coordinates of the feature points of the distortion correction pattern 32 into a two-dimensional projection target surface coordinate system on the surface of the shape model 35 using the estimated model parameters. The projection target surface coordinate system represents coordinates with a height h in the axial direction of the cylinder and an angle θ with reference to a reference axis perpendicular to the axis.

  By converting the three-dimensional coordinates of the feature points of the distortion correction pattern 32 to the two-dimensional projection target surface coordinate system, it is possible to perform projection with less distortion on the surface of the projection target 11 in multi-projection.

  FIG. 16 is an example of a diagram illustrating conversion of three-dimensional coordinates into a two-dimensional projection target surface coordinate system on the surface of the shape model 35. As shown, a cylindrical coordinate system is defined. Arbitrary direction of the bottom surface of the cylinder is the x-axis, the direction perpendicular to the x-axis is the y-axis, the direction perpendicular to the intersection of the x-axis and the y-axis is the z-axis, the z-axis direction is h, Let θ be the angle between the perpendicular and the x-axis.

The meanings of the symbols in FIG. 16 are as follows.
Vector s i: coordinate vector of feature points (vector connecting the origin O of the cylindrical coordinate system and the three-dimensional coordinates; i is the identification number of the three-dimensional coordinates)
Vector z _M : Unit vector vector in the z-axis direction x _M : Unit vector vector in the x-axis direction y _M : Unit vector in the y-axis direction From these, the three-dimensional coordinates (x, y, z) on the surface of the shape model 35 Can be converted to cylindrical coordinates (h, θ) by

ただし、本手法は円筒形状に限定させるものではなく、角柱や角錐など対象物側面の座標が柱座標（h, θ）により一意に決定できる形状であれば適用可能である。

However, this method is not limited to a cylindrical shape, and can be applied as long as the coordinates of the side of the object such as a prism or a pyramid can be uniquely determined by the column coordinates (h, θ).

From the above equation (5), h is equal to the z coordinate of the feature point. Note that the denominator of Equation (6) divides a vector connecting the z-axis and the feature point by the distance from the z-axis to the feature point for normalization. If the cylindrical coordinates (h, θ) can be calculated, the three-dimensional coordinates of the feature points can be arranged at a distance of √ (x ² + y ² ) from the z axis.

  By the way, even when the three-dimensional coordinates are converted into the two-dimensional projection target surface coordinate system on the surface of the shape model 35 as shown in FIG. 16, the three-dimensional coordinates are not always converted to the surface of the shape model 35. This is because there is an error in the three-dimensional coordinates, so that the three-dimensional coordinates may actually float or exist inside the surface of the shape model 35.

  Therefore, it is preferable to project the three-dimensional coordinates onto the surface of the shape model 35 as shown in FIG. FIG. 17 is a schematic diagram of two methods for realizing the projection of the three-dimensional coordinate shape model 35 onto the surface.

  FIG. 17A is projected by moving the three-dimensional coordinates closest to the surface of the shape model 35 (in the normal direction). Since this method can be used without changing the equations (5) and (6), there is an advantage that the implementation is simple. This is because the column coordinates (h, θ) of the feature points are obtained by the equations (5) and (6), but θ is obtained by normalization (the absolute values of x and y are not included). This is because when the three-dimensional coordinates are converted into the two-dimensional projection target surface coordinate system on the surface of the shape model 35, the three-dimensional coordinates may be converted into the position of the radius r.

  On the other hand, in FIG. 17B, projection is performed by correcting the coordinates of feature points in the camera direction in consideration of errors in the depth direction (distance direction) of the three-dimensional coordinates. For projection, the three-dimensional coordinates are changed so that the surface of the shape model 35 is projected onto the surface of the shape model 35 before the transformation of the surface of the shape model 35 to the two-dimensional projection target surface coordinate system according to the equations (5) and (6). Keep it.

式（７）（８）において記号は以下の意味である。
ベクトルｓi：円筒表面に射影された特徴点の座標ベクトル（円柱座標系の原点Ｏと円筒表面の三次元座標を結ぶベクトル）
ベクトルｓｉ´：特徴点の座標ベクトル（円柱座標系の原点Ｏと三次元座標を結ぶベクトル）
ベクトルＶcam：カメラ１の座標ベクトル（円柱座標系の原点Ｏとカメラ１を結ぶベクトル）
ベクトルｚ_M：ｚ軸方向の単位ベクトル
ベクトルｘ_M：ｘ軸方向の単位ベクトル
ベクトルｙ_M：ｙ軸方向の単位ベクトル
ベクトル「ｓｉ´−Ｖcam」：カメラ１と三次元座標を結ぶベクトル
ベクトル「ｓｉ´−ｓｉ」：円筒表面に射影された三次元座標と、射影前の三次元座標とを結ぶベクトル
ｔ：ベクトル「ｓｉ´−Ｖcam」の縮尺率（スカラー）
ここで、未知であるベクトルｓi、スカラーｔがあり、それ以外は既知であり、方程式が２つあるので、連立方程式を解くことができる。式（７）によりベクトルｓｉ´がベクトルｓiに変更されている。 For this projection, the following simultaneous equations are solved. FIG. 17C is a reference diagram of vectors of simultaneous equations.

In the formulas (7) and (8), the symbols have the following meanings.
Vector si: coordinate vector of feature points projected onto the cylindrical surface (vector connecting the origin O of the cylindrical coordinate system and the three-dimensional coordinates of the cylindrical surface)
Vector si ': feature point coordinate vector (vector connecting the origin O of the cylindrical coordinate system and the three-dimensional coordinates)
Vector Vcam: Coordinate vector of camera 1 (vector connecting origin O of cylindrical coordinate system and camera 1)
Vector z _M : Unit vector vector in the z-axis direction x _M : Unit vector vector in the x-axis direction y _M : Unit vector vector in the y-axis direction “si′−Vcam”: Vector vector “si” connecting the camera 1 and the three-dimensional coordinates '-Si': vector connecting the three-dimensional coordinates projected onto the cylindrical surface and the three-dimensional coordinates before the projection t: scale ratio (scalar) of the vector "si'-Vcam"
Here, there are an unknown vector s i and a scalar t, and the others are known and there are two equations, so that simultaneous equations can be solved. The vector si ′ is changed to the vector si by the equation (7).

  After obtaining the vector s i and the scalar t, by obtaining the column coordinates (h, θ) of the feature points using the equations (5) and (6), errors in the depth direction (distance direction) of the three-dimensional coordinates can be reduced.

  As shown in the figure, the scalar t is the scale ratio of the distance between the camera 1 and the feature point. Therefore, projection for correcting the coordinates of the feature points in the distance direction from the camera 1 can be performed. Thereby, it is possible to avoid an error in the depth direction of the camera from being an error in the two-dimensional projection target surface coordinates on the shape model surface.

<S80 Distortion Correction Parameter Calculation>
With the above processing, the distortion correction pattern 32 is represented in the two-dimensional projection target surface coordinate system on the surface of the shape model 35. However, since the distortion correction pattern 32 is projected onto the surface of the cylinder from a short distance, it is distorted as shown in FIG. For this reason, the distortion correction parameter calculation unit 25 compares the feature points (two-dimensional image) of the distortion correction pattern 32 expressed in the two-dimensional projection target surface coordinate system on the surface of the shape model 35 with the projector image. Then, the projective transformation matrix H is calculated.

  FIG. 18 is an example of a diagram for explaining corresponding points in the two-dimensional image 42 and the projector image 41. Since the projector image 41 is image data before projection and the circle is uniformly distributed (although it is not necessarily uniform), the two-dimensional image 42 is caused by the fact that the projection surface is cylindrical, etc. The position of the circle is distorted.

  The distortion correction parameter calculation unit 25 cuts out the largest rectangular area within the range of the two-dimensional image 42. Then, the feature points are detected in the same manner as when corresponding points are extracted for the alignment of the three-dimensional coordinates in step S50. In the case of a circular pattern, for example, the center of gravity of continuous black pixels is detected as a feature point.

  The distortion correction parameter calculation unit 25 assigns indexes to the circles detected from the two-dimensional image 42 and the projector image 41, for example, in order from the upper left. Among circles in the rectangular range, circles with the same index are a set of corresponding points corresponding to each other. Note that the feature points may be associated with each other by making a pair of circles of the two-dimensional image 42 and the projector image 41 one by one.

  FIG. 19 (a) is an example of a diagram illustrating a distortion correction parameter calculation process. The two-dimensional image is distorted corresponding to the cylindrical shape model 35 as shown in FIG. The distortion correction parameter calculation unit 25 calculates a distortion correction parameter (projection transformation matrix H) for distortion correction for correcting the projector image so that the two-dimensional image 42 becomes a projection image 44 after correction at the time of projection.

  As illustrated in FIG. 19B, the image deformation unit 26 corrects the projector image 41 using the calculated projective transformation matrix H, and generates a corrected image 43.

  As shown in FIG. 19C, the projector 13 projects the corrected image 43 corrected with the projective transformation matrix H, so that the corrected projected image 44 with reduced distortion can be projected. That is, the projector projects the correction image 43 that cancels the deformation corresponding to the shape of the projection target 11.

<< Calculation of projective transformation matrix H >>
FIG. 20 is an example of a diagram for explaining the calculation of the projective transformation matrix H. Since the two-dimensional image 42 is expressed in a cylindrical coordinate system, the processing is facilitated by converting it to the same xy coordinate system as the projector image 41. When the number of pixels of the projector image 41 does not match the number of pixels of the two-dimensional image 42, the number of pixels of the two-dimensional image 42 is matched with the number of pixels of the projector image 41, and both are adjusted to the same size.

The distortion correction parameter calculation unit 25 performs association for each region based on the correspondence of the feature points acquired in FIG. This region 45 is a quadrangle surrounded by four circles. The correspondence between one area of the projector image 41 and one area of the two-dimensional image 42 is described by projective transformation. Assuming that the point of the projector image 41 is m _p = (x _p , y _p ) and the point of the two-dimensional image is m _c (x _c , y _c ), the point m _p uses m _c by projective transformation as follows: Can be represented.

ｈ１〜ｈ８は未知の射影変換パラメータである。また、ｍ_ｐとｍ_ｃは同次座標を用いて次のように表現できる。

h1 to h8 are unknown projective transformation parameters. Also, m _p and m _c can be expressed as follows by using the homogeneous coordinates.

Ｈが射影変換行列である。

H is a projective transformation matrix.

  Further, the following constraint is obtained from the correspondence between the four circles of the projector image and the four circles of the two-dimensional image.

この式により、８つの未知パラメータに対して２つの拘束が得られる。したがって、１つの領域４５の頂点である４つの円の座標から射影変換行列Ｈの全ての要素を決定できる。 ＜Ｓ９０ 画像補正＞
画像変形部２６は、歪み補正パラメータを用いて二次元画像を変形する。
図２１は、プロジェクター画像４１の補正について説明する図の一例である。プロジェクター画像４１の補正は、プロジェクター１３が投影するプロジェクター画像を予め、歪みとは逆方向に変形させることで行う。

This equation gives two constraints on the eight unknown parameters. Therefore, all elements of the projective transformation matrix H can be determined from the coordinates of the four circles that are the vertices of one region 45. <S90 Image Correction>
The image deforming unit 26 deforms the two-dimensional image using the distortion correction parameter.
FIG. 21 is an example of a diagram for explaining the correction of the projector image 41. The projector image 41 is corrected by previously deforming the projector image projected by the projector 13 in the direction opposite to the distortion.

  First, the image transformation unit 26 determines a rectangular projection area in the two-dimensional image. The projection area is the maximum inscribed rectangle of the two-dimensional image. However, since the aspect ratio of the projector 13 is preferably the same, it is actually smaller than the maximum inscribed rectangle. Therefore, the projected image on which the corrected image is projected is smaller than the projected image before correction.

In this projection area, a reduced version of the projector image is displayed. Therefore, the point Ac of the projection area refers to the pixel of the point Ap of the projector image. Hereinafter, a procedure for associating the point Ac of the projection area with the point A of the corrected image will be described.
(1) First, an arbitrary point Ac in the projection area is taken out. Since the coordinates (xc, yc) of the point Ac can be converted to the point Ap of the projector image by the projective transformation, the point Ac can be associated with the point A of the corrected image 43 by a conversion opposite to the projective transformation. it can. That is, the point Ac is associated with the point A of the corrected image 43 with reference to the pixel value of the point Ap. Note that the coordinates (xc, yc) of the point Ac are coordinates in the coordinate system with the origin (0, 0) as the upper left of the projection range of the projector 13, not the projection area.
(2) Next, the position corresponding to the point Ac in the projection area is calculated. The upper left of the projection area is the origin (x0, y0). The coordinates of the point Ac with respect to this origin are (xc1, yc1). Let the coordinates of the point Ap of the original image (projector image) corresponding to the point Ac be (xp, yp). The projection area has the same aspect ratio as the projector image 41, and is different only in size. Therefore, from the horizontal length Wp of the projector image 41 and the horizontal length Wc of the projection area of the projected correction image 43, the coordinates of the point Ap can be expressed as follows.
(Xp, yp) = (Wp / Wc) × (xc1, yc1) (a)
(3) Since the origin of the projection area in the projected corrected image is (x0, y0), the coordinate value of the point Ac can be expressed as follows.
(Xc, yc) = (x0, y0) + (xc1, yc1) (b)
(4) Since the coordinates (xc, yc) of the point Ac are obtained, (xc1, yc1) is obtained from the equation (b), and the coordinates (xp, yp) of the point Ap are obtained from the equation (a). Can do.
(5) The pixel value of the coordinates Ap (xp, yp) is set as the pixel value of the point A (xp ′, yp ′) of the corrected image. Note that the coordinates of the point Ap corresponding to the point A are not necessarily integers. In this case, the coordinates are obtained by interpolation from the pixel values of the pixels around the point Ap.

  The above processing is performed for all the points in the projection area, and the correspondence is summarized in a table or the like. This is a distortion correction parameter. The image deformation unit 26 acquires the pixel value of each pixel of the corrected image 43 from the corresponding pixel of the projector image 41 according to the correspondence relationship of the table (distortion correction parameter), and creates the corrected image 43.

  The calculation of correction information is described in “Experimental study on geometric correction of projected image”, Society of Instrument and Control Engineers Tohoku Branch 235th Research Meeting (2007.5.18).

<< Edge Blending >>
Further, the distortion correction parameter calculation unit 25 performs edge blending on the projector image 41. Edge blending is a technology that applies a gradient to a video signal digitally, and uniformly overlaps overlapping areas.

  FIG. 22 is an example of a diagram for schematically explaining edge blending. The overlapping area 37 of the projector image 41 is determined in advance. The distortion correction parameter calculation unit 25 calculates the luminance of the overlapping region 37 from the two-dimensional image. When the overlapping region 37 has a higher luminance than the other regions, the pixel value of the overlapping region 37 of the projector image 41 is corrected so as to have a uniform luminance with the pixel value of the other region.

<Application example>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

<< Acquisition of surface information of cylindrical object >>
In step S70 of the present embodiment, the three-dimensional coordinates of the feature points of the distortion correction pattern 32 are converted into the two-dimensional projection target surface coordinates of the cylindrical surface. Further, as described in the alignment, in the present embodiment, not only the feature points but also the point cloud 33 can be aligned. Therefore, sufficiently dense image data of the cylindrical projection target 11 can be acquired and aligned.

  FIG. 23 is an example of a diagram schematically illustrating acquisition of surface information included in the cylindrical projection target 11. For example, there is a case where it is desired to take an image of the entire design of the can label. However, since the can has a cylindrical shape, it is necessary to shoot several times in order to image the entire design. Conventionally, it has been difficult to perform operations such as manually aligning a plurality of images to obtain the overall design of the can.

  On the other hand, in this embodiment, not only the surface design but also three-dimensional coordinates can be obtained by imaging the can from multiple directions, and each pixel of the image data can be converted into two-dimensional projection target surface coordinates. A two-dimensional image of the cylindrical surface can be developed in the xy plane. Therefore, image data (development drawing) in a state where the surface of a can or the like is developed can be generated simply by imaging at a different imaging location.

  In addition, not only for can labels, but when a three-dimensional object such as a cylinder is imaged by changing the location of the camera, the image data in the expanded state can be easily obtained, so it can be used for analysis of the surface of the structure. Can do.

<< About shape models other than cylinders >>
In the present embodiment, the cylindrical shape model 35 has been described as an example. However, the present invention can be widely applied to the projection target 11 that can be approximated to the shape model 35 that can be expanded such as a prism, a pyramid, or a cone other than the cylinder. Moreover, it is preferable that it is a shape which can be defined with a model (it can be defined with numerical formula) for fitting.

  FIG. 24A shows an example of a triangular prism and a development view. In the case of a prism, the projection surface is flat, and it is only necessary to fit a flat surface. Moreover, if it is converted into a two-dimensional projection target surface coordinate system by fitting, it can be developed as shown.

  FIG. 24B shows an example of a cone and a development view. In the case of a cone, the radius r is determined according to the axial height of the cone. By fitting the cone shape model 35 by giving the relationship between the height and the radius in advance, the three-dimensional coordinates of the projection image can be converted into a two-dimensional projection target surface coordinate system. Therefore, it can be developed as shown.

  FIG. 24C shows an example of a sphere and a development view. In the case of a sphere, fitting to a three-dimensional shape model 35 is possible as in the case of a cylinder. As a method for expanding the sphere, a method is known in which the upper and lower portions are gradually narrowed as shown in the drawing. In addition, since it is difficult to expand the sphere in two dimensions without distortion, only the vicinity of the equator of the sphere may be expanded, or only the range where the image is projected may be expanded.

DESCRIPTION OF SYMBOLS 11 Projection object 12 Camera 13 Projector 21 Projection image three-dimensional information acquisition part 22 Positioning part 23 Model calculation part 24 Coordinate conversion part 25 Distortion correction parameter calculation part 26 Image deformation part 27 Image data storage part 28 Distortion correction parameter storage part 29 Conversion Information creation unit 50 Information processing device 200 Image processing system

JP 2004-287433 A

Claims (10)

An image processing system for reducing distortion on a projection surface of a projected image,
Three-dimensional information acquisition means for acquiring three-dimensional information of a projection image obtained from a plurality of different locations and projected on the projection plane;
Applying a shape model to the three-dimensional information to determine a shape model of the projection plane;
Coordinate conversion means for projecting the three-dimensional information onto the surface of the shape model, while converting the three-dimensional information into the two-dimensional coordinates of the surface of the shape model,
A distortion correction parameter creating means for creating a distortion correction parameter by comparing the projection image converted into the two-dimensional coordinates and the original image projected as the projection image;
When the coordinate conversion means converts the three-dimensional information into the two-dimensional coordinates of the surface of the shape model, the three-dimensional information is moved and projected in the distance direction from the three-dimensional information acquisition means,
An image processing system characterized by that. The coordinate conversion means projects the three-dimensional information to the closest position on the surface of the shape model when converting the three-dimensional information into the two-dimensional coordinates of the surface of the shape model.
The image processing system according to claim 1. Compared with the three-dimensional information of the projected image captured redundantly from different locations, conversion information creation means for creating conversion information of the coordinates of the three-dimensional information,
Using the conversion information, and having three-dimensional information conversion means for converting each of the three-dimensional information into a common coordinate system,
The shape model determining means applies a shape model to the three-dimensional information of a common coordinate system, and determines the shape model of the projection plane.
The image processing system according to claim 1, wherein the image processing system is an image processing system. The three-dimensional information acquisition means acquires three-dimensional information of a pattern image for alignment as a projection image projected on the projection plane,
The conversion information creation means compares the three-dimensional information of the alignment pattern images that are imaged twice from different locations , and converts the three-dimensional information of the two locations in a common coordinate system. Decide
The three-dimensional information conversion means converts the three-dimensional information of an arbitrary projection image projected on the projection plane, acquired at a plurality of the locations, into a common coordinate system using the conversion information. The image processing system according to claim 3, wherein: The conversion information creating means includes the three-dimensional information of the projection image captured from the nth location and the three-dimensional information of the projection image captured redundantly from the (n + 1) th location adjacent to the nth location. And the conversion information for the (n + 1) th place for converting the three-dimensional information of the projection image at the (n + 1) th place into a common coordinate system with the three-dimensional information of the projection image at the nth place. Decide
Repetitively determining the conversion information until the n + 1th location matches the number N of locations,
The three-dimensional information conversion means uses the conversion information for the (n + 1) th place to convert the three-dimensional information of the (n + 1) th place equal to N into the coordinate system of the three-dimensional information of the nth place. the image processing system according to the process of converting is repeated until the conversion of the three-dimensional information of the second of said locations in the first coordinate system of the three-dimensional information of the location, it in請Motomeko 4, wherein. The three-dimensional information acquisition means acquires three-dimensional information of a distortion correction pattern image as the projection image projected on the projection plane from the N locations.
The three-dimensional information conversion means uses the conversion information for the (n + 1) th place to convert the three-dimensional information of the pattern image for distortion correction at the (n + 1) th place into the coordinate system of the three-dimensional information at the nth place. 6. The image processing system according to claim 5, wherein the conversion processing is repeated until the second three-dimensional information of the place is converted into the coordinate system of the first three-dimensional information of the place. The coordinate conversion means converts the three-dimensional information of the N distortion correction pattern images represented in the common coordinate system into the two-dimensional coordinates,
The distortion correction parameter creating means compares the N distortion correction pattern images converted into the two-dimensional coordinates with the N original distortion correction pattern images projected as the projection image to generate distortion. Create correction parameters,
The image processing system according to請Motomeko 6, characterized in that. The shape model image processing system according to any one of claims 1 to 7, deployment of the plane in the range of at least the three-dimensional information is obtained can be, be characterized. An image processing apparatus that reduces distortion on a projection surface of an image to be projected,
Three-dimensional information acquisition means for acquiring three-dimensional information of a projection image obtained from a plurality of different locations and projected on the projection plane;
Applying a shape model to the three-dimensional information to determine a shape model of the projection plane;
Coordinate conversion means for projecting the three-dimensional information onto the surface of the shape model, while converting the three-dimensional information into the two-dimensional coordinates of the surface of the shape model,
A distortion correction parameter creating means for creating a distortion correction parameter by comparing the projection image converted into the two-dimensional coordinates and the original image projected as the projection image ;
The coordinate conversion means, when converting the three-dimensional information into the two-dimensional coordinates of the surface of the shape model, moves and projects the three-dimensional information in the distance direction from the three-dimensional information acquisition means. Image processing device. On the computer,
A three-dimensional information acquisition step of acquiring three- dimensional information of a projection image projected on a projection plane from a plurality of different locations by a three-dimensional information acquisition unit ;
Applying a shape model to the three-dimensional information and determining a shape model of the projection plane; and
A coordinate conversion step of projecting the three-dimensional information onto the surface of the shape model, and converting the three-dimensional information into the two-dimensional coordinates of the surface of the shape model;
A program for executing a distortion correction parameter creating step of creating a distortion correction parameter by comparing the projection image converted into the two-dimensional coordinates and the original image projected as the projection image,
In the coordinate conversion step, when converting three-dimensional information into two-dimensional coordinates on the surface of the shape model, the three-dimensional information is moved and projected in a distance direction from the three-dimensional information acquisition means. program.

Images