JP2005326247A - Calibrator, calibration method, and calibration program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance the accuracy of shape measurement and to beautify a projected image irrespective of the shape of a projection surface. <P>SOLUTION: A pattern picture is generated by a pattern imaging part 14, the picture is transferred to a projector 11, and then an imaging order is issued to an imaging device 12. A pattern imaging picture imaged by the imaging device 12 is stored in a storage device 13 and outputted to a pattern imaging part 14. Nextly, when a pattern extraction part 15 is started, the imaging picture photographed by the imaging part 14 is read from the storage device 13. The extraction part 15 transfers the read imaging picture, as an extracted pattern picture, via a plane area extraction part 16 and a correspondence coordinate calculation part 17 to a plane projection matrix calculation part 18. Lastly, when a parameter estimation part 19 is started, plane ID and a plane projection matrix corresponding to the plane ID are received from a plane projection matrix estimation part 18. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a camera (imaging device) necessary for calculating the three-dimensional coordinates of a projection plane used when performing projection without distortion by a projector (projection device), and a camera that enables automatic calibration of the projector. The present invention relates to a calibration apparatus, a calibration method, and a calibration program in a projector, and in particular, the characteristics of a camera using an arbitrary plane without spatial positional constraints, and internal parameters representing the characteristics of the projector, and the position between the camera and the projector, The present invention relates to a calibration apparatus, a calibration method, and a calibration program for calculating external parameters representing a rotation relationship.

  When projecting an image input to the projector using a projection device such as a projector, the projected image is distorted in areas where the optical axis of the projection device and the normal of the projection surface (screen) do not match, such as an uneven surface. Will occur.

  The following documents are disclosed as conventional techniques for correcting such distortion (see Patent Document 1, Patent Document 2, and Patent Document 3).

  The one disclosed in Patent Document 1 is configured to manually adjust the optical system, and the one disclosed in Patent Documents 2 and 3 includes a correction function for manually adjusting and outputting an image corresponding to distortion. Yes.

  Further, there has been proposed a projector that automatically corrects the distortion of the projection screen even if the three-dimensional shape of the projection surface is arbitrary (see Patent Document 4).

  FIG. 22 is a block diagram showing an overall configuration of a projector having such a function. In FIG. 22, the pattern image generation unit 220 generates a predetermined pattern, and projects the pattern from the video output unit 221 onto the projection plane.

  The camera 222 captures the pattern on the projection plane and sends it to the projection plane acquisition unit 223. The projection plane acquisition unit 223 measures the projection distance and projection direction at each point constituting the projection plane using the principle of the light projection method, and outputs the measurement results to the video correction unit 224 and the projection control unit 225.

  The video correction unit 224 that has received the measurement result from the projection plane acquisition unit 223 measures the projection distance and the projection direction at each point constituting the projection plane from the measurement result, and then performs the process on the original image obtained from the video input unit 226. The normal vector of the projection plane of the projection plane has the same horizontal component as that of the projection plane, and an image without distortion is displayed on a virtual projection plane parallel to the floor surface. An image corrected by back projection simulation based on the perspective transformation model is output to the video output unit 221.

  On the other hand, in the projection control unit 225 that has received the measurement result from the projection plane acquisition unit 223, the video output unit 221 switches the video that is the projection output to the output from the video correction unit 224. By adopting such a configuration, an image whose distortion is automatically corrected is output even on a projection screen having an arbitrary three-dimensional shape.

  Among the above-described conventional techniques, those disclosed in Patent Document 1 to Patent Document 4 have a configuration in which an optical system is manually adjusted and an adjustment error occurs.

  Further, when measuring the shape of the projection plane by the method described with reference to FIG. 22, the relative position and orientation of the camera and the projector and the internal parameters of the camera and the projector based on the perspective transformation model are accurately determined by an error due to manual adjustment. If it is not obtained, it is difficult to measure the shape with high accuracy, and the projected image remains distorted.

  On the other hand, a calibration method for obtaining the relative position and orientation of the camera and projector and the internal parameters of the camera and projector has been proposed (see Patent Document 5).

  According to the technique described in Patent Document 5, a board on which a checkerboard pattern with a known lattice point distance is drawn is first photographed by the camera, and then the relative position and orientation of the board and the camera, the internal parameters of the camera are taken from the photographed image. Is estimated.

  Next, 3D in the space of the pattern points extracted from the image based on the relative position and orientation of the board estimated in the previous step, from the image obtained by projecting the pattern points from the projector onto the board Estimate the coordinates.

Finally, by calculating the internal parameters of the Zodiector using the 3D coordinates in the space of the pattern points and the 2D coordinates of the projected pattern points on the projection plane of the projector, The internal parameters of the camera and projector are accurately obtained.
Japanese Patent Laid-Open No. 4-131819 JP-A-8-336092 JP-A-8-289237 JP 2001-061121 A JP 2001-320652 A

  However, according to the technique described in Patent Document 5 described above, it is necessary to prepare a board on which a checkerboard pattern with known grid point distances is drawn at the time of calibration, or to prepare an image taken by moving the board several times. Therefore, there is a problem that it is troublesome and troublesome and takes time.

  Also, the relative position and orientation of the camera and projector calculated when the distance between the grid points given to the system and the board with the checkered pattern with known grid point distances and the system given at the time of calibration, There is a problem that an error occurs in the parameter, it is difficult to measure the shape with high accuracy, and the projected image remains distorted.

  The present invention has been made in view of the above-described problems of the prior art. In a calibration apparatus for a camera / projector, a calibration method thereof, and a calibration program thereof, an arbitrary plane having no spatial position constraint is provided. A calibration device, calibration method, and calibration program for automatically and accurately calculating internal parameters representing camera characteristics and projector characteristics, and external parameters representing the position and rotation relationship between the camera and projector are realized. The purpose is to do.

  In order to solve the above-described problems, a calibration apparatus according to the present invention projects a projection apparatus that outputs a projection image input to a projection plane composed of at least two planes, and captures a scene including the projection plane. An imaging device, a pattern imaging means for generating a predetermined pattern image, projecting and outputting the pattern image as a projected image by the projection device, and capturing the state of the projection plane at that time as a pattern captured image by the imaging device; and A pattern extraction unit that extracts a pixel corresponding to the pattern image from the pattern captured image using the pixel value of the image to obtain an extracted pattern image, and the geometry of the extracted pixels in the extracted pattern image and the pattern image A plane area extracting means for extracting a plane area in the projection plane from the extracted pattern image using the relationship with geometry; and the plane area extracting means Corresponding coordinate calculation means for performing processing for determining at least four or more points to be associated with the extracted plane area on the pattern image over all the extracted plane areas; A plane projection matrix calculating means for calculating for each plane region a plane projection matrix that expresses a relationship between points existing in the plane, and internal characteristics (each of the projection apparatus and the imaging apparatus from at least two plane projection matrices) And a parameter estimation means for calculating a relative position and a posture.

  Alternatively, in the pattern imaging means, the predetermined pattern image to be generated and projected is a checkered pattern in which a pixel value of one color is expressed by a pixel value that is different from the pixel value and is formed with vertical and horizontal gaps. It is characterized by being.

  Alternatively, in the pattern imaging unit, a predetermined pattern image generated and projected and output is a vertical stripe pattern in which a pixel value of one color is formed with a vertical gap on a pixel represented by a pixel value different from the pixel value. It is characterized in that it is at least two images of a horizontal stripe pattern in which a pixel value of one color is formed with a gap horizontally on a pixel represented by a pixel value different from the pixel value.

  Alternatively, in the pattern imaging means, the predetermined pattern image generated and projected and output is a two-color color on the image so that the color change in the time series of the projection plane after the projection output is expressed by a binary gray code. Are a plurality of binary gray coded pattern images arranged in stripes vertically and horizontally.

  Alternatively, in the pattern extraction unit, binarization is performed on the pixel values of the image from the pattern captured image, and the binarized image is subsequently connected to each connected figure included in the image. In this case, pixels corresponding to the pattern image are extracted to obtain an extracted pattern image by thinning to a line figure having a single line width without losing properties.

  Alternatively, in the pattern extraction unit, from the two pattern captured images obtained by imaging the state of the projection plane on which the vertical stripe pattern and the horizontal stripe pattern or a pair of vertical and horizontal stripe patterns of the binary gray coded pattern image are projected and output. A difference image is generated by calculating an absolute value of a pixel value difference for each pixel of the image, binarization is performed on the pixel value of the difference image, and a connected component of the binarized image is calculated. An expansion process for increasing the outline pixel by one layer and a reduction process for removing the outline pixel of the connected component and reducing it by one layer are performed a predetermined number of times, and then the image subjected to the expansion / reduction process is included in the image. Each of the included connected figures is thinned to a line figure having a single line width without losing connectivity, thereby extracting pixels corresponding to the pattern image to obtain an extracted pattern image. To.

  Alternatively, in the pattern extraction means, from at least two pattern captured images obtained by imaging the state of a projection plane obtained by projecting and outputting a pair of vertical and horizontal stripe patterns of the vertical stripe pattern and the horizontal stripe pattern or the binary gray coded pattern image. The pixel value is binarized with respect to each of the pattern imaged images, and the dilation processing and the connected component of each of the binarized images are increased by one layer of the connected component outline pixel. The outline pixel is removed, and the reduction process for reducing the size by one layer is performed a predetermined number of times, and then the connection to each connected figure included in the image is lost for each of the two images that have been expanded and reduced. The pixels corresponding to the pattern image are extracted by thinning to a line figure with a single line width without any subsequent extraction, and then the logical sum operation result of both images is extracted. Characterized by an over down image.

  Alternatively, in the plane area extraction means, the extraction pattern image is received from the pattern extraction unit, and all intersections of a plurality of straight lines described in the extraction pattern image are detected, and from all the extracted intersections Dividing into a quadrangle with each intersection as a vertex, and processing the square plane when the number of overlapping pixels between each side of the quadrangle and the straight line in the extracted pattern image exceeds a predetermined threshold value And a unique plane ID for identifying each plane is assigned to each of the integrated plane areas, and the plane ID and the image coordinates in the plane area corresponding to the plane ID are provided. Is recorded.

  Alternatively, in the corresponding coordinate calculation unit, at least four or more feature points that become corners or intersections on the image using the pixel values of the pattern captured image from the points on the plane region extracted by the plane region extraction unit Extracting, creating a collation pattern using pixel values of a region including the feature point, and determining a corresponding point corresponding to the feature point on the pattern image using the collation pattern.

  Alternatively, in the corresponding coordinate calculation means, the pattern captured image is binarized by two colors used in the binary gray coded pattern image, and the plurality of binarized images are patterned. A spatial code image arranged in an image projection order is generated, a gray code of each pixel of the pattern captured image is restored from the spatial code image, and a binary gray code expressed by the restored gray code and the pattern image Is used to determine at least four or more corresponding points corresponding to points on the plane area of the pattern-captured image.

  Alternatively, a pattern imaging process of generating a predetermined pattern image, projecting and outputting the pattern image as a projection image by the projection apparatus, and then imaging the state of the projection plane as a pattern captured image by the imaging apparatus, and the pattern captured image Then, after extracting a pixel corresponding to the pattern image using the pixel value of the image, a pattern extraction process to obtain an extraction pattern image, and the geometry of the extracted pixels in the extraction pattern image and the geometry in the pattern image A plane area extraction process for extracting a plane area in the projection plane from the extracted pattern image using the relationship of the above, and a point on the plane area extracted by the plane area extraction means and a point associated with the pattern image Corresponding coordinate calculation process in which at least four or more points are determined and then the processing is performed over all extracted planar areas, and the corresponding A plane projection matrix calculation process for calculating, for each plane area, a plane projection matrix that expresses a relationship between points existing on a plane from the corresponding coordinates obtained in the target calculation process, and at least two or more plane projection matrices And a parameter estimation process for calculating the internal characteristics, relative position, and orientation of each of the projection device and the imaging device.

  Alternatively, the predetermined pattern image generated and projected and output in the pattern imaging process is a checkered pattern in which a pixel value of one color is formed with a vertical and horizontal gap on a pixel represented by a pixel value different from the pixel value. It is characterized by being.

  Alternatively, in the pattern imaging process, the predetermined pattern image generated and projected and output is a vertical stripe pattern in which a pixel value of one color is formed with a vertical gap on a pixel represented by a pixel value different from the pixel value. It is characterized in that it is at least two images of a horizontal stripe pattern in which a pixel value of one color is formed with a gap horizontally on a pixel represented by a pixel value different from the pixel value.

  Alternatively, the predetermined pattern image generated and projected in the pattern imaging process has two colors on the image so that the color change in the time series of the projection plane after the projection output is expressed by a binary gray code. Are a plurality of binary gray coded pattern images arranged in stripes vertically and horizontally.

  Alternatively, in the pattern extraction process, after the binarization is performed on the pixel values of the image from the pattern captured image, each connected figure included in the binarized image is subsequently included in the image In this case, pixels corresponding to the pattern image are extracted to obtain an extracted pattern image by thinning to a line figure having a single line width without losing connectivity.

  Alternatively, in the pattern extraction process, from the two pattern captured images obtained by imaging the state of the projection plane on which the vertical stripe pattern and the horizontal stripe pattern or the pair of vertical and horizontal stripe patterns of the binary gray coded pattern image are projected and output. After calculating the absolute value of the pixel value difference for each pixel of the image and generating a difference image, binarization is performed on the pixel value of the difference image, and then the binarized image is Expansion processing for increasing the contour pixels of the connected component by one layer and reduction processing for removing the contour pixels of the connected component to reduce the size by one layer are performed a predetermined number of times, and then the expansion and reduction processing is performed on the image that has been expanded and contracted. Extracting the pixel corresponding to the pattern image by thinning the line graphic to one line graphic without losing connectivity to each connected graphic included in the image and extracting the pattern image And wherein the Rukoto.

Alternatively, in the pattern extraction process, from at least two or more pattern imaged images obtained by imaging the state of the projection plane obtained by projecting and outputting a pair of vertical and horizontal stripe patterns of the vertical stripe pattern and the horizontal stripe pattern or the binary gray coded pattern image Then, after performing binarization on the pixel values for each of the pattern imaged images, the dilation processing is performed to increase the contour pixel of the connected component by one layer outward for each of the two binarized images. The contour pixel of the component is removed, and a reduction process for reducing the size by one layer is performed a predetermined number of times, and then each of the two images subjected to the expansion / reduction process is connected to each connected figure included in the image. The pixels corresponding to the pattern image are extracted by thinning to a line figure with a line width of 1 without losing the image, and then the logical OR operation of both images. To the fruit and extract the pattern image,
It is characterized by.

Alternatively, in the planar area extraction process, after receiving the extraction pattern image from the pattern extraction process, all intersections of a plurality of straight lines described in the extraction pattern image are detected, and all the extracted intersections are detected. Is divided into quadrangular planes when the number of overlapping pixels between each side of the quadrilateral and the straight line in the extracted pattern image exceeds a predetermined threshold. The process is performed over all squares, and then the adjacent square planes are integrated, and a unique plane ID for identifying each plane is assigned to each of the integrated plane areas, and then the plane ID and the plane ID are associated. Storing image coordinates within a planar area;
It is characterized by.

  Alternatively, in the corresponding coordinate calculation process, at least four or more feature points serving as corners or intersections on the image using pixel values of the pattern captured image from the points on the plane area extracted in the plane area extraction process After the extraction, a matching pattern is created using the pixel value of the region including the feature point, and then a corresponding point corresponding to the feature point is determined on the pattern image using the matching pattern. To do.

  Alternatively, in the corresponding coordinate calculation process, the pattern captured image is binarized with two colors used in the binary gray coded pattern image, and then the plurality of binarized images Are generated in the pattern image projection order, and then the gray code of each pixel of the pattern captured image is restored from the spatial code image, and the restored gray code and the pattern image 2 It is characterized in that at least four or more corresponding points corresponding to points on the plane area of the pattern-captured image are determined by using a gray code.

  Alternatively, the process in the calibration method is a program for causing a computer to execute the process.

  In the present invention, the relative position and orientation of the imaging device (camera) and the projection device (projector) and the internal parameters of the camera and projector can be accurately estimated, and a highly accurate shape measurement is possible, so that a beautiful image can be obtained regardless of the shape of the projection surface. Can be projected.

  The reason for this is that the pattern projected by the projector and the characteristics of the pattern image captured by the camera are automatically extracted to calculate the relative position and orientation of the camera and projector and the internal parameters of the camera and projector. This is because no error occurs.

  In the present invention, even if the camera and projector are replaced or the position and orientation relationship is changed, calibration is performed immediately, and a beautiful image can be presented to the user at high speed regardless of the shape of the projection surface. .

  This is because only the pattern projected by the projector and the pattern captured image obtained by photographing the pattern with the camera are used, so that it is not necessary to prepare the board or photograph the board by moving it several times.

  As described above, according to the present invention, it is possible to accurately estimate the relative position and orientation of the camera and the projector and the internal parameters of the camera and the projector. By using the result, it is possible to measure the shape with high accuracy and to project a beautiful image regardless of the shape of the projection surface. The reason for this is that the pattern of the pattern projected by the projection device and the characteristics of the pattern captured image obtained by capturing the pattern are automatically extracted to calculate the relative position and orientation of the camera and projector and the internal parameters of the camera and projector. It does not occur.

  In addition, according to the present invention, even if the camera and projector are replaced or the position and orientation relationship is changed, the calibration is performed immediately, and a beautiful image can be presented to the user at high speed regardless of the shape of the projection surface. It becomes possible. This is because only the pattern projected by the projection device and the pattern captured image obtained by photographing the pattern are used, so that it is not necessary to prepare the board or to photograph the board several times.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, a configuration example of a calibration apparatus (hereinafter referred to as a camera / projector calibration apparatus) according to the present invention will be described.

  FIG. 1 is a block diagram for explaining a camera / projector calibration apparatus according to this configuration example. The camera / projector calibration apparatus of the configuration example shown in FIG. 1 includes a projection device 11, an imaging device 12, a storage device 13, a pattern imaging unit 14, a pattern extraction unit 15, a plane area extraction unit 16, a corresponding coordinate extraction unit 17, a plane The projection matrix calculation unit 18 and the parameter estimation unit 19 are roughly configured.

  The projection device 11 is a device that projects and displays an input projection image on a projection surface, and can be realized by, for example, a projector or a laser projector that includes a liquid crystal element or a DMD semiconductor element.

  The imaging device 12 is an area sensor camera having a two-dimensional CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor) inside a camera such as a video camera, a high-speed camera, an omnidirectional sensor, an infrared sensor, or an ultraviolet sensor. The scene is captured, and the captured image is output to the storage device 13.

  The storage device 13 is realized by a semiconductor storage device such as a RAM or a ROM, a hard disk, a flexible disk, an optical disk, or the like included in the information processing apparatus.

  The pattern imaging unit 14 generates a predetermined pattern image and outputs it to the projection device 11 as a projection image. The pattern imaging unit 14 records the pattern captured image captured by the imaging device 12 in the storage device 13 at that time. .

  The pattern extraction unit 15 reads the pattern captured image from the storage device 13, extracts pixels corresponding to the pattern image, and outputs the extracted pattern image.

  The plane area extraction unit 16 receives the extracted pattern image from the pattern extraction unit 15, extracts all plane areas in the projection plane imaged from the pattern image, and determines the plane ID and the plane that uniquely determine each plane. All image coordinates in the plane area corresponding to the ID are transferred to the corresponding coordinate calculation unit 17.

  The corresponding coordinate calculation unit 17 receives the plane ID and all the image coordinates in the plane area corresponding to the plane ID, determines at least four or more points to be associated with the points in the plane area on the pattern image, and The coordinates are output to the plane projection matrix calculation unit 18.

  The plane projection matrix calculating unit 18 receives the corresponding coordinates, calculates a plane projection matrix expressing the relationship between points existing on the plane from the corresponding coordinates, and outputs the calculated plane projection matrix for each plane area.

  The parameter estimation unit 19 receives the planar projection matrix and calculates internal parameters, relative positions, and orientations of the projection device 11 and the imaging device 12 from the planar projection matrix.

  Next, the processing flow performed by the camera / projector calibration apparatus will be described in detail. The entire process is shown in FIG. In the camera / projector calibration apparatus, the pattern imaging unit 14 is first activated. When the pattern imaging unit 14 is activated, a predetermined pattern image is first generated (S1).

  Here, the predetermined pattern image is, for example, a pattern image expressed by a checkered pattern or the like in which a pixel value of one color is formed with a vertical and horizontal gap on a pixel expressed by a pixel value different from the pixel value. is there.

  Subsequently, the generated pattern image is transferred to the projection apparatus 11 and a projection output command is issued. When a projection output completion signal is received from the projection device 11, an imaging command is issued to the imaging device 12.

  When the imaging device 12 receives the imaging command, the imaging device 12 performs imaging, stores the next captured pattern image in the storage device 13, and outputs an imaging completion signal 1 to the pattern imaging unit 14. The pattern imaging unit 14 receives the imaging completion signal from the imaging device 12 and ends the process.

  Subsequently, the pattern extraction unit 15 is activated (S2). The processing in the pattern extraction unit 15 is shown in FIG. When the pattern extraction unit 15 is activated, first, a pattern captured image captured by the pattern imaging unit 14 is read from the storage device 13 (S21).

  The pattern extraction unit 15 binarizes the pixel values of the read pattern captured image (S22), and subsequently loses the connectivity of each connected figure included in the image with respect to the binarized image. The line is thinned to a line figure having a single line width (S23). Since the thinning is not particularly different from the conventional technology, detailed description thereof is omitted here. Finally, the thinned image is transferred as an extraction pattern image to the planar area extraction unit 16 and the process is terminated (S24).

  Subsequently, the plane area extraction unit 16 is activated (S3). FIG. 4 is a diagram showing the flow of processing in the planar area extraction unit 16. In the plane area extraction unit 16, when the square pattern is projected from the image plane of the projection device 11 onto a plane projection plane existing in the space and the state is imaged on the image plane of the imaging device 12, the imaging device 12 outputs the image. A plane area is extracted by utilizing the fact that a square image is observed in the captured image (S4) (S5).

  In addition, when the projection surface which exists in space is not a plane, it is not observed as a square in a captured image. When the plane area extraction unit 16 is activated, an extraction pattern image is first received from the pattern extraction unit 15 (S31).

  Subsequently, all intersections of a plurality of straight lines expressed in the extracted pattern image are detected. This processing is performed by, for example, obtaining an intersection of straight lines obtained by performing the Hough transform on the extracted pattern image (S32), or applying a filter such as a SUSAN operator known as a corner extraction filter to the extracted pattern image. realizable.

  Subsequently, the obtained plurality of intersection groups are divided into quadrangular elements (S33). For example, the quadrilateral division processing is a method in which Delaunay triangulation is extended to a quadrilateral element, or by searching for neighboring regions in the direction of two straight lines that form the intersection from the intersection of the extracted straight lines, the other two intersections are determined. Subsequently, the last one point is determined by a similar method from the two determined intersection points.

  Subsequently, one rectangle is selected from the divided rectangles (S34), and when the number of overlapping pixels between the four sides constituting the rectangle and the straight line in the extracted pattern image exceeds a predetermined threshold value, the rectangle is selected in the space. A square plane representing a plane area is set (S35).

  This process is performed on all square elements (S36). Subsequently, adjacent square planes are connected and integrated (S37), and a unique plane ID for individually identifying each plane is assigned to form a plane area (S38). This processing can be realized by, for example, clustering using a square area or a square angle difference as a threshold. Finally, all the plane IDs and at least four or more coordinates on the plane area respectively corresponding to the plane ID are transferred to the corresponding coordinate calculation unit 17 and the process is terminated (S39).

  Subsequently, the corresponding coordinate calculation unit 17 is activated. First, the corresponding coordinate calculation unit 17 receives all the plane IDs and all image coordinates in the plane area respectively corresponding to the plane ID from the plane area extraction unit 16 (S6).

  Next, the pattern image and the pattern-captured image are read from the storage device 13, and then the coordinates of feature points that are corners and intersections on the image are extracted from the pattern-captured image.

  As the feature point, the intersection of the straight lines obtained by the plane area extraction unit 16 may be used as it is. Next, a matching pattern is created using feature points on the planar area of the pattern captured image and pixel values near the feature point coordinates, and corresponding points corresponding to the feature points in the pattern image using the matching pattern To decide.

  This corresponding point determination process is performed on at least four points for each plane ID. Further, it is performed for at least two plane IDs. Finally, the entire plane ID, the image coordinates on the plane ID, and the corresponding coordinates of the image coordinates are transferred to the plane projection matrix calculation unit 18 and the process is terminated.

  Subsequently, the planar projection matrix calculation unit 18 is activated (S7). When the plane projection matrix is activated, it first receives all plane IDs, image coordinates on the plane ID, and corresponding coordinates of the image coordinates from the corresponding coordinate calculation unit 17.

  Subsequently, a plane projection matrix H is calculated from four or more corresponding points between the pattern image and the pattern photographed image for each received plane ID. The planar projection matrix H is a matrix representing the projective relationship of points existing on the same plane between two images obtained by photographing the same plane as shown in FIG. 23, and has nine parameters as shown in Expression (1). (Hij).

  For example, “Hartley R.,“ Indefence of the eightpoint algorithm, ”IEEE Transactions of Pattern Analysis and Machine Intelligence, Vol.19, No.6, pp580, is used to calculate the planar projection matrix H from four or more corresponding points. -593, 1997. " Finally, the plane ID and the plane projection matrix H corresponding to the plane ID are transferred to the parameter estimation unit 19 and the process is terminated.

  Finally, the parameter estimation unit 19 is activated (S8). When the parameter estimation unit 19 is activated, it first receives a plane ID and a plane projection matrix H corresponding to the plane ID from the plane projection matrix estimation unit 18.

  Subsequently, the internal characteristics (focal length of each device), relative position, and orientation of each of the projection device 11 and the imaging device 12 are calculated from the received at least two or more plane projection matrices H.

The relationship between the planar projection matrix H, the internal characteristics of each of the projection device 11 and the imaging device 12, and the relative position and orientation are derived as follows. First, matrixes representing the internal characteristics of the projection device 11 and the imaging device 12 are represented by the following expressions A and A ′ (f ₁ and f ₂ are focal lengths), respectively.

Let n be the normal vector of a plane existing in space,

The relative position vector of the imaging device 12 with the projection device 11 as a reference is expressed as T and the relative attitude determinant R is expressed as

Assuming that the distance from the projection device 11 to the plane is d, the plane projection matrix H is expressed by Expression (2).

  In consideration of the fact that the actually observed planar projection matrix H includes the scale component S, the equation (2) is transformed into the following equation (3).

  Also, an epipole e, which is a point obtained by projecting the projection centers of the projection device 11 or the imaging device 12 to each other, can be obtained from the planar projection matrix H expressed by Expression (4).

Here, H ₁ represents H obtained for the plane with the plane ID = 1, and H ₂ represents H obtained for the plane with the plane ID = 2.

  From the relationship between the epipoles calculated by the equations (3) and (4), A, A ′, the relative position T, and the posture R indicating the internal characteristics of the projection device 11 and the imaging device 12 can be derived.

  This derivation is `` Xu, Gang, Terai Junichi and Shum, Heung-Yeung., “A linear algorithm for camera self-calibration, motion and structure recovery for multi-planar scenes from two perspective images”, Proceedings of IEEE Conference on Computer Vision. and Pattern Recognition, 2000. ”. Finally, the parameter estimation unit 19 outputs the calculated internal characteristics A and A ′, the relative position T, and the relative attitude R, and ends the process.

  Then, the 2nd structure of the information presentation apparatus in this invention is demonstrated.

  The camera / projector calibration apparatus of this example includes a projection apparatus 11, an imaging apparatus 12, a storage device 13, a pattern imaging section 14, a pattern extraction section 15, a plane area extraction section 16, a corresponding coordinate extraction section 17, and a plane projection matrix calculation section 18. And a parameter estimation unit 19.

  The configuration of this apparatus is substantially the same as the first configuration, except that the pattern imaging unit 14 generates two images of a vertical stripe pattern and a horizontal stripe pattern and performs projection output and imaging, and the pattern extraction unit 15 By newly providing a step of generating an extraction pattern image from the difference image between the two images, it is possible to generate an extraction pattern image that is not affected by the color of the projection plane. As a result, the projection device 11 and the imaging device 12 have high accuracy. The difference is that the calibration is realized.

  The overall processing of the camera / projector calibration apparatus of this example is the same as the processing of the first configuration shown in FIG. The processes in the plane area extraction unit 16, the corresponding coordinate extraction unit 17, the plane projection matrix calculation unit 18 and the parameter estimation unit 19 are the plane area extraction unit 16, the corresponding coordinate extraction unit 17, the plane projection matrix in the case of the first configuration. Since it is the same process as the calculation part 18 and the parameter estimation part 19, detailed description here is abbreviate | omitted.

  A processing flow performed in the pattern imaging unit 14 will be described. When activated, the pattern imaging unit 14 first generates a predetermined pattern image. Here, the predetermined pattern is, for example, a vertical stripe pattern in which a pixel value of one color is expressed by a pixel value different from the pixel value, and a pixel value of one color is the pixel value. This is a pattern expressed by two images of a horizontal stripe pattern formed by interposing a gap horizontally on pixels expressed by different pixel values.

  Subsequently, the generated pattern images are transferred to the projection device 11 one by one and a projection output command is issued. When a projection output completion signal is received from the projection device 11, an imaging command is issued to the imaging device 12. Upon receiving the imaging command, the imaging device 12 captures each pattern image, stores the captured pattern captured image in the storage device 13, and outputs an imaging completion signal to the pattern imaging unit 14. The pattern imaging unit 14 receives the imaging completion signal from the imaging device 12 and ends the process.

  Next, a processing flow in the pattern extraction unit 15 will be described with reference to FIG. When the pattern extraction unit 15 is activated, it first reads out from the storage device 13 a pattern captured image when the vertical stripe pattern imaged by the pattern imaging unit 14 is projected and two pattern captured images when the horizontal stripe pattern is projected (S51, S52). ).

  Next, a difference image between the two images is generated (S53). The difference process can be realized by calculating an absolute value of a difference between pixel values having the same image coordinates. Subsequently, the pixel values of the generated difference image are binarized to generate a binarized image (S54).

  Subsequently, expansion / reduction processing is performed on the binarized image in order to fill in the blurred / interrupted line segments constituting the pattern in the captured image generated by the difference processing and binarization processing (S55).

  This expansion / reduction process is performed by performing a predetermined number of times a process of increasing the outline pixel of the connected component by one layer on the binarized image and a process of removing the outline pixel of the connected component and reducing it by one layer. Realized.

  Subsequently, thinning is performed on the image after expansion / reduction processing so as to reduce the line width to one line figure without losing the connectivity of each connected figure included in the image (S56). Since the thinning is not particularly different from the conventional technology, detailed description thereof is omitted here. Finally, the thinned image is transferred as an extraction pattern image to the planar area extraction unit 16 and the process is terminated (S57).

  Next, a third configuration of the information presentation device in the present invention will be described. The camera / projector calibration apparatus of this example includes a projection apparatus 11, an imaging apparatus 12, a storage device 13, a pattern imaging section 14, a pattern extraction section 15, a plane area extraction section 16, a corresponding coordinate extraction section 17, and a plane projection matrix calculation section 18. And a parameter estimation unit 19.

  The configuration of this apparatus is substantially the same as that of the first configuration, except that the pattern imaging unit 14 generates two images of a vertical stripe pattern and a horizontal stripe pattern and performs projection output and imaging, and the pattern extraction unit 15 By providing a step of generating an extraction pattern image from a logical sum image obtained by performing binarization, expansion / reduction processing, and thinning of both images, the light from the projection device 11 is emitted when the shape of the projection surface is complicated. In addition to the position of the desired projection plane, it is possible to generate an extraction pattern image that suppresses the influence of secondary reflection that reflects to other projection plane positions. As a result, accurate calibration of the projection apparatus 11 and the imaging apparatus 12 can be performed. What is realized is different.

  The entire process of the camera / projector calibration apparatus of this example is the same as the process of the first configuration shown in FIG. The processes in the plane area extraction unit 16, the corresponding coordinate extraction unit 17, the plane projection matrix calculation unit 18 and the parameter estimation unit 19 are the plane area extraction unit 16, the corresponding coordinate extraction unit 17, the plane projection matrix in the case of the first configuration. Since it is the same process as the calculation part 18 and the parameter estimation part 19, detailed description here is abbreviate | omitted.

  A processing flow performed in the pattern imaging unit 14 will be described. When activated, the pattern imaging unit 14 first generates a predetermined pattern image. Here, the predetermined pattern is, for example, a vertical stripe pattern in which a pixel value of one color is formed with a vertical gap on a pixel represented by a pixel value different from the pixel value, and a pixel value of one color is the pixel value. This is a pattern expressed by two or more images of a horizontal stripe pattern formed by interposing a gap horizontally on pixels expressed by different pixel values.

  Subsequently, the generated pattern images are transferred to the projection device 11 one by one and a projection output command is issued. When a projection output completion signal is received from the projection device, an imaging command is issued to the imaging device 12 next.

  When receiving the imaging command, the imaging device 12 captures each pattern image, then stores the captured pattern captured image in the storage device 13, and outputs an imaging completion signal to the pattern imaging unit. The pattern imaging unit 14 receives the imaging completion signal from the imaging device 12 and ends the process.

  Next, a processing flow in the pattern extraction unit 15 will be described with reference to FIG. When the pattern extraction unit 15 is activated, first, the pattern captured image obtained by projecting the vertical stripe pattern captured by the pattern imaging unit 14 and two or more pattern captured images obtained by projecting the horizontal stripe pattern are read from the storage device 13 (S61). .

  Subsequently, it is determined in step S62 whether both vertical and horizontal stripes have been processed. If NO, the pixel values of both pattern captured images are binarized to generate a binarized image (S63).

  In order to avoid the influence of the color of the projection plane, this binarization processing may be applied to a difference image obtained by subtracting an image of the projection plane before pattern projection that has been captured in advance from both pattern captured images.

  Subsequently, expansion / reduction processing is performed on the two binarized images in order to fill in the blurred / discontinuous line segments constituting the pattern in the captured image generated by the binarization processing (S64).

  This expansion / reduction processing is performed by performing processing for increasing the connected component outline pixel by one layer on the binarized image and processing for removing the connected component contour pixel and reducing it by one layer a predetermined number of times. Realized.

  Subsequently, the two images after the expansion / reduction processing are thinned so as to reduce the line width to one line figure without losing the connectivity of each connected figure included in the image (S65).

  Since the thinning is not particularly different from the conventional technology, detailed description thereof is omitted here. Finally, a logical sum image of both the thinned images is generated and used as an extracted pattern image.

  If YES in S62, the logical sum image generation processing can be realized by setting 1 if any of the pixel values having the same image coordinates has a value, and setting it to “0” otherwise (S66). Moreover, if both images are binary images, it can also be realized by adding both images. Finally, the extracted pattern image is transferred to the planar area extracting unit 16 and the process is terminated (S67).

  Next, a fourth configuration of the camera / projector calibration apparatus according to the present invention will be described. The camera / projector calibration apparatus of this example includes a projection apparatus 11, an imaging apparatus 12, a storage device 13, a pattern imaging section 14, a pattern extraction section 15, a plane area extraction section 16, a corresponding coordinate extraction section 17, and a plane projection matrix calculation section 18. And a parameter estimation unit 19.

  The configuration of this apparatus is almost the same as the second and third configurations, but the pattern imaging unit 14 generates a binary gray coded pattern image including two images of a vertical stripe pattern and a horizontal stripe pattern, and outputs the projection output / image. And a step of determining corresponding points between the pattern image and the pattern captured image using a pattern captured image obtained by projecting and capturing the binary gray coded pattern image. Projection apparatus 11 and imaging apparatus that can eliminate the generation of a matching pattern in a pattern image and the matching process between the matching pattern and the pattern captured image, which are necessary when detecting one point, thereby reducing the amount of calculation processing The difference is that 12 calibrations with high accuracy are realized.

  The entire process of the camera / projector calibration apparatus of this example is the same as the process of the first configuration shown in FIG. The processes in the plane area extraction unit 16, the plane projection matrix calculation unit 18 and the parameter estimation unit 19 are the same as the plane area extraction unit 16, the corresponding coordinate extraction unit 17, the plane projection matrix calculation unit 18 and the parameter setting in the case of the first configuration. Since the processing is the same as that of the estimation unit 19, detailed description thereof is omitted here.

  Further, since the processing in the pattern extraction unit is the same as that in the second and third configurations and the pattern extraction unit, detailed description thereof is omitted here.

  A processing flow performed in the pattern imaging unit 14 will be described. When activated, the pattern imaging unit 14 first generates a predetermined pattern image. Here, the predetermined pattern is, for example, that two colors are arranged as a striped pattern vertically and horizontally on the image so that the color change in the time series of the projection surface after the projection output is expressed by a binary gray code. FIG. 5 is a plurality of binary gray coded pattern images.

  Subsequently, the generated pattern images are transferred to the projection device 11 one by one and a projection output command is issued. When a projection output completion signal is received from the projection device 11, an imaging command is issued to the imaging device 12. When the imaging device 12 receives the imaging command, the imaging device 12 performs imaging, sequentially saves the captured pattern images to the storage device 13, and outputs an imaging completion signal to the pattern imaging unit 14.

  The pattern imaging unit 14 receives the imaging completion signal from the imaging device 12 and ends the process.

  Next, the reasoning performed by the corresponding coordinate calculation unit 17 will be described with reference to FIG. The corresponding coordinate calculation unit 17 first receives all plane IDs and all image coordinates in the plane area respectively corresponding to the plane ID from the plane area extraction unit 16 (S71).

  Next, the pattern image and the pattern captured image are all read from the storage device 13 (S72). Subsequently, binarization is performed on all the pattern-captured images using the two colors used in the pattern image (S73).

  Subsequently, a spatially coded image in which the plurality of binarized images are arranged in the pattern image projection order is generated (S74), and the gray code in each pixel of the pattern captured image is restored from the spatially coded image. (S75) Using the restored gray code and the binary gray code represented by the pattern image (S76), a corresponding point corresponding to the feature point is determined on the plane area of the pattern captured image. This corresponding point determination process is performed for at least four points for each plane ID, and it is determined whether all plane IDs have been processed (S77). If NO, one plane ID is selected (S78).

  Thereafter, it is determined whether the number of designated image coordinates has been processed (S79). If NO, one image coordinate on the plane ID is selected (S80), and then the gray code of the image coordinate and the binary gray code of the pattern image are selected. Are matched points, and the coordinates are stored as corresponding coordinates in the storage device 13 (S81).

  Further, it is performed for at least two plane IDs (Yes in S77). Finally, the entire plane ID, the image coordinates on the plane ID, and the corresponding coordinates of the image coordinates are transferred to the plane projection matrix calculation unit 18 and the process is terminated (S82).

[First Embodiment]
Next, the first embodiment of the present invention will be described with reference to specific data. First, a screen 20 composed of at least two planes in a three-dimensional space, a projection device 11 for projecting and outputting a projection image input to the screen 20, and a projection for capturing a scene including the projection surface of the projection image Assume that the devices 11 are arranged as shown in FIGS.

Here, f ₁ and f ₂ indicate focal lengths for determining the internal parameters of the projection device 11, T is a relative position of the projection center of the imaging device 12 from the projection center of the projection device, and R is a liquid crystal in the projection device 11. The relative orientation of the imaging surface such as the CCD inside the imaging device 12 is shown with respect to the panel surface. In the present embodiment, a projector / camera calibration apparatus for determining the f ₁ , f ₂ and T, R will be described below.

  In the camera / projector calibration apparatus, the pattern imaging unit 14 is first activated. When activated, the pattern imaging unit 14 first generates a predetermined pattern image. In this example, it is assumed that a checkered pattern expressed in black is patterned on a white background as shown in FIG.

  Subsequently, the generated pattern image is projected and output by the projection device 11. Subsequently, the image pickup device 12 picks up an image, saves it as a pattern picked-up image in the storage device 13, and ends the processing. In this example, it is assumed that an image as illustrated in FIG. 10 is captured as a pattern captured image.

  Subsequently, the pattern extraction unit 15 is activated. The pattern extraction unit 15 first reads the pattern captured image from the storage device 13. Next, binarization is performed on the pixel values of the read pattern image. In this example, as a result of binarization using a predetermined threshold value, a binarized image as shown in FIG. 11 is obtained.

  Subsequently, a thinning process is performed on the binarized image. In this example, the line is thinned as shown in FIG. Finally, the thinned image is transferred as an extraction pattern image to the planar area extraction unit 16 and the process is terminated.

  Subsequently, the planar area extraction unit 16 is activated. The plane area extraction unit 16 detects all intersections of a plurality of straight lines expressed in the extraction pattern image. In this example, it is assumed that an intersection is obtained at a position indicated by a circle in FIG. 13 by applying a corner detection filter to the extracted pattern image.

  Subsequently, the obtained plurality of intersection groups are divided into quadrangular elements. In this example, the following description will be made on the assumption that a part of FIG. 12 is divided into squares P, Q, R, S, T, U indicated by thick lines in FIG.

  Subsequently, one quadrangle is selected from the divided quadrangles, and when the number of overlapping pixels between the four sides constituting the quadrangle and the straight line in the extracted pattern image exceeds a predetermined threshold value, the quadrangular region in the space is defined as the quadrangle. The square plane to represent. In this example, among P, Q, R, S, T, and U, Q, R, S, T, and U have overlapping pixels of four sides of a rectangle and a straight line in the extracted pattern image that are equal to or less than a threshold value. Assume that S is determined to be a square plane.

  Subsequently, adjacent square planes are connected and integrated, and a unique plane ID for individually identifying each plane is assigned to form a plane area. For example, in the example of FIG. 13, clustering is performed using a change in the area of a square, that is, an area difference between adjacent squares as a threshold, and P and S are integrated as the same area. As a result of applying this processing to all the rectangular elements, two planar regions are obtained as shown in FIG.

  Finally, the entire plane ID and all image coordinates of the plane area respectively corresponding to the plane ID are transferred to the corresponding coordinate calculation unit 17 and the process is terminated.

In this example, if the white plane ID in FIG. 14 is 1 and the light gray plane ID is 2,
{Plane ID: image coordinates} = {1: (1,1), (1,2), (2,1), (2,2) ...}, {2: (10,10), (10,11 ), (11.10), (11,11)...} Is transferred.

Subsequently, the corresponding coordinate calculation unit 17 is activated. In the corresponding coordinate calculation unit 17, first, the entire plane ID from the plane region extraction unit 16 and all image coordinates in the plane region respectively corresponding to the plane ID {plane ID: image coordinates} = {1: (1,1), (1,2), (2,1), (2,2)...}, {2: (10,10), (10,11), (11.10), (11,11).

  Next, the pattern image and the pattern captured image are read out from the storage device 13, and then the coordinates of feature points that are corners and intersections on the image are extracted from the pattern captured image. As the feature point, the intersection of the straight lines obtained by the plane area extraction unit 16 may be used as it is.

In this example, all plane IDs from the plane area extraction unit 16 received as feature points and all image coordinates in the plane areas respectively corresponding to the plane IDs {plane ID: image coordinates} = {1: (1,1) , (1,2), (2,1), (2,2) ...}, {2: (10,10), (10,11), (11.10), (11,11) ...} are selected Shall be.

Next, a matching pattern is created using the feature points on the planar area of the pattern captured image and the pixel values in the vicinity of the feature point coordinates, and corresponding points corresponding to the feature points in the pattern image are used using the matching pattern. decide. FIG. 15 shows a result of creating a matching pattern for the image coordinates (1, 1) of the first feature point of the plane ID 1 and determining corresponding points from the pattern image using the matching pattern.

  Here, a region indicated by a white frame in FIG. 15 indicates a collation pattern, and (2, 3) is determined as a corresponding point corresponding to a feature point in the pattern image by using the collation pattern. Show. Actually, for example, each line of the grid pattern is made to be the only type of broken line, or a pattern that becomes a marker is embedded in the grid pattern so that it can be easily distinguished from other feature points. The association is realized using.

  This corresponding point determination process is performed on at least four or more of the feature point coordinates belonging to each plane ID. Further, it is performed for at least two plane IDs. Finally, the entire plane ID, the image coordinates on the plane ID, and the corresponding coordinates of the image coordinates are transferred to the plane projection matrix calculation unit 18 and the process is terminated. In this example, the feature point coordinates and corresponding points on the plane ID are determined as shown in FIG. 16, and transferred to the plane projection matrix.

  Subsequently, the planar projection matrix calculation unit 18 is activated. When the plane projection matrix is activated, it first receives all plane IDs, image coordinates on the plane ID, and corresponding coordinates of the image coordinates from the corresponding coordinate calculation unit 17. In this example, {plane ID: image coordinates: corresponding coordinates} = {1: (1,1), (1,2), (2,1), (2,2) ... :( 2,3), (2 , 4), (3,3), (3,4)…}, {2: (10,10), (10,11), (11.10), (11,11)…: (12,11), (12,12), (13,11), (13,12).

Subsequently, a plane projection matrix H is calculated from four or more corresponding points between the pattern image and the pattern captured image for each received plane ID. In this example, the plane projection matrix H ₁ corresponding to the plane ID ₁ and the plane projection matrix H ₂ corresponding to the plane ID ₂ are obtained from the relationship of the expression (1) by the following expressions, respectively.

Finally, the plane ID and the plane projection matrices H ₁ and H ₂ corresponding to the plane ID are transferred to the parameter estimation unit 19 and the process is terminated.

Finally, the parameter estimation unit 19 is activated. When the parameter estimation unit 19 is activated, it first receives the plane ID and the plane projection matrices H ₁ and H ₂ corresponding to the plane ID from the plane projection matrix estimation unit 18.

  Subsequently, the internal characteristics A and A ′, the relative position T, and the posture R of each of the projection device 11 and the imaging device 12 are obtained from at least two or more received planar projection matrices using Equation (4) and Equation (3). Calculate and finish the process.

[Second Embodiment]
Next, an embodiment of the second configuration of the present invention will be described based on specific data. In the present embodiment, two pattern captured images obtained by capturing the projection output results of two images of a vertical stripe pattern and a horizontal stripe pattern generated by the pattern imaging means (imaging unit) 14 at the time of pattern extraction in the first embodiment are used. The case will be described.

  In the first embodiment, an extraction pattern is binarized by a pattern extraction unit (extraction unit) 15 using a pattern captured image obtained by imaging a projection output result of a lattice pattern image generated by the pattern imaging unit (imaging unit) 14. In this case, depending on the binarization threshold, the pattern drawn on the screen itself and the color of the screen itself may be included in the extracted pattern image, and the calibration result includes an error. End up.

  In the second embodiment, the difference between two pattern captured images obtained by capturing the projection output results of two images of the vertical stripe pattern and the horizontal stripe pattern generated by the pattern imaging unit 14 is extracted, and the pattern is extracted from the difference image. And By doing so, an extracted pattern image can be generated without being affected by the pattern drawn on the screen itself or the color of the screen itself.

  In the present embodiment, the pattern imaging unit 14 of the first embodiment generates two images of a vertical stripe pattern and a horizontal stripe pattern and performs projection output and imaging, and the pattern extraction unit 15 performs a difference between both images. A step for generating an extracted pattern image from the image is newly provided.

  The arrangement of the projection device 11, the imaging device 12, and the screen 20 in the camera / projector calibration device of this example is the same as that of the first embodiment shown in FIG.

  Processing in the pattern imaging unit (unit) 14 in the present embodiment will be described. When the pattern imaging unit (imaging unit) 14 is activated, it first generates a predetermined pattern image. Here, the predetermined pattern is, for example, a vertical stripe pattern in which a pixel value of one color is expressed by a pixel value different from the pixel value, and a pixel value of one color is the pixel value. This is a pattern expressed by two images of a horizontal stripe pattern formed by interposing a gap horizontally on pixels expressed by different pixel values.

  Subsequently, the generated pattern image is projected and output by the projection device 11. Subsequently, the imaging device 12 captures an image, saves it as a pattern captured image in the storage device 13, and ends the processing. In this example, an image as shown in FIG. 17 is obtained as a pattern captured image.

  Next, processing in the pattern extraction unit 15 will be described. First, the pattern extraction unit 15 reads two pattern captured images from the storage device 13. Next, a difference image between the two images is generated. In this example, a difference image as shown in FIG. 18 is generated.

  Subsequently, binarization is performed on the pixel values of the generated difference image to generate binary images. Subsequently, expansion / reduction processing is performed on the binarized image in order to fill in the blurred / discontinuous line segments constituting the pattern in the captured image generated by the difference processing and binarization processing.

  Subsequently, the image after the expansion / reduction process is thinned, transferred to the planar area extraction unit 16 as an extraction pattern image, and the process ends.

  Since the following processing is the same as that of the first embodiment, a detailed description thereof is omitted here.

[Third Embodiment]
Next, an embodiment of the third configuration of the present invention will be described based on specific data. In the present embodiment, two images obtained by imaging the projection output results of two images of a vertical stripe pattern and a horizontal stripe pattern generated by the pattern imaging means (imaging unit) 14 at the time of pattern extraction in the first embodiment, and a projection in which nothing is projected A case will be described in which a total of three pattern-captured images in which the surface state is captured are used.

  In the first embodiment, a pattern captured image obtained by capturing a projection output result of a lattice pattern image generated by the pattern imaging unit (imaging unit) 14 is binarized by the pattern extraction unit to generate an extraction pattern. As described above, depending on the binarization threshold, the pattern drawn on the screen itself and the color of the screen itself may be included in the extracted pattern image, and an error is included in the calibration result.

  In addition, when the shape of the projection surface is complicated, the light beam from the projection device 11 is influenced by the secondary reflection that reflects to the position of the other projection surface in addition to the position of the desired projection surface, as shown in the upper part of FIG. When a pixel corresponding to the projection pattern image is extracted from the captured image, a deviation from the original lattice position may occur, and an error is included in the calibration result.

  In the third embodiment, two pattern captured images obtained by imaging the projection output results of two images of a vertical stripe pattern and a horizontal stripe pattern generated by the pattern imaging means (imaging unit) 14 and a state of a projection plane on which nothing is projected A difference is taken between each of the captured images obtained by picking up images and a pattern is extracted from the difference image. By doing so, an extracted pattern image can be generated without being affected by the pattern drawn on the screen itself or the color of the screen itself.

  Also, an extraction pattern that is not affected by secondary reflection as shown in the lower part of FIG. 25 by generating an extraction pattern image by taking the logical sum of the two images that have been subjected to binarization, expansion / reduction processing, and thinning of the difference image An image can be generated.

  In this embodiment, the pattern imaging unit of the first embodiment generates two images of vertical stripes and horizontal stripes and performs projection output and imaging, and the pattern extraction unit binarizes and expands both images. A step of generating an extraction pattern image from a logical sum image that has been reduced and thinned is newly provided.

  The arrangement of the projection device 11, the imaging device 12, and the screen 20 in the camera / projector calibration device of this example is the same as that of the first embodiment shown in FIG. Processing in the pattern imaging unit (imaging unit) 14 in the present embodiment will be described.

  When the pattern imaging unit (imaging unit) 14 is activated, it first generates a predetermined pattern image. Here, the predetermined pattern is, for example, a vertical stripe pattern in which a pixel value of one color is expressed by a pixel value different from the pixel value, and a pixel value of one color is the pixel value. This is a pattern expressed by two images of a horizontal stripe pattern formed by interposing a gap horizontally on pixels expressed by different pixel values.

  Subsequently, the generated pattern image is projected and output by the projection device 11. Subsequently, the imaging device 12 captures the pattern projection result and the state of the projection surface when the pattern projection is not performed, saves it in the storage device 13 as a pattern captured image, and ends the processing. In this example, the image shown in FIG. 19 is obtained in addition to the image similar to FIG. 17 as the pattern captured image.

  Next, processing in the pattern extraction unit 15 will be described. First, the pattern extraction unit 15 reads three pattern captured images from the storage device 13. Next, a difference image from the image of FIG. 19 is generated for each of the images of FIG.

  Subsequently, binarization is performed on the pixel values of the generated difference image to generate a binarized image. Subsequently, expansion / reduction processing is performed on the binarized image in order to fill in the blurred / discontinuous line segments constituting the pattern in the captured image generated by the difference processing and binarization processing.

  Subsequently, thinning is performed on the image after the expansion / reduction processing. Thereafter, the logical sum of the two thinned images is taken to generate an extracted pattern image. Finally, the extracted pattern image is transferred to the planar area extracting unit 16 and the process is terminated. Since the following processing is the same as that of the first embodiment, a detailed description thereof is omitted here.

[Fourth Embodiment]
Next, an embodiment of the fourth configuration of the present invention will be described based on specific data. In the present embodiment, a projection output result of a binary gray coded pattern image including two images of a vertical stripe pattern and a horizontal stripe pattern generated by the pattern imaging means (imaging unit) 14 when determining corresponding points in the first embodiment is captured. A case where a plurality of pattern captured images are used will be described.

  In the first, second, and third embodiments, the corresponding coordinate calculation means (calculation unit) 17 uses the pixel values of the pattern captured image to generate from the area near the extracted feature point that becomes the corner or intersection on the image. The case where a matching pattern is used and corresponding points corresponding to the feature points on the pattern image are determined using the matching pattern has been described. In this case, one matching pattern is moved over the entire area of the pattern image. Processing time is required because it is necessary to search for points.

  In addition, if there is a pattern similar to the matching pattern at a place other than the corresponding point when searching for the corresponding point, an incorrect correspondence occurs, and as a result, an error is included in the calibration result.

  In the fourth embodiment, the corresponding coordinate calculation unit (calculation unit) 17 captures the projection output result of the binary gray coded pattern image including two images of the vertical stripe pattern and the horizontal stripe pattern generated by the pattern imaging unit. Corresponding coordinates are determined by the spatial coding method from the pattern captured image.

  By doing so, it is possible to calibrate the projection apparatus 11 and the imaging apparatus 12 that can reduce the amount of calculation processing with high accuracy. In this embodiment, the pattern imaging unit 14 of the second and third embodiments generates a binary gray coded pattern image including two images of a vertical stripe pattern and a horizontal stripe pattern, and performs projection output and imaging. In the corresponding coordinate calculation unit 17, a step of determining the corresponding coordinates from the pattern captured image by the spatial coding method is newly provided.

  The arrangement of the projection device 11, the imaging device 12, and the screen 20 in the camera / projector calibration device of this example is the same as that of the first embodiment shown in FIG.

  Processing in the pattern imaging unit in the present embodiment will be described. When the pattern imaging unit (imaging unit) 14 is activated, it first generates a predetermined pattern image. In this example, the two colors are arranged in a striped pattern vertically and horizontally on the image so that the color change in the time series of the projection surface after the projection output as shown in FIG. 20 is expressed by the binary gray code. It is a pattern expressed by a plurality of binary gray coded pattern images.

  Here, for simplification of the following description, it is assumed that the binary code pattern images shown by the pattern images 1 to 3 in FIG. 21 are used as the pattern image instead of the binary gray coded pattern image.

  Subsequently, the generated pattern images 1 to 3 are projected and output by the projection device 11. Subsequently, the image pickup device 12 picks up an image, saves it as a pattern picked-up image in the storage device 13, and ends the processing. In this example, it is assumed that images such as pattern captured images 1 to 3 in FIG. 21 are obtained as pattern captured images.

  Next, processing performed by the corresponding coordinate calculation unit 17 will be described. In the corresponding coordinate calculation unit 17, first, the entire plane ID and all image coordinates in the plane region respectively corresponding to the plane ID are received from the plane region extraction unit 16.

  Next, the pattern images 1 to 3 and the pattern captured images 1 to 3 are all read from the storage device 13. Subsequently, binarization is performed on all of the pattern captured images using the two colors used in the pattern image.

  Subsequently, the plurality of binarized images are arranged in the imaging order of pattern images 1 to 3 to generate a spatial code image in which pattern captured images as shown in FIG. 21 are arranged in time series, and the spatial code image is used to generate the spatial code image. The gray code at each pixel of the pattern captured image is restored.

  When the gray code in both directions of the vertical stripe and horizontal stripe can be restored, the pattern image and the pattern captured image are associated with each other as shown in the bottom of FIG. Determined from the coordinates of the attached point.

  This corresponding point determination process is performed on at least four points for each plane ID. Further, it is performed for at least two plane IDs. Finally, the entire plane ID, the image coordinates on the plane ID, and the corresponding coordinates of the image coordinates are transferred to the plane projection matrix calculation unit 18 and the process is terminated.

  Although the present invention has been specifically described above based on the embodiments, it is needless to say that the present invention is not limited to the above-described embodiments and can be variously modified without departing from the gist thereof.

  For example, the pattern image generation unit and the planar area extraction unit have described the method of projecting a square pattern, but a plurality of pattern images are generated so that a polygon is configured in the pattern generation image, A planar area may be extracted using the polygon.

  The above-described apparatus of the present invention shown in FIG. 1 can be realized by a computer and a program, or the processing steps of the flowchart shown in FIG. 2 are configured by a computer program, and the program is executed by the computer. Needless to say, the program for realizing the function by the computer or the program for executing the processing steps by the computer can be read by a recording medium such as a flexible disk, a CD, a DVD, It can be recorded on an MO, ROM, memory card, removable disk, etc. for storage or distribution.

  It is also possible to provide the above program through a network such as the Internet or electronic mail. The present invention can be implemented by installing the program from these recording media into a computer or by downloading the program from a network and installing the program into the computer.

It is a block diagram for demonstrating the structural example of the calibration apparatus by this invention. It is a figure for demonstrating the processing flow in the calibration apparatus by this invention. It is a figure for demonstrating the processing flow in the calibration apparatus by this invention. It is a figure for demonstrating the processing flow in the calibration apparatus by this invention. It is a figure for demonstrating the processing flow in the calibration apparatus by this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the process in the camera projector correction apparatus by one Embodiment of this invention. It is a block diagram for demonstrating a prior art. It is a figure for demonstrating the planar projection matrix used with the camera projector correction apparatus by one Embodiment of this invention. It is a figure for demonstrating the corresponding coordinate extraction part process flow in the calibration apparatus by this invention. It is a figure explaining the influence of the secondary reflection in the calibration apparatus by this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Projection apparatus 12 ... Imaging device 13 ... Memory | storage device 14 ... Pattern imaging part 15 ... Pattern extraction part 16 ... Planar area extraction part 17 ... Corresponding coordinate calculation part 18 ... Planar projection matrix calculation part 19 ... Parameter estimation part

Claims (21)

Pattern imaging means for generating a predetermined pattern image, projecting and outputting the pattern image as a projection image by a projection apparatus, and imaging the state of the projection plane at that time as a pattern captured image by the imaging apparatus;
A pattern extraction unit that extracts pixels corresponding to the pattern image from the pattern captured image using pixel values of the image to form an extracted pattern image;
A plane area extracting means for extracting a plane area in the projection plane from the extracted pattern image using the relationship between the geometry of the extracted pixels in the extracted pattern image and the geometry in the pattern image;
Corresponding coordinate calculation means for performing processing for determining at least four or more points to be associated with the pattern area on the plane area extracted by the plane area extraction means over all the extracted plane areas;
A plane projection matrix calculating means for calculating, for each plane region, a plane projection matrix expressing the relationship between points existing on the plane from the corresponding coordinates;
Parameter estimation means for calculating internal characteristics and relative positions and orientations of the projection device and the imaging device from at least two or more planar projection matrices;
A calibration apparatus comprising: In the pattern imaging means,
The predetermined pattern image to be generated and projected and output is a checkered pattern in which a pixel value of one color is formed with a gap vertically and horizontally on a pixel represented by a pixel value different from the pixel value,
The calibration device according to claim 1. In the pattern imaging means,
A predetermined pattern image to be generated and projected is a vertical stripe pattern in which a pixel value of one color is expressed by a pixel value different from the pixel value and a pixel value of one color is formed with a vertical gap. At least two images of a horizontal stripe pattern formed with a gap horizontally on a pixel represented by a pixel value different from the value,
The calibration device according to claim 1. In the pattern imaging means,
The predetermined pattern image to be generated and projected is arranged in a striped pattern in two colors vertically and horizontally so that the time-series color change of the projection plane after the projection output is expressed in binary gray code. A plurality of binary gray coded pattern images,
The calibration device according to claim 1. In the pattern extraction means,
From the pattern captured image, the pixel values of the image are binarized, and then the binarized image has a line width without losing connectivity to each connected figure included in the image. Extracting a pixel corresponding to the pattern image by thinning to one line figure to obtain an extracted pattern image;
The calibration apparatus according to claim 1 or 2, characterized by the above. In the pattern extraction means,
From two pattern imaged images obtained by imaging the state of a projection plane obtained by projecting and outputting a pair of vertical and horizontal stripe patterns of the vertical stripe pattern and the horizontal stripe pattern or the binary gray coded pattern image,
A difference image is generated by calculating an absolute value of a pixel value difference for each pixel of both images, and binarization is performed on the pixel value of the difference image,
An expansion process for increasing the connected component outline pixel by one layer on the binarized image and a reduction process for removing the connected component outline pixel and reducing it by one layer are performed a predetermined number of times.
Subsequently, the pixels corresponding to the pattern image are thinned by thinning the line figure to one line figure without losing connectivity to each connected figure included in the image with respect to the expanded / reduced image processing. Extracting it into an extracted pattern image,
The calibration apparatus according to claim 1, wherein: In the pattern extraction means,
From at least two pattern captured images obtained by imaging the state of the projection plane obtained by projecting and outputting a pair of vertical and horizontal striped patterns of the vertical striped pattern and the horizontal striped pattern or the binary gray coded pattern image, a pixel for each pattern captured image Binarize the value,
For each of the two binarized images, expansion processing for increasing the contour pixel of the connected component by one layer outward and reduction processing for removing the contour pixel of the connected component and reducing it by one layer are determined in advance. Several times,
Subsequently, for each of the two images subjected to the expansion / reduction processing, the pattern image is thinned by thinning the line figure to one line figure without losing connectivity to each connected figure included in the image. Extract the relevant pixels,
Subsequently, the logical sum operation result of both images is used as an extraction pattern image,
The calibration apparatus according to claim 1, wherein: In the planar area extracting means,
Receiving the extracted pattern image from the pattern extracting means;
Detect all intersections between a plurality of straight lines described in the extracted pattern image,
Divide the extracted intersections into quadrilaterals with each intersection as a vertex,
When the number of overlapping pixels between each side of the quadrilateral and the straight line in the extracted pattern image exceeds a predetermined threshold, the process of making a rectangular plane is performed over all the divided quadrangles,
A unique plane ID that identifies adjacent planes and identifies each plane individually in the combined plane region;
Storing the plane ID and the image coordinates in the plane area corresponding to the plane ID;
The calibration apparatus according to claim 1, wherein: In the corresponding coordinate calculation means,
Extracting at least four or more feature points that are corners or intersections on the image using pixel values of the pattern-captured image from points on the plane region extracted by the plane region extraction means,
Create a matching pattern using the pixel values of the area including the feature points,
Determining a corresponding point corresponding to the feature point on the pattern image using the matching pattern;
The calibration apparatus according to claim 1, wherein: In the corresponding coordinate calculation means,
Binarization with the two colors used in the binary gray coded pattern image is performed on the pattern captured image;
Generating a spatial code image in which the plurality of binarized images are arranged in a pattern image projection order;
Restoring the gray code of each pixel of the pattern-captured image from the spatial code image;
Determining at least four or more corresponding points corresponding to points on the planar region of the pattern image using the restored gray code and the binary gray code expressed by the pattern image;
The calibration apparatus according to claim 1, characterized in that: A pattern imaging process of generating a predetermined pattern image, projecting and outputting the pattern image as a projection image by the projection apparatus, and then imaging the state of the projection plane at that time as a pattern captured image by the imaging apparatus;
A pattern extraction process of extracting a pixel corresponding to the pattern image from the pattern captured image using a pixel value of the image and then making an extracted pattern image;
A planar area extraction process for extracting a planar area in the projection plane from the extracted pattern image using the relationship between the geometry of the extracted pixels in the extracted pattern image and the geometry in the pattern image;
Corresponding coordinates for performing processing for determining at least four points to be associated with the pattern area on the plane area extracted by the plane area extracting means, and performing the process over all the extracted plane areas Calculation process,
A plane projection matrix calculation process for calculating, for each plane area, a plane projection matrix expressing a relationship between points existing on a plane from the corresponding coordinates obtained in the corresponding coordinate calculation process;
A parameter estimation process for calculating internal characteristics and relative positions and orientations of the projection apparatus and the imaging apparatus from at least two or more planar projection matrices;
A calibration method comprising the steps of: In the pattern imaging process,
The predetermined pattern image to be generated and projected and output is a checkered pattern in which a pixel value of one color is formed with a gap vertically and horizontally on a pixel represented by a pixel value different from the pixel value,
The calibration method according to claim 11. In the pattern imaging process,
A predetermined pattern image to be generated and projected is a vertical stripe pattern in which a pixel value of one color is expressed by a pixel value different from the pixel value and a pixel value of one color is formed with a vertical gap. At least two images of a horizontal stripe pattern formed with a gap horizontally on a pixel represented by a pixel value different from the value,
The calibration method according to claim 11. In the pattern imaging process,
The predetermined pattern image to be generated and projected is arranged in a striped pattern in two colors vertically and horizontally so that the time-series color change of the projection plane after the projection output is expressed in binary gray code. A plurality of binary gray coded pattern images,
The calibration method according to claim 11. In the pattern extraction process,
After binarization is performed on the pixel values of the image from the pattern-captured image, each connected figure included in the image is then lined without losing connectivity to the binarized image. Extracting a pixel corresponding to the pattern image by thinning to a line figure having a width of one to obtain an extracted pattern image;
The proofreading method according to claim 11 and 12. In the pattern extraction process,
From two pattern imaged images obtained by imaging the state of a projection plane obtained by projecting and outputting a pair of vertical and horizontal stripe patterns of the vertical stripe pattern and the horizontal stripe pattern or the binary gray coded pattern image,
After calculating the absolute value of the pixel value difference for each pixel of both images and generating a difference image, after binarizing the pixel value of the difference image,
An expansion process for increasing the connected component outline pixel by one layer on the binarized image and a reduction process for removing the connected component outline pixel and reducing it by one layer are performed a predetermined number of times.
Subsequently, the pixels corresponding to the pattern image are thinned by thinning the line figure to one line figure without losing connectivity to each connected figure included in the image with respect to the expanded / reduced image processing. Extracting it into an extracted pattern image,
The calibration method according to claim 11, characterized in that: In the pattern extraction process,
From at least two pattern captured images obtained by imaging the state of the projection plane obtained by projecting and outputting a pair of vertical and horizontal striped patterns of the vertical striped pattern and the horizontal striped pattern or the binary gray coded pattern image, a pixel for each pattern captured image After binarizing the value,
For each of the two binarized images, expansion processing for increasing the contour pixel of the connected component by one layer outward and reduction processing for removing the contour pixel of the connected component and reducing it by one layer are determined in advance. Several times,
Subsequently, for each of the two images subjected to the expansion / reduction processing, the pattern image is thinned by thinning the line figure to one line figure without losing connectivity to each connected figure included in the image. After extracting the relevant pixels,
Subsequently, the logical sum operation result of both images is used as an extraction pattern image,
The calibration method according to claim 11, characterized in that: In the planar region extraction process,
After receiving the extracted pattern image from the pattern extraction process,
Detect all intersections between a plurality of straight lines described in the extracted pattern image,
After dividing all the extracted intersections into rectangles with each intersection as a vertex,
When the number of overlapping pixels between each side of the quadrilateral and the straight line in the extracted pattern image exceeds a predetermined threshold, the process of making a rectangular plane is performed over all the divided quadrangles,
Then, after consolidating adjacent square planes and giving each plane of the integrated plane area a unique plane ID identifying each plane,
Storing the plane ID and the image coordinates in the plane area corresponding to the plane ID;
The calibration method according to claim 11, wherein: In the corresponding coordinate calculation process,
After extracting at least four or more feature points that are corners or intersections on the image using pixel values of the pattern-captured image from points on the plane region extracted in the plane region extraction process,
Create a matching pattern using the pixel values of the area including the feature points,
Then, determining corresponding points corresponding to the feature points on the pattern image using the matching pattern;
The calibration method according to claim 11, wherein: In the corresponding coordinate calculation process,
After performing binarization with the two colors used in the binary gray coded pattern image on the pattern captured image,
After generating a spatial code image in which the plurality of binarized images are arranged in the pattern image projection order,
Restoring the gray code of each pixel of the pattern-captured image from the spatial code image;
Determining at least four or more corresponding points corresponding to points on the planar region of the pattern image using the restored gray code and the binary gray code expressed by the pattern image;
The calibration method according to any one of claims 1, 4, 6, 7, and 8. 21. A calibration program characterized in that the process in the calibration method according to claim 11 is a program for causing a computer to execute the process.