JP2010033397A - Image composition device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image composition device virtually erasing an image of an object included in an actually captured image captured by a camera from the actually captured image by use of technology of AR(Augmented Reality). <P>SOLUTION: This image composition device 1 sets a sampling area by use of a position of a copy marker 2, and generates a texture from an image included in the sampling area. The image composition device 1 combines the actually captured image with a virtual plane with seamless texture added thereto in accordance with a position of a paste marker 3, to virtually erase the image of the object 4 from the actually captured image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to augmented reality technology for synthesizing computer graphics with live-action images, and more particularly, to an object that interferes when synthesizing computer graphics using augmented reality technology. The present invention relates to a technology for virtually erasing an image from a photographed image.

  Augmented Reality (AR) is a technique for assisting human perception and realizing a realistic image experience by combining computer graphics (CG) with live-action images. The technology used.

  As described in Non-Patent Document 1, Non-Patent Document 2, and the like, in a general computer device that implements augmented reality, a photographed image is captured by a camera so as to include a marker on which a pattern is printed. The apparatus calculates the position and inclination of the marker from the pattern of the marker, and synthesizes the CG with the actual image captured by the camera according to the position of the marker and the inclination of the marker.

  However, if the size of the CG is small with respect to the marker, the marker appears to protrude from the CG after the CG is synthesized. In Patent Document 1, a marker concealing image for deleting the marker is previously stored. When the camera synthesizes the CG at the marker position in the captured image, the marker appears to protrude from the CG by synthesizing the marker concealing image at the marker position of the captured image and then combining the CG. An invention for preventing this is disclosed.

  However, not only the marker is an image that is troublesome to see out of the CG, but also when the CG is combined instead of the object (for example, a desk) included in the photographed image, the image of the object photographed in the photographed image is captured from the CG. It may appear out of the way.

  For example, if a desk CG is synthesized on a wide floor with nothing, using augmented reality, the desk CG appears to be naturally synthesized on the floor, but there is a real desk or chair near the marker. When an object to be present exists, if a part of the image of the actual object remains in the image after combining the CG of the desk, an unnatural impression is given.

  If an object such as a real desk or chair can be moved, there is no problem, but it is often difficult to move an actual object. Therefore, when synthesizing CG with a live-action image, it is a hindrance when synthesizing CG. It is desirable that the image of the object to be can be virtually erased from the photographed image.

Kato, H., Billinghurst, M. (1999) Marker Tracking and HMD Calibration for a video-based Augmented Reality Conferencing System.In Proceedings of the 2nd International Workshop on Augmented Reality (IWAR 99) .October, San Francisco, USA. Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., Tachibana, K. (2000) Virtual Object Manipulation on a Table-Top AR Environment.In proceedings of the International Symposium on Augmented Reality, pp. 111-119, (ISAR 2000), Munich, Germany. JP 2005-234757 A

  Therefore, the present invention uses an augmented reality (AR) technology to synthesize an image of a target object included in a photographed image taken by a camera from the photographed image virtually. It is an object to provide an apparatus and an image composition method.

  A first invention that solves the above-described problem is stored in a marker image used in augmented reality (AR) and is included in a photographed image taken by a camera using the marker pattern. From the marker recognition means for recognizing the marker and the marker pattern recognized by the marker recognition means, the marker coordinate system of the marker recognized by the marker recognition means is calculated and converted from each marker coordinate system to the camera coordinate system. Using a transformation matrix calculation means for calculating a matrix and an offset value determined as the position of the marker (copy marker) for designating a copy source image, the XY plane of the marker coordinate system of the copy marker has a predetermined size. A sampling area is set, and a transformation matrix from the marker coordinate system to the camera coordinate system of the copy marker is used, and the photographed image included in the sampling area is The copy marker that arranges a virtual plane to be combined with the live-action image using a texture generation unit that generates a texture of a predetermined size from the source and a coordinate origin of the marker (paste marker) that specifies a paste destination An arrangement coordinate calculation means for calculating a coordinate point on the marker coordinate system, and a virtual plane of a predetermined size pasted by the texture generation means with the coordinate point calculated by the arrangement coordinate calculation means as a central point A virtual plane forming unit for forming the virtual plane, and an image synthesis unit for synthesizing the virtual plane formed by the virtual plane forming unit with the photographed image using a conversion matrix from a marker coordinate system to a camera coordinate system of the copy marker And an image synthesizing apparatus.

  According to the first invention, the paste marker is placed on a target object (for example, a desk) that is virtually erased from the photographed image, and an object (when used to virtually erase the image of the target object) For example, if the copy marker is placed on a floor surface, the virtual plane on which the texture generated from an image near the copy marker (for example, an image of the floor surface) is pasted is synthesized in the vicinity of the paste marker. Therefore, the image of the target object (for example, a desk) can be virtually erased from the photographed image.

  Furthermore, the second invention is the image composition device according to the first invention, wherein the arrangement coordinate calculation means uses the paste as a coordinate point on a marker coordinate system of the copy marker on which the virtual plane is arranged. An image synthesizing apparatus that calculates a point where a straight line passing through a coordinate origin of a marker and a coordinate origin of the camera intersects the same plane as an XY coordinate plane of the copy marker.

  According to the second invention, the copy in which the virtual plane is arranged at a point where a straight line passing through the coordinate origin of the paste marker and the coordinate origin of the camera intersects the same plane as the XY coordinate plane of the copy marker By using the coordinate point of the marker on the marker coordinate system, when the virtual plane is combined with the real image, the depth of the virtual plane with respect to the real image can be adjusted.

  For example, when the copy marker is placed on the floor surface, the XY plane in the marker coordinate system of the copy marker is the floor surface, so that an extension line from the camera viewpoint to the center point of the paste marker is the copy marker. If a virtual plane is formed around a point intersecting the XY plane of the marker coordinate system, the virtual plane is formed at a position where the sense of depth is adjusted from the viewpoint of the camera.

  Furthermore, a third invention is the image composition device according to the first invention or the second invention, wherein the transformation matrix calculation means is configured to determine a camera coordinate from a marker coordinate system of the marker recognized by the marker recognition means. In addition to the transformation matrix to the system, the transformation matrix between the marker coordinate system of the copy marker and the paste marker is computed and stored, and the transformation matrix between the marker coordinate system of the copy marker and the paste marker is computed and stored. Later, when the marker recognizing unit can detect only one of the copy marker or the paste marker from the photographed image, the conversion matrix computing unit converts the copy marker and the paste marker between the marker coordinate systems. An image synthesizing apparatus that uses a matrix to calculate a transformation matrix related to the marker that could not be detected from the photographed image. .

  According to the third invention, for example, when the image synthesizing apparatus processes a video, even if one of the copy marker and the paste marker is not included in the photographed image due to a human figure or the like, the process is executed. Will be able to.

  Furthermore, the fourth invention is an image composition method using augmented reality (AR), which uses a computer device image and a pattern printed on a marker used in augmented reality, Recognizing the marker included in the photographed image taken in step a, and calculating the marker coordinate system of the marker recognized by the marker recognizing means from the recognized marker pattern by the computer device. A step b for calculating a transformation matrix from the system to the camera coordinate system, and the computer apparatus using the offset value determined as the position of the marker (copy marker) for designating the copy source image; A sampling area of a predetermined size is set on the XY plane of the marker coordinate system, and a conversion matrix from the marker coordinate system to the camera coordinate system of the copy marker is used. Then, a step c for generating a texture of a predetermined size from pixels of the photographed image included in the sampling area, and the computer device uses the coordinate origin of the marker (paste marker) for specifying the paste destination A step d for calculating a coordinate point on the marker coordinate system of the copy marker for arranging a virtual plane to be combined with the photographed image, and the computer device having the coordinate point calculated in the step d as a center point; A step e of forming a virtual plane of a predetermined size to which the texture generated by the texture generation unit is pasted, and the computer device uses a conversion matrix from the marker coordinate system of the copy marker to the camera coordinate system, Synthesizing the virtual plane formed by the virtual plane forming means with the photographed image. Which is the image composition method according to claim.

  Furthermore, a fifth invention is the image composition method according to the fourth invention, wherein, in the step d, the computer apparatus uses coordinate points on the marker coordinate system of the copy marker on which the virtual plane is arranged. The image composition method is characterized in that a point where a straight line passing through the coordinate origin of the paste marker and the coordinate origin of the camera intersects the same plane as the XY coordinate plane of the copy marker is calculated.

  Furthermore, a sixth invention is the image composition method according to the fourth invention or the fifth invention, wherein in the step b, the computer device converts the marker from a marker coordinate system to a camera coordinate system. In addition to the matrix, the conversion matrix between the marker coordinate system of the copy marker and the paste marker is calculated and stored, and after the conversion matrix between the marker coordinate system of the copy marker and the paste marker is calculated and stored, the step When only one of the copy marker and the paste marker can be detected from the photographed image in a, the computer device calculates a conversion matrix between the marker coordinate systems of the copy marker and the paste marker in the step b. And calculating a transformation matrix related to the marker that could not be detected from the photographed image. An image synthesizing method for.

  According to the fourth to sixth inventions described above, the same effects as those of the first to third inventions can be obtained.

  As described above, according to the present invention, an image synthesizing device and an image that can virtually erase an image of a target object included in a photographed image captured by a camera from the photographed image using the AR technique. A synthesis method can be provided.

  From here, the image composition device 1 and the image composition method according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram for explaining a usage form of an image composition apparatus 1 according to the present invention.

  As shown in FIG. 1, a camera 1a that captures a photographed image is connected to an image composition device 1 that is realized by a computer device such as a personal computer or a server, and the image composition device 1 has a camera as a computer program. 3DC: 3 Dimensional Computer Graphics (3DGC: 3 Dimensional Computer Graphics) prepared in advance by detecting a marker printed with a special pattern from the image taken in 1a and matching the marker position to the detected marker position. In addition to a computer program that realizes a general augmented reality (AR) function for synthesizing graphics (AR), an image of the target object 4 (desk in FIG. 1) that becomes an obstacle when synthesizing 3DCG A computer program for realizing a function of virtually erasing from a photographed image using technology is implemented.

  In the present embodiment, the image composition device 1 generates a texture from an image included in a sampling region that is a partial region of a live-action image, forms a virtual plane on which the texture is seamlessly pasted, and forms By synthesizing the virtual plane at the designated position of the photographed image, the image of the target object is virtually erased from the photographed image.

  The markers used when virtually erasing the image of the target object from the photographed image are two markers, the copy marker 2 and the paste marker 3, and the copy marker 2 is a marker used for designating a sampling area. The paste marker 3 is a marker used for designating the position where the virtual plane is to be synthesized. The copy marker 2 and the paste marker 3 are general markers used in the AR technology, and do not have special technical characteristics. However, the copy marker 2 and the paste marker 3 are copied so that the image synthesizer 1 can recognize each marker. The patterns printed on the marker 2 and the paste marker 3 are different.

  As shown in FIG. 1, each of the camera 1a, the copy marker 2 and the paste marker 3 has a three-dimensional coordinate system (Coordinate). In the future, the coordinate system of the camera 1a will be referred to as a camera coordinate system. The coordinate system is referred to as a copy marker coordinate system, and the coordinate system of the paste marker 3 is referred to as a paste marker coordinate system.

  From here, the process in which the image composition device 1 virtually deletes the image of the target object from the actual image captured by the camera 1a will be described in detail. FIG. 2 is a flowchart for explaining processing when a target object is virtually erased from an image. This explanation also serves as explanation of an image composition method according to the present invention.

  First, the image synthesizing apparatus 1 acquires parameters necessary for performing a process of virtually erasing the image of the target object 4 from a real image captured by the camera 1a (S1 in FIG. 2). The method by which the image synthesizing apparatus 1 obtains the parameters may be arbitrary, and examples include input with a keyboard, reading from a file, and a combination thereof.

  FIG. 3 is a diagram for explaining an example of parameters required when the image composition device 1 performs a process of virtually erasing the image of the target object 4 from the actual image captured by the camera 1a.

  In FIG. 3, the parameters acquired by the image synthesizing apparatus 1 include, as items related to texture, the size of the texture generated by the image synthesizing apparatus 1 and the size of the sampling area for acquiring the texture image (here, the radius r1). ), An offset value from the center of the copy marker 2 to the center of the sampling area, and an overlap size for seamlessly processing the texture.

  Further, in FIG. 3, the parameters acquired by the image composition device 1 include the size of the virtual plane as an item related to the virtual plane. The size of the virtual plane depends on the shape of the virtual plane, and is described so that the shape of the virtual plane is uniquely determined. For example, if the shape of the virtual plane is a rectangle, the vertical and horizontal size (unit: pixels) of the rectangle is described as data for determining the size of the rectangle. Here, the shape of the virtual plane is a circle, and the size of the circle is As data to be determined, the image composition device 1 acquires a radius r2 (unit: pixel) of the virtual plane.

  Note that FIG. 3 is merely an example of parameters acquired by the image composition device 1 and, of course, differs depending on the contents of the image composition device 1.

  When the image synthesizing apparatus 1 obtains parameters necessary for performing a process of virtually erasing the image of the target object 4 from the actual image captured by the camera 1a, the image synthesizing device 1 reads the actual image captured by the camera in real time. The image synthesizing apparatus 1 acquires a real image to be processed (S2 in FIG. 2) by reading a file storing a real image taken by the camera 1a.

  FIG. 4 is a diagram for explaining an example of a live-action image. The real image shown in FIG. 4 includes a copy marker 2 and a paste marker 3 in addition to an image of the target object 4 to be erased (here, a desk image). In FIG. The copy marker 2 is placed on the floor on which the desk is installed.

  5A and 5B are diagrams for explaining the copy marker 2 and the paste marker 3. FIG. 5A shows the copy marker 2, and FIG. As shown in FIG. 5, the patterns printed by the copy marker 2 and the paste marker 3 are different. As shown in FIG. 5A, the pattern of the copy marker 2 is a square frame surrounding the character “C”. As shown in FIG. 5B, the pattern of the paste marker 3 is a square frame surrounding the character “P”. The image composition device 1 stores patterns printed by the copy marker 2 and the paste marker 3 in advance, analyzes a photographed image using these patterns, and recognizes a marker included in the photographed image.

  When the actual image is acquired, the image composition device 1 checks whether the initialization process has been performed, and branches the process depending on whether the initialization process has been performed (S3 in FIG. 2). First, a flow when the initialization process is performed will be described. When it is determined in S3 of FIG. 2 that the initialization process has not been performed, the process proceeds to S10 of FIG. 2 and the image composition device 1 starts performing the initialization process.

  In the present embodiment, as an initialization process, the image composition device 1 performs a process for calculating / storing the relative positional relationship between the copy marker 2 and the paste marker 3 and a process for generating / storing a texture. As will be described later, in order to virtually erase the image of the target object from the photographed image, two of the copy marker 2 and the pace marker are required, but the relative positional relationship between the copy marker 2 and the paste marker 3 is known. If there is, if either the copy marker 2 or the paste marker 3 is included in the live-action image, it is possible to virtually erase the image of the target object.

  When it is determined in S3 of FIG. 2 that the initialization process has not been performed, the image composition device 1 confirms whether both the copy marker 2 and the paste marker 3 are included in the live-action image, and performs processing according to the confirmed content. Branch (S10 in FIG. 2).

  The method in which the image composition device 1 confirms whether the copy marker 2 and the paste marker 3 exist at the same time is a general method used in the AR technology. ) And (b) are used to recognize the markers included in the live-action image, and from the characters (in this case, “P” or “C”) included in the live-action image, Recognize the type of marker included in the image.

  If both of the copy marker 2 and the paste marker 3 are present in the photographed image, the image composition device 1 determines that the initialization process can be performed, and proceeds to S11 in FIG. If neither the copy marker 2 nor the paste marker 3 exists, the image composition device 1 determines that the initialization process cannot be performed, and returns to S2 in FIG.

  As initialization processing, first, the image synthesizing apparatus 1 copies, as information indicating the relative positional relationship between the copy marker 2 and the paste marker 3, the conversion matrix Tcp from the copy marker coordinate system to the paste marker coordinate system and the paste marker coordinate system. A transformation matrix Tpc for the coordinate system is calculated, and the calculated transformation matrices Tcp and Tpc are stored (S11 in FIG. 2).

  FIG. 6 is a flowchart for explaining the processing of S11 of FIG. 2, and is a flowchart of processing for calculating the transformation matrix Tcp and the transformation matrix Tpc. The image synthesis apparatus 1 calculates a conversion matrix Tce from the copy marker coordinate system to the camera coordinate system and a conversion matrix Tpe from the copy marker coordinate system to the camera coordinate system in order to calculate the conversion matrix Tcp and the conversion matrix Tpc. (S40 in FIG. 6).

  The image synthesizing apparatus 1 uses a general AR technique to convert a copy matrix (Tce in this case) from a pattern printed on the copy marker 2 into a camera coordinate system Tce and a copy marker. A transformation matrix Tpe from the coordinate system to the camera coordinate system is calculated.

In the present embodiment, the image synthesizing apparatus 1 treats the dimension of the three-dimensional coordinate system value as a four-dimensional one by one, as in the case of handling general 3DCG, so that the transformation matrix Tce and the transformation matrix Tpe are 4 This is a matrix with 4 rows. For example, the conversion formula from the copy marker coordinate system to the camera coordinate system is expressed by Equation 1.

Next, the image synthesizing apparatus 1 calculates the transformation matrix Tec from the camera coordinate system to the copy marker coordinate system and the transformation matrix Tep from the camera coordinate system to the paste marker coordinate system using the inverse matrix computation according to Equation 2. (S41 in FIG. 6).

Then, the image composition device 1 uses the transformation matrix calculated so far, and according to Equation 3, the transformation matrix Tcp from the copy marker coordinate system to the paste marker coordinate system and the transformation matrix Tpc from the paste marker coordinate system to the copy marker coordinate system. And the conversion matrix Tcp and the conversion matrix Tpc are stored (S42 in FIG. 6).

  From here, it returns to description of FIG. The image synthesizer 1 uses, as information indicating the relative positional relationship between the copy marker 2 and the paste marker 3, a conversion matrix Tcp from the copy marker coordinate system to the paste marker coordinate system, and conversion from the paste marker coordinate system to the copy marker coordinate system. After calculating and storing the matrix Tpc, the image composition device 1 next generates a texture to be pasted on the virtual plane (S12 in FIG. 2).

  When the image synthesizing apparatus 1 generates a texture to be pasted on the virtual area, the parameters acquired in S1 of FIG. 2 are used. The image synthesizing apparatus 1 sets the area within the radius r1 as a sampling area in the copy marker coordinate system with the point shifted by the offset value included in the parameter from the origin of the copy marker coordinates as the center.

  Then, the image synthesizing apparatus 1 uses the transformation matrix Tce, and from the pixels of the real image extracted from the sampling area, uses the texture size and the overlap size of the parameter to make the texture seamless by a known technique widely known in the computer field. Is generated.

  When generating the texture, the image synthesizing apparatus 1 stores a flag indicating that the initialization process has been completed (S13 in FIG. 2), and ends the initialization process.

  When performing the initialization process, the image composition device 1 calculates a coordinate point that is the center point of the virtual plane in the copy marker coordinate system (S30 in FIG. 2). Although the center point of the virtual plane in the copy marker coordinate system may be an arbitrary coordinate point, here, the image composition device 1 is configured such that the line passing through the center of the paste marker 3 from the viewpoint of the camera is the XY plane in the copy marker coordinate system. Is obtained as a coordinate point that becomes the center point of the virtual plane. By forming the virtual plane on the XY plane in the copy marker coordinate system, the depth of the virtual plane with respect to the actual captured image can be matched when the virtual plane is combined with the actual captured image.

  For example, in FIG. 4, since the copy marker 2 is placed on the floor surface, the XY plane in the copy marker coordinate system is the floor surface. Therefore, if the extension line from the camera viewpoint to the center point of the paste marker 3 forms a virtual plane centering on the point intersecting the XY plane of the copy marker coordinate system, the position where the sense of depth is adjusted from the camera viewpoint Thus, a virtual plane is formed.

FIG. 7 is a flowchart for explaining the content of calculating the coordinate point of the copy marker coordinate system that is the center point of the virtual plane. The image synthesizing apparatus 1 uses the transformation matrix Tpc from the paste marker coordinate system to the copy marker coordinate system, using the theory well known in the technical field of AR, and the origin coordinate Vp [ px, py, pz] are extracted according to Equation 4 (S50 in FIG. 7).

Next, the image synthesizing apparatus 1 uses the transformation matrix Tec from the camera coordinate system to the copy marker coordinate system, using the theory well known in the technical field of AR, and the origin coordinate Ve of the camera coordinate system in the copy marker coordinate system. [Ex, ey, ez] is extracted according to Equation 5 (S51 in FIG. 7).

Then, from the origin coordinate Vp [px, py, pz] of the paste marker coordinate system in the copy marker coordinate system and the origin coordinate Ve [ex, ey, ez] of the camera coordinate system in the copy marker coordinate system, the image composition device 1 Coordinates Vf [fx, fy, fz] based on the copy marker coordinate system in which the virtual plane is arranged are calculated according to Equation 6 (S52 in FIG. 7).

  Returning to the description of FIG. When calculating the coordinate point in the copy marker coordinate system that is the center point of the virtual plane, the image composition device 1 generates a virtual plane on the XY plane of the copy marker coordinate system (S31 in FIG. 2). For example, the image synthesizing apparatus 1 performs the seamless processing and pastes the texture into the area of the radius r2 acquired at S1 in FIG. A virtual plane is generated on the XY plane.

  Then, when generating the virtual plane, the image composition device 1 uses the transformation matrix Tce from the copy marker coordinate system to the camera coordinate system to convert the coordinate points included in the virtual plane formed on the XY plane of the copy marker coordinate system, By converting the coordinate point of the camera coordinate system, the virtual plane is rendered on the real image, and the real image obtained by synthesizing the virtual plane is output to the display of the image composition device 1 (S32 in FIG. 2).

  After outputting the real image combined with the virtual plane to the display of the image synthesizing apparatus 1, it is confirmed whether or not the next process is to be performed. When the next process is performed, the process returns to S2 in FIG. 2 and the next process is not performed. If so, the flow illustrated in FIG. 2 is terminated (S33 in FIG. 2).

  From here, the process when the initialization process has been performed will be described. In step S <b> 3 of FIG. 2, when the image composition device 1 determines that the initialization process has been performed, that is, when the initialization process flag has been set, the image composition device 1 sets the copy marker 2. Using the printed pattern, it is determined whether or not the copy marker 2 is included in the photographed image (S20 in FIG. 2).

  When the copy marker 2 is included in the photographed image, the image composition device 1 calculates a conversion matrix Tce from the copy marker coordinate system to the camera coordinate system from the pattern printed on the copy marker 2 (S21 in FIG. 2). ).

  If it is determined that the copy marker 2 is not included in the photographed image, the image composition device 1 determines whether the paste marker 3 is included in the photographed image using the pattern printed on the paste marker 3. (S22 in FIG. 2) If the paste marker 3 is not included in the photographed image, the process returns to step S2 in FIG.

  When the photographed image includes the paste marker 3, the image composition device 1 uses the conversion matrix Tpc generated and stored in the initialization process to convert the copy marker coordinate system to the camera coordinate system Tce. Is calculated (S23 in FIG. 2).

  FIG. 8 is a flowchart for explaining the detailed contents of S22 of FIG. When calculating the transformation matrix Tce from the copy marker coordinate system to the camera coordinate system using the transformation matrix Tpc generated and stored in the initialization process, first, the image composition device 1 is printed on the paste marker 3. The transformation matrix Tpe from the paste marker coordinate system to the camera coordinate system is calculated from the pattern (S60 in FIG. 8).

Then, using the transformation matrix Tpc from the paste marker coordinate system to the copy marker coordinate system calculated and stored by the initialization process, the transformation matrix TTce from the copy marker coordinate system to the camera coordinate system is computed according to Equation 7 (FIG. 8). S61).

  After performing S21 in FIG. 2 or S23 in FIG. 2, the image composition apparatus 1 proceeds to S30 in FIG. 2, forms a virtual plane, and then synthesizes the virtual plane with the photographed image.

  Next, functions provided in the image composition apparatus 1 according to the present invention will be described. FIG. 9 is a hardware block diagram of the image composition device 1, and FIG. 10 is a functional block diagram of the image composition device 1. The image composition device 1 is realized by a computer device such as a personal computer or a server. As illustrated in FIG. 9, the image composition device 1 includes a CPU 10 that controls the entire image composition device 1, a RAM 11 that is a main memory, A hard disk 12 that is a large-capacity data storage device and an image processing board 13 for processing a real image acquired from the camera 1a are provided, and a camera 1a, a keyboard 1b, a display 1c, and the like are connected.

  As illustrated in FIG. 10, in order to virtually erase the image of the target object 4 that becomes an obstacle when combining 3DCG from the actual image using the AR technique, the image composition device 1 Parameter acquisition means 101 for acquiring parameters used when virtually erasing the four images from the live-action video, real-action image acquisition means 102 for acquiring the real-action image, and a marker pattern used for augmented reality is stored. Then, using the marker pattern, the marker recognizing unit 103 for recognizing the marker included in the photographed image taken by the camera 1a, and the marker recognizing unit 103 recognized from the marker pattern recognized by the marker recognizing unit 103. Calculation of marker coordinate system of marker, conversion matrix calculation means 104 for calculating conversion matrix from each marker coordinate system to camera coordinate system, and designation of copy source image A sampling area of a predetermined size is set on the XY plane of the marker coordinate system of the copy marker 2 by using the offset value determined as the position of the copy marker 2 to be copied, and the marker coordinate system of the copy marker 2 to the camera coordinate system is set. Using the transformation matrix, the texture generation means 105 that generates a texture of a predetermined size from the pixels of the actual image included in the sampling area and the coordinate origin of the paste marker 3 that specifies the paste destination, The coordinate generation unit 106 that calculates the coordinate point on the marker coordinate system of the copy marker that arranges the virtual plane to be combined with the image, and the coordinate point calculated by the arrangement coordinate calculation unit 106 are used as the center point, and the texture generation unit 105 generates the coordinate point. Virtual plane forming means 107 for forming a virtual plane of a predetermined size to which the texture is pasted, and the copy marker 2 Using the transformation matrix from the coordinate system to the camera coordinate system, the image synthesis means 108 for synthesizing the virtual plane formed by the virtual plane formation means 107 into a real image, and these means are controlled and shown in FIG. Control means 100 for executing the procedure is provided.

  The parameter acquisition means 101 of the image composition apparatus 1 is a means for executing S1 in FIG. 2, and the parameter acquisition means 101 of the image composition apparatus 1 can be realized in various forms. For example, a screen for inputting parameters is displayed on the display 1c and data input from the keyboard 1b is acquired, or a screen for inputting a file name is displayed on the display 1c and stored in the hard disk 12. This is realized by reading data from a file specified by a file name.

  The photographed image acquisition means 102 of the image composition apparatus 1 is a means for executing S2 of FIG. 2, in which a photographed image is read into the RAM 11 in real time from the camera 1a connected to the image composition apparatus 1, or stored in the hard disk 12. This is realized in the form of reading the image file being read into the RAM 11.

  The marker recognizing means 103 of the image synthesizing apparatus 1 is a means for executing S10, S20 and S22 of FIG. 2, and as shown in FIGS. 5 (a) and 5 (b) as in the general AR technique. By recognizing the square frame, the marker included in the live-action image is recognized, and further, the type of the marker included in the live-action image is determined from the characters (in this case, “P” or “C”) included in the square frame. It is a means to recognize.

  2 is a means for executing S11, S21 and S23 of FIG. 2, and when executing S11, S21 and S23 of FIG. 2, the transformation matrix calculation means 104 is similar to the general AR technique. A conversion matrix Tce and a conversion matrix Tpe from the marker coordinate system to the camera coordinate system are calculated from a pattern (here, a square frame) printed on the marker recognized by the marker recognition means.

  Furthermore, the transformation matrix calculation means 104 uses the transformation matrix Tce from the copy marker coordinate system to the camera coordinate system and the transformation matrix Tpe from the paste marker coordinate system to the camera coordinate system when executing S11 of FIG. Matrix calculation is performed to calculate a conversion matrix Tec from the camera coordinate system to the copy marker coordinate system and a conversion matrix Tep from the camera coordinate system to the paste marker coordinate system.

  The texture generation unit 105 of the image composition apparatus 1 is a unit that executes S12 of FIG. 2, and is centered on a point that is shifted from the origin of the copy marker coordinates by the offset value included in the parameter acquired by the parameter acquisition unit 101. Is set in the copy marker coordinate system as a sampling area, and the texture size and overlap size of the parameter are determined from pixels of the captured image extracted from the sampling area using the transformation matrix Tce. Used to create a texture that is seamless by known techniques widely known in the computer field.

  The arrangement coordinate calculation means 106 of the image synthesizing apparatus 1 is a means for executing S30 of FIG. 2, and calculates a coordinate point that becomes the center point of the virtual plane in the XY plane of the copy marker coordinate system according to the procedure shown in FIG. It is means to do.

  The virtual plane forming unit 107 of the image synthesizing apparatus 1 is a unit that executes S31 of FIG. 2 and a copy calculated by the arrangement coordinate calculating unit 106 by pasting the texture generated by the texture generating unit 107 by performing a seamless process. This is a means for generating a virtual plane of a predetermined size around the coordinate point of the XY plane of the marker coordinate system.

  The image synthesizing unit 108 of the image synthesizing apparatus 1 is a unit that executes S32 of FIG. 2, and uses the transformation matrix Tce from the copy marker coordinate system to the camera coordinate system to create the virtual plane formed by the virtual plane forming unit 107. This is a means for rendering a virtual plane on a live-action image by converting it into a coordinate point in the camera coordinate system, and outputting the real-image obtained by synthesizing the virtual plane to the display of the image composition device 1.

  The control unit 100 of the image composition apparatus 1 is a unit for controlling the above-described units and executing the procedure illustrated in FIG.

  Each means provided in the image composition apparatus 1 is realized by a computer program for causing the image composition apparatus 1 to function as each means. The hard disk 12 and the image processing board 13 of the image composition apparatus 1 have the program of this computer program. The code is implemented. The control means 100 of the image composition apparatus 1 corresponds to a main routine of a computer program, and means other than the control means 100 correspond to a subroutine called from the main routine.

  As described above, if the image composition device 1 virtually erases the image of the target object 4 that becomes an obstacle when composing 3DCG from the actual image, and then composes 3DCG using the AR technology, It is possible to synthesize 3DCG into a real image without losing naturalness.

The figure explaining the utilization form of the image composition apparatus concerning this invention. The flowchart when demonstrating the process when erase | eliminating a target object from an image virtually. The figure explaining an example of the parameter which an image composition apparatus acquires. The figure explaining an example of a real image. The figure explaining a copy marker and a paste marker. The flowchart for demonstrating the process of S11 of FIG. The flowchart which calculates the coordinate point of the copy marker coordinate system used as the center point of a virtual plane. The flowchart explaining the detailed content of S22 of FIG. The hardware block diagram of an image composition apparatus. The functional block diagram of an image synthesizing | combining apparatus.

Explanation of symbols

1 Image composition device 2 Copy marker 3 Paste marker 4 Target object

Claims (6)

  Marker recognition means for storing a marker pattern used in Augmented Reality (AR), recognizing the marker included in a photographed image taken by a camera using the marker pattern, and the marker A transformation matrix computing means for computing a marker coordinate system of the marker recognized by the marker recognition means from a marker pattern recognized by the recognition means, and calculating a transformation matrix from each marker coordinate system to the camera coordinate system; and a copy source A sampling area of a predetermined size is set on the XY plane of the marker coordinate system of the copy marker using the position of the marker (copy marker) that designates the image and the offset value determined, and the marker coordinates of the copy marker Using a transformation matrix from the system to the camera coordinate system, a texture of a predetermined size is obtained from pixels of the photographed image included in the sampling area. Using the coordinate origin of the marker (paste marker) for designating the paste destination, and the coordinate point on the marker coordinate system of the copy marker for arranging the virtual plane to be combined with the live-action image Arrangement coordinate calculation means for calculating; a virtual plane forming means for forming a virtual plane of a predetermined size with the texture generated by the texture generation means as a central point, the coordinate point calculated by the arrangement coordinate calculation means; Image synthesizing means for synthesizing the virtual plane formed by the virtual plane forming means with the photographed image using a transformation matrix from the marker coordinate system to the camera coordinate system of the copy marker; An image composition device.   2. The image composition apparatus according to claim 1, wherein the arrangement coordinate calculation unit is configured to use a coordinate origin of the paste marker and a coordinate of the camera as coordinate points on a marker coordinate system of the copy marker on which the virtual plane is arranged. An image synthesizing apparatus that calculates a point where a straight line passing through an origin intersects the same plane as an XY coordinate plane of the copy marker.   3. The image composition apparatus according to claim 1, wherein the conversion matrix calculation unit adds the copy matrix in addition to a conversion matrix from the marker coordinate system to the camera coordinate system of the marker recognized by the marker recognition unit. After calculating and storing the conversion matrix between the marker coordinate system of the marker and the paste marker, and calculating and storing the conversion matrix between the marker coordinate system of the copy marker and the paste marker, the marker recognition means When only one of the marker and the paste marker can be detected from the photographed image, the transformation matrix calculation means detects from the photographed image using a transformation matrix between the marker coordinate systems of the copy marker and the paste marker. An image synthesizing apparatus that calculates a transformation matrix related to the marker that could not be obtained.   An image composition method that uses augmented reality (AR), which is included in live-action images taken with a camera using a computer device image and a pattern printed on a marker used in augmented reality. The step of recognizing the marker, and the computer device calculates the marker coordinate system of the marker recognized by the marker recognition means from the recognized marker pattern, and converts each marker coordinate system to the camera coordinate system A step b for calculating a matrix, and the computer device uses the offset value determined as the position of the marker (copy marker) for designating the copy source image on the XY plane of the marker coordinate system of the copy marker. A sampling area of a predetermined size is set, and the sample of the copy marker is converted using the conversion matrix from the marker coordinate system to the camera coordinate system. A step c for generating a texture of a predetermined size from the pixels of the photographed image included in the image area, and the computer device image, using the coordinate origin of the marker (paste marker) for specifying the paste destination, A step d for calculating a coordinate point on the marker coordinate system of the copy marker for arranging a virtual plane to be combined with a real image, and the computer generating unit having the coordinate point calculated in the step d as a central point; Forming a virtual plane of a predetermined size pasted with the texture generated by the computer, and the computer apparatus forms the virtual plane using a conversion matrix from the marker coordinate system of the copy marker to the camera coordinate system And f for synthesizing the virtual plane formed by the means with the photographed image. Method.   5. The image composition method according to claim 4, wherein in the step d, the computer device sets the coordinate origin of the paste marker as the coordinate point on the marker coordinate system of the copy marker on which the virtual plane is arranged, and the An image composition method, comprising: calculating a point where a straight line passing through a coordinate origin of a camera intersects the same plane as an XY coordinate plane of the copy marker. 6. The image synthesizing method according to claim 4, wherein, in the step b, the computer device adds the copy marker and the paste in addition to a conversion matrix from the marker coordinate system to the camera coordinate system of the marker. After calculating / storing the conversion matrix between the marker coordinate systems of the markers and calculating / storing the conversion matrix between the marker coordinate systems of the copy marker and the paste marker, in step a, the copy marker or the paste marker The computer device cannot detect from the photographed image using the transformation matrix between the marker coordinate systems of the copy marker and the paste marker in step b. An image synthesizing method comprising calculating a transformation matrix related to the marker.