KR20080018397A - Computer-readable recording medium for recorded program for providing 3d modeling based real-time images - Google Patents

Abstract

A computer-readable recording medium having a program supporting real time image-based three dimensional modeling is provided to shorten a working time of contents manufacturers and perform precise modeling on an actual object. A three-dimensional matching module(110) recognizes a square candidate region which is to be a region of a marker of an image transferred from a camera, detects an actual region of the marker from the square candidate region, calculates a three-dimensional position of the marker, and matches a modeled object to an actual object. A laser point recognizing module(120) recognizes points illuminated by shooting a laser point to the actual object from two camera images, calculates three-dimensional coordinates at the points by using a marker-based three-dimensional positioning technique, and provides the same. A plug-in fabricating module(130) reads a value from a data structure of a 3D(three dimensional) max model, inputs the coordinates recognized by the laser point to the data structure, embeds the camera images to a 3D max program interface, and matches the 3D max model to the camera images.

Description

Computer-Readable Recording Medium for Recorded Program for Providing 3D Modeling based Real-time Images}

1 is a conceptual structural diagram illustrating a program supporting real-time image-based three-dimensional modeling according to a preferred embodiment of the present invention;

2 is an exemplary view for explaining marker recognition and 3D model registration according to a preferred embodiment of the present invention;

3 is an exemplary view for explaining a method for automatically calculating a threshold value according to a preferred embodiment of the present invention;

4 is an exemplary view for explaining a blind spot detection process according to a preferred embodiment of the present invention;

5 is an exemplary view for explaining a screen for performing 3D matching according to a preferred embodiment of the present invention;

6 is an exemplary view for explaining a laser point recognition process according to a preferred embodiment of the present invention;

7 is an exemplary view showing a user interface screen of the plug-in production module according to an embodiment of the present invention.

<Description of Symbols for Main Parts of Drawings>

110: 3D modeling program 110: 3D matching module

112: image acquisition module 114: marker recognition module

116: marker position tracking module 120: laser point recognition module

122: point recognition module 124: point position calculation module

130: plug-in creation module 132: Max model data structure access module

134: Max model image registration module

The present invention relates to a computer-readable recording medium that records a program supporting real-time image-based three-dimensional modeling. More specifically, a computer that records a program that supports real-time image-based three-dimensional modeling that provides intuitive three-dimensional modeling using techniques such as camera matching and laser pointer tracking based on three-dimensional objects reflected on the screen of 3D Max. It relates to a recording medium that can be read from.

At present, domestic 3D production is mainly done in a labor-intensive manner. However, in the case of overseas, it has been a long time since the method of producing 3D using a 3D input device has been established, which is contrasted with the poor production conditions in Korea.

However, these 3D input devices have a very high introduction price per unit, which makes it difficult to make purchase decisions easily in the domestic poor 3D production conditions. Due to these conditions, domestic 3D-related industries rely on simple labor for a long time, and the industrial development is delayed due to low productivity compared to the administration personnel. In addition, 3D producers have been unable to secure enough capacity to be invested in low-cost 3D simple production manpower management and be able to inject into the creative development and planning areas of highly profitable cultural contents. There is a situation.

Therefore, in order to develop the 3D-related industry, one of the important areas of the cultural contents market, and to create a synergy effect of the development of the domestic 3D contents market, it is required to secure production technology related to 3D modeling. It can be said to develop a 3D modeling solution suitable for the situation. Acquisition of life-based 3D modeling solutions can lead to improved production productivity of 3D-related industries, which can lead to investments in qualitative development and creative development capabilities of those industries, which have been degraded as low-cost labor-intensive industries.

In general, when generating 2D or 3D content, object-based modeling is performed. A mouse is used to manually manipulate an object on a two-dimensional plane based on the captured image. However, because the mouse on the two-dimensional plane provides two degrees of freedom of movement, it is not a suitable method for use in the object-based modeling that requires manipulating objects in three-dimensional space requiring six degrees of freedom of movement.

In order to allow spatial rotation of the object to examine all aspects of a three-dimensional space object, six degrees of freedom must be provided. However, most computers have two-dimensional input tools such as mice, trackballs and joysticks.

In addition, common commercial 3D modeling software, such as Maya or 3D Max, utilize existing mouse / monitor interface systems to produce three-dimensional images.

However, content creators who pursue high-quality 3D production are inconveniently feeling the indirect space generation method by 2D mouse based on this mathematical method. Workers try to produce real objects with expensive 3D scanner devices, but these methods are expensive in reality. In addition, producers can learn and produce 3D production tools by themselves, which is difficult for most artists to access due to the training time and the engineering approach in the production process. This is because the existing production method using 2D mouse has various methods for surface creation in space, and it is necessary to learn this.

Based on the real model, the actual content producers take a picture of the front and the side, and put it in the background on the 3D Max and then overlap the wire frame. However, this method requires a lot of time for the manufacturer and requires very long modeling time because the model must be matched on three planes rather than working directly in the 3D camera window.

In order to solve the above problems, the present invention, based on the three-dimensional real image reflected on the screen of 3D Max, real-time image-based three-dimensional to provide intuitive three-dimensional modeling using techniques such as camera matching and laser pointer tracking An object of the present invention is to provide a computer-readable recording medium recording a program supporting modeling.

In order to achieve the above object, the present invention is a program that supports real-time image-based three-dimensional modeling, (a) recognizes the blind candidate area to be the marker area in the image from the camera, and the actual in the blind candidate area A three-dimensional matching module for detecting a region of the marker and calculating a three-dimensional position of the marker to match an object modeled with a real object; (b) Recognizing the point of light emitted by shooting a laser point on a real object in each of the two camera images, and then calculates and provides three-dimensional coordinates using a marker-based three-dimensional position tracking technology at the point Laser point recognition module; And (c) reading a value on a data structure of a 3D max model, inputting coordinates recognized as the laser point into the data structure of the 3D max model, embedding a camera image in a 3D max program interface, and It provides a computer-readable recording medium that records a program that supports a three-dimensional modeling based on the real-time image, characterized in that it comprises a plug-in production module for performing a function of matching the model of the 3D Max to the image.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are designated as much as possible even if displayed on different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

1 is a conceptual structural diagram for explaining a program supporting real-time image-based three-dimensional modeling according to a preferred embodiment of the present invention.

The 3D modeling program 110 according to the preferred embodiment of the present invention receives the image of the camera, recognizes the marker, calculates the position of the marker, and calculates the calculated position value to the position value of the result of the 3D max. It performs the function of providing the matched image by applying. Also, it recognizes a laser point in the camera image and calculates three-dimensional coordinates to provide coordinate data of the three-dimensional result.

The 3D modeling program 110 includes a 3D matching module 110, a laser point recognition module 120, and a plug-in manufacturing module 130.

The 3D matching module 110 according to an embodiment of the present invention is an image acquisition module 112 that performs two control functions to acquire an image from the camera, or does not acquire the image, the area of the marker in the image from the camera A marker recognition module 114 for finding a rectangular candidate area to be obtained and obtaining a real marker area in the rectangular candidate area, and a marker position tracking module 116 for performing a function of calculating a three-dimensional coordinate relative to the marker and the camera do.

The 3D matching module 110 recognizes a marker (barcode, space tag) of the image coming from the camera by using the 3D matching technology, tracks the 3D position, and models the actual object shown in the image. Perform corrections by matching one object.

In addition, the 3D matching module 110 uses augmented reality technology, wherein the augmented reality technology is a technology that provides a user with improved realism by synthesizing additional information such as text and graphics in a real world environment in real time. .

2 is an exemplary diagram for explaining marker recognition and 3D model registration according to a preferred embodiment of the present invention.

The 3D matching module 110 performs marker recognition and 3D position tracking through the marker recognition module 114 and the marker position tracking module 116. The camera image obtained by the image acquisition is transferred to the marker recognition module 114, and the marker recognition module 114 finds a rectangular area matching the marker information DB. Once the marker area is confirmed, the 3D coordinates are tracked and 3D output is output based on the tracked coordinates. That is, the 3D matching module 110 binarizes the input image of the camera, recognizes the marker region, and tracks the 3D position of the marker to render the 3D model.

In more detail, after obtaining an image from a camera, calculating a threshold for finding a rectangular candidate area, the rectangular candidate area is labeled with the calculated threshold. Detects the outlines of the labeled areas, obtains vertices from the detected points, determines the rectangular area, analyzes the information in the rectangle, detects the marker area, and finds 3D location information from the distorted shape of the rectangle. Do the justice.

To acquire video from the camera, use the VFW (Video For Windows) library. The VFW library is a video library provided by Microsoft, which provides functions in vfw32.dll. The two most important functions provided by default are capCreateCaptureWindow (), which creates a capture window, and capGetDriveDescription (), which retrieves the version information of the capture driver.

After specifying the window area to show the captured screen and creating a video capture window in the designated area, the capture window and the driver are connected and displayed in the preview mode in the capture window.

Through this process, the captured image from the host camera is displayed on the screen, the video format is set for compatibility of the callback function, the information of the captured image is obtained, the callback function is specified, and the information required for capture is set. Save and capture.

In order to find a specific marker in an image obtained from a camera, it is necessary to first select pixels having a higher value based on a specific threshold. This operation is called labeling. In the present invention, the candidate region is found by using a specific threshold to find the candidate region of the marker. However, since the threshold set to fixed is easily affected by the ambient brightness (lighting, sunlight), a method for extracting an appropriate threshold value according to the environment when the program is running is needed. A method as shown in FIG. 4 is used to automatically calculate the threshold value according to such an environment.

As shown in FIG. 3, a method using an average value is used to find a threshold. First, the average value of all the acquired pixels is found, the average value of the pixels that does not become the average value of all the pixels thus found is again found, and the average value of the two is applied as a threshold.

4 is an exemplary view for explaining a blind spot detection process according to an exemplary embodiment of the present invention.

In order to perform location tracking based on image recognition, a process of detecting a marker in an acquired camera image should be performed first. The present invention obtains an image from the camera, calculates a threshold value according to the environment, binarizes the image to 0 and 1, and then displays a candidate square region recognized as a marker to perform labeling to select a portion to be processed. The edges are detected by finding consecutive points in the area, and four vertices are found from the detected edges.

In order to implement 3D matching technology using marker-based augmented reality technology, after marker recognition is completed, the 3D position of the marker is tracked, and 3D max model data is synthesized at the calculated 3D position. Therefore, the shape or characteristics of the marker are an important part of the present invention. In the present invention, an extended marker technology that can solve the disadvantage of the ID recognition portion of the ARToolkit marker was used.

First, a threshold value is obtained from an RGB image, converted into a binary image, and recognized as a marker region. Find the vertex on the marker and flatten the oblique trapezoid. Find the inner vertex of the marker, extract only the ID region, divide the inner region into quarters, and compare it with the mask. After determining whether the marker is a marker and checking the direction of the marker, if it is determined as a marker, each part is divided into 4 equal parts and divided into 16 equal parts. Recognize the rest of the code except the marker check area as a binary number and calculate the ID value.

The marker in the image obtained from the camera has a shape in which the rectangular shape is transformed into a trapezoidal shape according to the direction in which the camera shines. To make this trapezoid from a square, you can determine the coordinates when you turn the initial square little by little in three-dimensional space to form the nearest shape.

After the marker region is determined, three-dimensional position information of the marker is calculated. At this time, when the 3D object is synthesized to the image using only the 3D value calculated by the ARToolkit Core function, a slight error occurs when matching the real object to the real image. In order to calculate the position more accurately, the improvement method is used to automatically correct the error according to the measured distance. Position correction is calculated by applying the camera's vertical shift correction and vertical shift correction.

As described above, when the marker recognition module 114 recognizes the marker in the image obtained from the camera and tracks the three-dimensional position of the marker, a process of matching the camera image with the three-dimensional object and displaying it on the screen is performed.

When the image is acquired from the camera and the marker recognition and 3D position tracking are performed, the active view window to be applied in the 3D max is acquired to update the image of the camera in the view window, and the image of the camera to enter the activated view window is acquired. Update Get the matrix of the active view window, update it with the position value of the marker, and update the screen to show the applied position value.

By performing such a process, the 3D matching module 110 receives the image of the camera, recognizes the marker, calculates the position of the marker, and applies the calculated position value to the position value of the target object of the 3D max. As shown, a matched image may be provided.

The laser point recognition module 120 according to a preferred embodiment of the present invention performs a function of manually setting a threshold to find a laser point, and the point recognition module 112 recognizes a point where a laser point shines in a camera image. And a point position calculation module 124 that calculates three-dimensional coordinates of the laser point shown in the camera image.

The laser point recognition module 120 recognizes points emitted by shooting a laser point on a real object in two camera images, and then calculates three-dimensional coordinates based on the marker-based three-dimensional position tracking technology. . The calculated three-dimensional coordinates can be output to a text file after the calibration process and used in 3D Max programs.

That is, the laser point recognition module 120 acquires two-dimensional coordinates in the image through image processing by recognizing the laser point in the image acquired by the camera, and three-dimensional coordinates and three-dimensional coordinates of the laser pointers obtained by the two cameras. Calculate the three-dimensional coordinates of the laser point by acquiring the intersection of the two segments created by creating the segment connecting the camera coordinates of the image.

6 is an exemplary view for explaining a laser point recognition process according to a preferred embodiment of the present invention.

Two cameras are used to recognize four markers on the axis, and two and three coordinate axes are generated from the four markers. The vertices of the hexahedron are input using a correction hexahedron, and the correction values of three-dimensional coordinates are calculated using the input vertices. Let two cameras illuminate the object to be modeled, shoot the laser point onto the surface of the object to be modeled, and then locate the laser point in the two camera images. Obtain three-dimensional coordinates of the laser pointer and correct the obtained coordinates.

That is, in order to recognize the laser point and calculate the 3D position, the marker search is performed, the three-dimensional axis is formed, the vertex coordinates of the cube are input, and the correction coordinates inside the cube must be calculated. After the laser point is recognized, the coordinates of the laser point are calculated based on the internal calibration coordinates.

Two camera images are needed to find the laser point. You can get the image using the VFW library already described, but you can expect faster performance by using DirectShow technology in DirectX. Therefore, DirectShow technology is used for laser point recognition. However, there is a disadvantage that it is difficult to apply it to each project because the usage method for acquiring camera image in DirectShow is complicated. In addition, since the simultaneous image acquisition function for a plurality of cameras is not implemented, it is not suitable for various applications, and thus the source of the camera acquisition portion of AMCap, which is a basically provided sample source, is modified and used. AMCap takes one image from among several cameras and outputs it, and provides options for adjusting and capturing the camera. In addition, since the camera is accessed using a library called BaseClasses, some setting values of this library are changed and adjusted so that it can be used when writing general applications. Using this modified library, the CSampleGrabberCBXX class is added to match the number of cameras to be used to implement simultaneous image acquisition.

Four markers are required to form a two-dimensional axis (hereinafter referred to as a two-dimensional axis) formed by the marker on the image or a three-dimensional axis (hereinafter referred to as a three-dimensional axis) in three-dimensional space. If no one of these four markers is found, the position of the axis cannot be calculated accurately. Therefore, in the present invention, four markers must be used.

 If all four markers that form the two-dimensional axis are found, then two axes must be created. The laser point to be found later is calculated as the relative position value of these axes.

 First, the 2D axis is determined on the image and appears in various shapes depending on the angle and position of the camera. Using two pairs of markers facing each other out of four markers recognized in the image, two line segments are formed. The intersection of these two segments becomes the center of the two-dimensional axis, and each segment is divided into X and Y axes according to the ID value of the marker, so the image can be divided into quadrants.

You can use ARToolkit to get the 2D and 3D coordinates of a marker. The three-dimensional coordinates of the marker are independent spaces according to the position of the camera, and change these spaces to one common camera position. Through this process, each position with respect to the camera appears in one three-dimensional space.

The relative coordinates of the three-dimensional coordinate axis can be obtained using the relative coordinates of the two-dimensional axis obtained above. Although two-dimensional coordinates cannot be directly applied to three dimensions, the (X, Z) plane without the y-axis was created through the transformation process of the plane formed by the marker. Conversion to dimensional coordinates is possible.

In summary, the laser point recognition module 120 finds four markers that serve as axes to find the three-dimensional position of the laser point and generates two-dimensional and three-dimensional coordinate axes. Two axes are created because the axes must be created separately for each image coming from two cameras.

Here, the 2D coordinate axis obtains the midpoint of the 2D position values of the marker and the extension lines of the facing markers become the coordinate axis, and the 3D coordinate axis can be obtained by rotating the 3D position values of the marker on the X axis and the Z axis. have. When rotating the marker position values, apply the markers of all markers and the camera together.

In addition, the laser point recognition module 120 finds the position of the laser point shot on the object in the camera image to recognize the laser point. To find the location of the laser point, you need to adjust the camera's options. The camera's options depend on the camera's type and environment. It is generally a good idea to adjust the color value on the screen to be dark. In darker images, the laser point appears brighter and finds the location of the benefit.

In order to find the position value of the laser point, the option is changed to obtain an image of a dark color value from the camera, and to find a threshold for labeling. When the position value of the brightest part estimated by the laser pointer is found, the threshold value is applied to binarize the image. The binarized image is labeled and divided by regions, and a labeling region including the brightest portion is obtained. The center of the acquired labeling area is found and set as the position value of the laser pointer.

In addition, in order to recognize the three-dimensional position of the laser point, the position value in the two-dimensional illuminated by the laser pointer is converted to fit the two-dimensional axis and then applied to the three-dimensional coordinates. Since the plane formed by the marker in 3D is a (x, z) plane without a y-axis, a 2D point can be applied to the 3D.

A line segment is created by connecting the camera position and the object position in three dimensions obtained from the two images. The intersection of the two straight lines generated is the three-dimensional coordinates of the laser pointer and is calculated by the following process.

After converting 2D laser coordinates to coordinates based on 2D axis, applying 2D coordinates to 3D coordinates, create two line segments connecting the 3D camera position and the object's position. Obtain

By performing such an operation, the laser point recognition module 120 can easily generate a three-dimensional result by identifying a three-dimensional position of the laser point emitted.

Plug-in manufacturing module 130 according to a preferred embodiment of the present invention by using the results generated from the three-dimensional matching module 110 and laser point recognition module 120, by utilizing the functions provided by the SDK of the 3D Max It is provided as a plug-in module that is embedded in 3D Max program.

The plug-in production module 130 performs a function of reading a value from the data structure of the 3D max model, and inputs a coordinate recognized as a laser point on the data structure of the 3D max model. 132) and a Max model image registration module 134 for embedding a camera image in the 3D max program interface and matching a model of the 3D max to the camera image.

The plug-in manufacturing module 130 according to the preferred embodiment of the present invention includes a 3D matching function and a laser point recognition function to support intuitive modeling based on a real-time image. Using the 3D matching function, the modeling user can make the camera image appear in the 3D Max program, and by illuminating the marker on the camera image, the augmented reality image in which the 3D model information is matched in the real world can be seen. Therefore, modeling work through this can increase work efficiency. The laser point recognition function finds laser points in two image images of different angles and calculates three-dimensional coordinates of the laser points. The calculated coordinates are passed to the 3D Max program for use in 3D modeling work.

7 is an exemplary view showing a user interface screen of the plug-in production module according to an embodiment of the present invention.

The interface screen of the plug-in production module 130 includes a camera adjustment button, a camera binary image, an active view width, a threshold adjustment button, and a 3D matching dialog execution button, so that the camera image is embedded and options for the camera are included. You can also set.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

As described above, according to the present invention, by supporting real-time image-based three-dimensional modeling, it is possible to reduce the working time of content creators and to enable accurate modeling in real life.

In addition, the improvement in production productivity of 3D-related industries has the effect of investing in the qualitative development and creative development capability of the industry, which has been degraded as a low-cost labor-intensive industry.

Claims (5)

This program supports real-time image based 3D modeling. (a) Recognizing the rectangular candidate area to be the marker area in the image from the camera, detecting the actual marker area in the rectangular candidate area, calculating the three-dimensional position of the marker and modeling the object on the real object. A three-dimensional matching module performing a matching function; (b) Recognizing the point of light emitted by shooting a laser point on a real object in each of the two camera images, and then calculates and provides three-dimensional coordinates using a marker-based three-dimensional position tracking technology at the point Laser point recognition module; And (c) read values on the data structure of the 3D Max model, input the coordinates recognized as the laser points into the data structure of the 3D Max model, embed the camera image in the 3D Max program interface, and Plug-in production module that performs the function of matching the model of the 3D Max to A computer-readable recording medium recording a program for supporting real-time image-based three-dimensional modeling, comprising a. The method of claim 1, wherein the three-dimensional matching module, An image acquisition module configured to acquire an image from the camera or perform a control function not acquiring an image; A marker recognition module for finding the rectangular candidate area to be the area of the marker in the image coming from the camera, and obtaining a real marker area from the rectangular candidate area; And Marker position tracking module for calculating a three-dimensional coordinates of the marker and the camera A computer-readable recording medium recording a program for supporting real-time image-based three-dimensional modeling, comprising a. The method of claim 1, wherein the laser point recognition module, A point recognition module for setting a threshold to find the laser point, and recognizing a point where the laser point shines in the image of the camera; And Point position calculation module for calculating the three-dimensional coordinates of the laser point seen in the image of the camera A computer-readable recording medium recording a program for supporting real-time image-based three-dimensional modeling, comprising a. According to claim 1, wherein the program, After acquiring an image from the camera, calculating a threshold value according to a surrounding environment, binarizing the image to 0 and 1 using the threshold value, and labeling the blind spot region to recognize the marker as the marker. The real-time image-based three-dimensional modeling is performed by selecting the area to be processed by displaying the rectangular candidate area, and detecting the four candidate points by detecting an outline in the labeled area. Computer-readable recording media that records supported programs. According to claim 1, wherein the program, Convert two-dimensional laser coordinates to coordinates based on two-dimensional axes, apply two-dimensional coordinates to three-dimensional coordinates, and then create two line segments connecting the three-dimensional camera position and the object's position to the intersection of three-dimensional straight lines. And calculating the three-dimensional coordinates of the laser point by performing a process of obtaining the data.

Images