KR20220014513A - 3D Model Control System and Method Based on multi-screen Projection - Google Patents

Abstract

The present invention relates to a portable 3D model control system based on multi-screen projection. According to the present invention, a normalized 3D model is projected using a projection device so that a user can watch on a multi-screen projection basis, the user can check, control, and correct the projected 3D model with an intuitive user interface, and thus even an ordinary person without professional knowledge as to professional design programs can easily check and correct with the intuitive user interface. In addition, projection and watching on any surface are possible based on work site portability and multiple work-related persons can perform simultaneous and real-time design check, correction and collaboration from various viewpoints and perspectives.

Description

3D Model Control System and Method Based on multi-screen Projection

The present invention relates to a three-dimensional model control system using augmented reality.

Augmented reality is a technology that enables a user to acquire desired information by displaying information generated through signal processing in reality and allowing the user to control it. These augmented reality technologies are being developed and utilized in various forms in the fields of broadcasting, manufacturing process processing, architectural design, games, and mobile solutions.

For example, augmented reality technology is being commercialized by displaying current traffic information or living information around a place where a user is located on an actual 2D or 3D map.

3D model technology, which is widely used in construction and product manufacturing processes, uses a computer to design a structure to be built or a product to be manufactured in 3D in reality, and the structure or product according to the design can be transformed into various 3D models. It is a technology that allows you to check from a hierarchical and perspective, and to revise a design. Through such 3D model technology, users can design work in a more efficient and convenient way compared to design work using only two-dimensional design drawings.

Existing 3D model technology is mainly a method in which the user designs a 3D model using a keyboard, mouse, or pen through a professional design program running on a computer, and checks the 3D model, the design result, through the monitor screen. was done As a result, in the existing 3D model technology, it is difficult for the general public who does not have the knowledge of the professional design program to check and correct the design, and also check the design in real time at the job site and work, There is a problem in that it is difficult for a large number of work related parties to proceed with the work while simultaneously checking the design.

An object of the present invention is to project a 3D model that has undergone normalization so that the user can project it onto an arbitrary multi-sided screen and see it using a projection device, and a 3D model projected by the user through an intuitive user interface By making it possible to check, control, and modify, ordinary people who do not have professional knowledge about professional design programs can easily check and modify through an intuitive user interface, and can project and view it on any surface while carrying it at the job site, In addition, it is to provide a multi-faceted screen projection-based 3D model control system and method that allows multiple work related parties to simultaneously check and revise designs in real time from various viewpoints and viewpoints, and to collaborate.

In order to achieve the above object, a portable projection-based model control system according to an embodiment of the present invention is a marker that is an identification tag having a specific pattern; a camera unit for photographing the marker located on the image projection plane on which the image is to be projected; a control command processing unit that receives a user's control command for a specific model to be processed by the user and generates a model control command; and recognizing the marker photographed by the camera unit, controlling the model according to a model control command generated by the control command processing unit, processing information about the model, and using the recognized marker information to obtain the image It may include an image processing unit that generates an image of the model to be projected on the projection surface.

Here, the portable projection-based model control system may further include an image projection unit that projects the image generated by the image processing unit on the image projection surface.

In one embodiment, the camera unit may be characterized in that the user touches the image projection surface or the motion of the hand or finger to photograph.

In one embodiment, the model control command is a command to rotate the model in two or three dimensions, a command to scale the size of the model, a command to select some of the components of the model, or a command to modify the model It may be characterized in that it includes a command.

In one embodiment, the portable projection-based model control system further comprises a sensor unit for recognizing a motion of a user's hand or finger, wherein the control command processing unit receives a user's control command from the sensor unit to control the model It may be characterized in that the command is generated.

In one embodiment, the portable projection-based model control system further comprises an input terminal unit for a user to input a command for controlling the model, wherein the control command processing unit receives the user's control command from the input terminal unit It may be characterized in that the model control command is generated.

In one embodiment, the control command processing unit may be characterized in that the recognition of the user's motion photographed by the camera unit to generate the model control command.

In an embodiment, the image processing unit comprises: a marker recognition unit for detecting and recognizing the marker located on the image projection surface; a model processing unit for controlling the model according to the model control command generated by the control command processing unit and processing information about the model; and generating model image content including information on the model controlled by the model processing unit and the processed model, and using the model image content and information on the marker recognized by the marker recognition unit, the image projection surface It may include a projection image generator for generating a projection image of the model to be projected on the.

In an embodiment, the model processing unit rotates the model in two or three dimensions, scales the size of the model, selects some of the model components, or selects the model according to the model control command. It may be characterized in that it performs a control operation to correct the.

In one embodiment, the model processing unit may be characterized in that according to the model control command, the structure information and the constituent entity information of the model are processed.

In one embodiment, the projection image generating unit calculates the coordinates of the image projection surface on which the marker is located, using the shape and size information of the marker recognized by the marker recognition unit, and calculates the coordinates of the image projection unit and generating a spatial transformation matrix between the coordinates of the image projection surface and the coordinates of the image projection unit, transforming the model image content using the generated spatial transformation matrix, and projecting the image projection image of the model to the image projection surface It may be characterized by generating

In an embodiment, the image projection unit projects a calibration image of a certain pattern on the image projection surface, the camera unit captures the calibration image projected on the image projection surface, and the projection image generation unit is the captured calibration image It may be characterized in that a pre-coordinate correction operation between the image projection unit and the camera unit is performed using the information of .

In an embodiment, the projection image generating unit may be characterized in that the coordinates of the image projection unit are calculated by using the information of the pre-coordinate correction operation.

In an embodiment, the marker may include: a first marker positioned and used on the first image projection surface to specify the model and to generate a first image to be displayed on a first image projection surface on which an image is to be projected; and a second marker positioned and used on the second image projection surface to generate a second image to be displayed on the second image projection surface on which the image is to be projected.

In an embodiment, the marker recognition unit recognizes the first marker repeatedly at a certain period, and recognizes the second marker at least once, and the projection image generator includes: A first spatial transformation matrix is newly generated by repeating at a predetermined period to generate a first projection image of the model to be projected on the first image projection surface, and a second spatial transformation matrix is generated at least once for the second image projection surface Thus, it may be characterized in that the second projection image of the model to be projected on the second image projection surface is generated by using the generated second spatial transformation matrix.

In an embodiment, the control command processing unit generates the model control command for rotating the model according to the rotation of the first marker when the first marker photographed by the camera unit rotates on the first image projection surface. It can be characterized as

In order to achieve the above object, a portable projection-based model control method according to another embodiment of the present invention is to photograph a marker, which is an identification tag having a specific pattern, located on an image projection surface on which an image is to be projected using a camera, and a marker recognition step of recognizing the marker; a control command input step of inputting a user's control command for a specific model to be processed by the user; a control command processing step of generating a predetermined model control command according to the command input in the control command input step; a model processing step of controlling the model according to the model control command and processing information about the model; a projection image generating step of generating a projection image of the model to be projected on the image projection surface; and an image projection step of projecting the projection image of the model onto the image projection surface using a projector.

In one embodiment, the model processing step rotates the model in two or three dimensions, scales the size of the model, selects some of the model components, or It may be characterized in that the operation of modifying the model is performed, and the structure information and constituent entity information of the model are processed.

In one embodiment, the projection image generating step creates model image content including information about the model controlled in the model processing step and the processed model, and in the model image content and the marker recognition step It may be characterized in that a projection image of the model to be projected on the image projection surface is generated using the recognized information of the marker.

In an embodiment, the generating of the projection image uses information on the shape and size of the marker recognized in the step of recognizing the marker, calculating the coordinates of the image projection surface on which the marker is located, and calculating the coordinates of the projector space. Calculating, generating a spatial transformation matrix between the coordinates of the image projection surface and the coordinates of the projector space, transforming the model image content using the generated spatial transformation matrix, and projecting the model to be projected on the image projection surface It may be characterized in that it generates an image.

In one embodiment, the projection image generating step projects a calibration image of a certain pattern on the image projection surface with the projector, photographing the projected calibration image with the camera, and information of the photographed calibration image It may be characterized in that a prior coordinate correction operation between the projector and the camera is performed using the information, and the coordinates of the projector space are calculated using the information of the prior coordinate correction operation.

According to the present invention, in the multi-faceted screen projection-based portable 3D model control system, the 3D model that has undergone normalization is projected using a projection device so that the user can project and view it on any multi-sided screen, and the user intuitively By making it possible to check, control, and modify the projected 3D model through the user interface, ordinary people who do not have professional knowledge in professional design programs can easily check and modify it through the intuitive user interface, and carry it at the job site. It has the effect of being able to project and view the design on an arbitrary surface while simultaneously confirming and modifying the design in real time from a variety of perspectives and perspectives by multiple work related parties.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as possible even if they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, preferred embodiments of the present invention will be described below, but the technical spirit of the present invention is not limited thereto and may be variously implemented by those skilled in the art without being limited thereto.

1 is a block diagram of a portable projection-based model control system according to an embodiment of the present invention. A portable projection-based model control system according to an embodiment of the present invention includes a marker 100, a camera unit 200, and a control command. It may include a processing unit 300 , an image processing unit 400 , an image projection unit 500 , a sensor unit 600 , and an input terminal unit 700 . The above embodiment is an optimal embodiment, the sensor unit 600 and the input terminal unit 700 may be omitted if necessary, and the portable projection-based model control system according to another embodiment of the present invention is a marker ( 100 ), a camera unit 200 , a control command processing unit 300 , an image processing unit 400 , and an image projection unit 500 . Hereinafter, the optimal embodiment will be described in detail.

The marker 100 is an identification tag having a specific pattern. Here, the marker 100 may be a marker having a rectangular grid pattern, partially filled with black and the rest with white. Here, the marker 100 may have two or more different colors or a grid pattern having different brightnesses. Here, the marker 100 has a certain surface in addition to the grid pattern, and various types of markers having a specific pattern having a mutually distinguishable color or brightness on the surface may be used.

Here, the marker 100 is preferably located on the image projection surface on which the image is to be projected, and may be continuously located on the image projection surface as needed, or may be removed after being located for a predetermined time.

The camera unit 200 photographs the marker 100 positioned on the image projection surface on which the image is to be projected. Here, the image projection surface means the surface on which the user projects the image through the image projection unit 500, which It can be the prepared screen for the projector, or it can be any surface on which the image can be projected, such as any vertical wall or paper surface, or a horizontal floor, paper or desk surface. In addition, the image projection surface may be a flat surface or a curved surface.

Here, the image projection surface may be a plurality of surfaces that can be separated by two or more separated or angled, or a curved surface. In this way, when the image projection surface becomes a plurality of surfaces, a plurality of markers 100 are positioned on each surface, and the image processing unit 400 converts the image to be projected on each surface by using the information of the markers 100 , respectively. It can be created through This will be described in detail below in the section describing the image processing unit 400 .

Here, the camera unit 200 may photograph a motion of a user touching the image projection surface or a motion of a hand or a finger. Here, in using the portable projection-based model control system according to the present invention, the user can control the model by touching the image projection surface or by using a hand or finger operation. 200 may take a picture and input the motion to the system. This will be described in more detail while explaining the control command processing unit 300 below.

The portable projection-based model control system according to the present invention provides a model control system that a user can carry and use in a desired working environment by using any surface other than the previously fixed surface as the image projection surface as described above. can do. Also, for example, the user can select two arbitrary surfaces located at the work site as the image projection surface, perform an operation to control the model through the image projection surface, and check the result in real time. Preferably, the two surfaces selected as the image projection surface by the user may be a wall surface and a floor surface or a desk surface positioned perpendicular to the wall surface.

In addition, here, it is preferable that the camera unit 200 is bound to the image projection unit 500 so that the image projection unit 500 can take an image projected by the image projection unit 500 while looking in the same direction as the image projection unit 500 moves. do. And, through this binding, the pre-coordinate correction (calibration) operation between the camera unit 200 and the image projection unit 500 can be performed more efficiently, as will be described in detail below.

The control command processing unit 300 generates a model control command by receiving a user's control command for a specific model to be processed by the user.

Here, the model may be a three-dimensional model or a two-dimensional model. And here, the model may be an architectural design model, a product design model, or various types of models having a certain structure and form. For example, in the case of an architectural design model, the model includes three-dimensional building information, information on each component such as walls, columns, and stairs constituting the building, that is, information such as the width of the wall or the height of the column, and It can be an architectural design model including information on drawings of a building including , floor plans, etc., and information on various furniture and accessories located in the building, that is, for example, information on the location and volume of furniture.

Here, the model control command is a control command promised in advance to control the model, and various types of commands can be defined according to a user's desire. For example, the model control command may be a command to rotate the model in two or three dimensions, or a command to scale the size of the model. Also, for example, the model control command may be a command for selecting a component or component of a model, a command for selecting a specific layer of a model, or a command for selecting a drawing of a model and displaying information about it. Also, for example, the model control command may be a command for modifying information about constituent objects, layers, and drawings of the selected model, or a command for modifying and processing structural information of the model.

For example, the model control command may be a command to rotate the 3D model of the building in 3D so that the user can see it from various directions.

In addition, for example, a user dealing with a building design model needs to enlarge or reduce the three-dimensional design model as needed. can be a command.

Also, for example, a user handling a building design model needs to access specific drawing information of the design model during the building process, so in this case, the model control command selects a specific drawing of the model to display the drawing and also It may be a command to display information about the configuration of the drawing.

Also, for example, since the user needs to modify the design model, in this case, the model control command selects and moves some of the components of the model, corrects the contents, or adds or deletes components to the model. It can be a correction order, such as

Here, the control command processing unit 300 may generate the model control command by recognizing the user's motion photographed by the camera unit 200 . Here, the camera unit 200 captures an operation in which the user touches the image projection surface, and the control command processing unit 300 interprets the user's motion information photographed as described above to obtain a previously promised model control command for the model. can create For example, when a user touches a part of the model with a finger or a pen in the image of the model displayed on the image projection surface, the camera unit 200 captures the user's movement, and the control command processing unit 300 performs the above It is possible to generate a model control command for performing an operation of selecting a part of the touched model by analyzing the user's touch operation photographed together. Then, when the user performs an operation to correct the selected part using a pen or a finger, the control command processing unit 300 may generate a model control command for performing an operation to correct the selected part of the model accordingly. .

The image processing unit 400 recognizes the marker 100 photographed by the camera unit 200, controls the model according to the model control command generated by the control command processing unit 300, and processes information about the model, , the image projection unit 500 generates an image for the model to be projected on the image projection surface by using the recognized information of the marker 100 . The image processing unit 400 will be described in more detail below with drawings of the image processing unit 400 .

The image projection unit 500 projects the image generated by the image processing unit 400 on the image projection surface. Here, the image projector 500 may be a beam projector that projects an image using light on a surface.

Here, the portable projection-based model control system according to the present invention generates an image for the model by performing a spatial coordinate transformation operation using the marker 100 in the image processing unit 400, and an image projection unit having only one projector ( 500), by projecting the above image onto at least two of the image projection surfaces at a time, at least two images of models that are distinguished from each other and whose display content can be adjusted respectively can be displayed on the image projection surfaces, respectively.

The sensor unit 600 may recognize various movements of the user, such as a motion of the user's hand or fingers. Here, when the sensor unit 600 operates, the control command processing unit 300 receives the user's movement from the sensor unit 600 . The control command may be input and the model control command may be generated.

Here, when the user performs a predetermined movement using both hands, one hand, or a finger, the sensor unit 600 recognizes the user's movement, interprets the movement, and transmits the user's movement information to the control command processing unit 300 . ), and the control command processing unit 300 interprets the user's motion information input from the sensor unit 600 again to generate a model control command for the model corresponding to the predetermined motion.

For example, if the user has promised in advance to rotate the model when the user raises both hands and rotates them in the air, the sensor unit 600 recognizes the rotation of both hands of the user and sends it to the control command processing unit 300 . In response to the user's movement, the control command processing unit 300 may generate the model control command corresponding to the content of rotating the model and send it to the image processing unit 400 .

The input terminal unit 700 is a device for the user to input a command for controlling the model. Here, when the input terminal unit 700 is used, the control command processing unit 300 receives the user's input from the input terminal unit 700 . The control command may be input and the model control command may be generated.

Here, the input terminal unit 700 may be a pen capable of recognizing a location and digitally documenting it using a sensor, or a mobile phone or tablet capable of screen display and screen touch. For example, when the user performs a screen touch operation to rotate the image for the model displayed on the mobile phone or tablet using a finger or a touch pen, the control command processing unit 300 receives the user's It is possible to generate the model control command corresponding to the model rotation by receiving the input information and analyzing it. Also, for example, when a user performs an operation of moving a component object of a model displayed on the image projection surface using a position-recognizable digital pen, the control command processing unit 300 receives information from the digital pen and uses it It is possible to generate the model control command corresponding to the model movement by analysis.

Hereinafter, the operation of the image processing unit 400 will be described in more detail.

1 , the image processing unit 400 may include a marker recognition unit 410 , a model processing unit 420 , and a projection image generation unit 430 .

The marker recognition unit 410 detects and recognizes the marker 100 located on the image projection surface. Here, the marker recognition unit 410 may receive an image of the marker 100 located on the image projection surface photographed by the camera unit 200 and detect the marker 100 from the image of the marker 100 .

The model processing unit 420 controls the model according to the model control command generated by the control command processing unit 300 and processes information about the model.

Here, the model processing unit 420 may control the model according to the content of the model control command as described above. For example, the model processing unit 420 rotates the model in two or three dimensions, scales the size of the model, selects some of the model components, or the model according to the content of the model control command. You can perform a control operation to modify

For example, when the user wants to check a specific plan view of the model, the user may select a plan view of a specific floor of the model through the input terminal unit 700, and the control command processing unit 300 provides The model control command is generated, and the model processing unit 420 may select a specific plan view of the model according to the model control command. Also, when the user wants to modify a specific plan view of the model, the user can modify a specific plan view of the model through the input terminal unit 700, and the control command processing unit 300 controls the model with such contents. A command is generated, and the model processing unit 420 may modify a specific plan view of the model according to the model control command.

Here, the model processing unit 420 may process structure information and constituent entity information of the model according to the model control command as described above.

For example, the information about the model processed by the model processing unit 420 may be floor plan information of a specific floor of the design model when the model is a three-dimensional architectural design model, and the shape of the wall surface and the floor surface of the design model It can be structural information such as width or length, or it can be information about constituent objects such as the location, shape, and volume of furniture arranged in the design model. Here, the model processing unit 420 may perform processing tasks such as storing, selecting, modifying, acquiring, and displaying information about the model as described above.

The projection image generating unit 430 generates model image content including the model controlled by the model processing unit 420 and information on the processed model, and recognized by the model image content and the marker recognition unit 410 . A projection image of the model to be projected on the image projection surface may be generated using the information of the marker 100 .

Here, the projection image generation unit 430 calculates the coordinates of the image projection surface on which the marker 100 is located, using the shape and size information of the marker 100 recognized by the marker recognition unit 410, and the image projection unit Calculate the coordinates of (500), generate a spatial transformation matrix between the coordinates of the image projection surface and the coordinates of the image projection unit 500, and transform the model image content using the generated spatial transformation matrix, and the image It is preferable to generate a projection image of the model to be projected onto a projection plane.

Here, the spatial transformation matrix may be a transformation matrix that preferably performs an Affine transformation. Here, the Affine transformation is a transformation that transforms one coordinate space into another coordinate space, and consists of a linear transformation and a translation transformation.

For example, such an Affine transformation may be performed through a transformation operation between each coordinate space as shown in Equation 1 below.

Claims (4)

In the portable projection-based model control system,
a marker that is located on an arbitrary image projection surface designated by a user, is movable, and is an identification tag having a specific pattern;
a camera unit for photographing the marker located on the image projection surface;
a control command processing unit that receives a user's control command for a specific model to be processed by the user and generates a model control command; and
Recognizes the marker photographed by the camera unit, controls the model according to a model control command generated by the control command processing unit, processes information about the model, and uses the recognized marker information to obtain the image An image processing unit for generating an image for the model to be projected on a projection plane,
The image processing unit specifies the model using the marker, calculates the position of the image projection surface on which the marker is located, using the shape and size of the recognized marker, and obtains the calculated position information of the image projection surface. A portable projection-based model control system, characterized in that it generates an image for the model to be projected on the image projection surface by using it.
According to claim 1,
Portable projection-based model control system, characterized in that it further comprises an image projection unit for projecting the image generated by the image processing unit on the image projection surface.
3. The method according to any one of claims 1 to 2,
The camera unit is a portable projection-based model control system, characterized in that the user touches the image projection surface or captures a motion of a hand or a finger.
3. The method according to any one of claims 1 to 2,
The model control command includes a command to rotate the model in two or three dimensions, a command to scale the size of the model, a command to select some of the components of the model, or a command to modify the model A portable projection-based model control system, characterized in that.