KR102236917B1 - Three dimensional modeling method using three dimensional depth map and region segmentation based on an single camear for assembly type education apparatus with polyomino - Google Patents

Abstract

The present invention relates to a method of performing three-dimensional modeling by using three-dimensional depth map generation and image segmentation technology based on a single camera, with respect to an assembly type teaching tool using a polyomino. The assembly type teaching tool using a polyomino includes: a base plate formed into a plate shape having a predetermined area to be placed horizontally and having first coupling parts aligned and placed in an upper part to form rows and columns; and a plurality of block parts comprising a polyomino having a shape of a single hexahedron or a shape of a plurality of hexahedrons coupled laterally, having second coupling parts formed in the bottom to be coupled with the first coupling parts, and assembled on the base plate by the first and second coupling parts to form various forms of block groups. The method of performing three-dimensional modeling by using three-dimensional depth map generation and image segmentation technology includes the steps of: (S100) creating a three-dimensional depth map of the base plate and the block parts from an original image obtained from a single camera; (S200) binarizing a background of the base plate and a panorama of the block parts for improving the contrast of the three-dimensional depth map; and (S300) generating a cube mesh of the block parts, based on the binarization result, generating a block when there are pixels, which are no less than predetermined threshold values, among pixels in each block area, and mapping colors of the block parts to generate a three-dimensional model of the block parts.

Description

3D modeling method using three-dimensional depth map generation and image segmentation technology based on a single camera for prefabricated teaching aids using polyomino {THREE DIMENSIONAL MODELING METHOD USING THREE DIMENSIONAL DEPTH MAP AND REGION SEGMENTATION BASED ON AN SINGLE CAMEAR FOR ASSEMBLY TYPE EDUCATION APPARATUS WITH POLYOMINO}

The present invention relates to a method of performing three-dimensional modeling using a three-dimensional depth map generation and image segmentation technology based on a single camera for a prefabricated teaching tool using polyomino. More specifically, a plurality of blocks made of polyomino are used. In the prefabricated teaching tool using polyomino where assembly learning is performed in the form of assembling on a plate, it is possible to create accurate three-dimensional modeling of the block part with a single low-end camera or a single webcam of a smartphone. Regarding the teaching tool, it relates to a method of three-dimensional modeling using a three-dimensional depth map generation and image segmentation technology based on a single camera.

Polyomino is a generic term for polygons created by sharing at least one side of n (n>1) squares, which was first used by Dr. Solomon W. Golomb of Harvard University in a mathematics lecture. . This class with Poliomino can greatly help students such as infants, children, and elementary school students learn and improve spatial and three-dimensional senses, and improve their creativity, so some related products are emerging. .

On the other hand, looking at the depth map generation technology through image processing and the 3D model generation method, the approach to interpreting and applying depth information of an image has been treated as a major problem in the field of computer vision for a long time. The most common approach was to analyze feature points using images captured using multiple cameras, and reconstruct the depth map through the mapping between them.

However, this method requires not only various equipment, but also various parameters reflecting the shooting environment to be measured. Due to this, in addition to cost constraints of having to have several equipments, there are constraints that must be subordinated only to a specific environment so that the change of parameters is small.

As a related conventional patent document, Korean Patent No. 10-0544677 (Applicant: Electronics and Telecommunications Research Institute, title of invention: 3D object tracking device and method using multi-view image and depth information {Apparatus and method for the 3D) In object tracking using multi-view and depth cameras}), a technique for implementing a depth map using at least two or more cameras is disclosed.

As another related conventional patent document, in U.S. Patent No. 10,096,115 (registered on October 9, 2018, applicant: BlackBerry Limited, title of invention: Building a depth map using movement of one camera), In fact, a method for generating a depth map based on the same information as captured by several cameras is disclosed.

Regarding this, recently, approaches have been attempted to overcome the existing constraints using only a single image rather than using multiple images for depth map reconstruction. In order to reconstruct depth information using only a single image, the semantic interpretation of elements of the scene must be the basis, and for accurate results, there is a problem that the analysis must be performed on a pixel-by-pixel basis.

In short, some products related to Polyomino are being released in the past, and in terms of image processing, it was a stage in which an attempt was made to overcome the existing limitations with only a single image. In other words, even if the prior art for polyomino covered by the present invention and the prior art related to depth map and 3D modeling are combined, a folio in which assembly learning is performed in the form of assembling a plurality of blocks made of polyomino on a base plate. In the prefabricated teaching aid using Mino, it is possible to create an accurate three-dimensional modeling for the block part even with a single low-end camera of a smartphone or a single webcam.For prefabricated teaching aids using polyomino, a three-dimensional depth map is generated based on a single camera There was no disclosure of how to perform 3D modeling using the image segmentation technique.

Korean Patent No. 10-0544677 (Applicant: Electronics and Telecommunications Research Institute, title of invention: Apparatus and method for the 3D object tracking using multi-view and depth cameras}) U.S. Patent No. 10,096,115 (registered on October 9, 2018, Applicant: BlackBerry Limited, title of invention: Building a depth map using movement of one camera)

The present invention was created in order to solve the above-described problems, and an object of the present invention is to assemble a plurality of blocks made of polyomino on a base plate in a prefabricated teaching tool using polyomino in which assembly learning is performed, a smart phone It is possible to generate accurate depth maps and 3D modeling for the block part even with a single camera or a single webcam, and for prefabricated teaching aids using polyomino, it is possible to create a 3D depth map and image segmentation technology based on a single camera. It is to provide a way to do three-dimensional modeling.

The method according to the present invention for achieving the above object is a method of performing three-dimensional modeling using a three-dimensional depth map generation and image segmentation technology based on a single camera, for an assembly-type teaching tool using polyomino. The prefabricated teaching aid is formed in a plate shape having a predetermined area, and is horizontally arranged, and a base plate arranged so as to form a row with a first coupling portion at an upper portion thereof; And a polymino having a shape of a single hexahedron or a shape in which a plurality of hexahedrons are laterally fastened, and a second coupling portion coupled to the first coupling portion is formed on the bottom surface, and a second coupling portion with the first coupling portion A block unit that forms a set of blocks of various types while being assembled on the base plate by a plurality of units, wherein the three-dimensional modeling method using the single camera-based three-dimensional depth map generation and image segmentation technology is obtained from a single camera. Generating a three-dimensional depth map of the block portion and the base plate from an original image (S100); Binarizing the foreground of the block portion and the background of the base plate to improve the contrast of the three-dimensional depth map (S200); And generating cube mesh data of the block unit based on the binarization result, generating a block when the pixel in each block area is equal to or higher than a predetermined threshold, and mapping the color of the block unit to the block unit It includes; generating a three-dimensional model (S300).

In addition, in the step of generating a three-dimensional depth map of the block portion and the base plate from the image acquired from the single camera (S100), spatial information on the block portion and the base plate existing in the image is extracted, and the depth It uses a multiscale deep neural network that can estimate information.

In addition, in the step of generating a three-dimensional depth map of the block portion and the base plate from the image acquired from the single camera (S100), the multi-scale deep neural network consists of a bottleneck module continuously configured, and the The bottleneck module performs convolution using a dilated layer that does not increase parameters while maintaining the size of the feature map, and uses an asymmetric kernel to set an interval within the convolution filter by a certain distance, so that only fewer parameters are used. An asymmetric convolutional layer and a full convolutional layer to have a receptive field, and are configured to include three convolutional layers.

In addition, in the step of generating a three-dimensional depth map of the block unit and the base plate from the image acquired from the single camera (S100), when inferring the depth map from the output value of the multiscale deep neural network, a depth range ) Has a section between a predetermined depth α and a depth β, the larger the depth value is, the larger the weight is, so that the larger depth value has a wider error range in order to reduce the error during the depth inference.

In addition, in the step of generating a three-dimensional depth map of the block unit and the base plate from the image acquired from the single camera (S100), when inferring the depth map from the output value of the multiscale deep neural network, a depth range ) Has a predetermined range of depth α and depth β (depth α <depth β), the possible value ti as depth is

를 가지며, 상기 K는 설계자에 의한 상수값이고, i는 자연수이며, ti의 최소값이 깊이 α이며 ti의 최대값이 깊이 β이다.

And K is a constant value by the designer, i is a natural number, the minimum value of ti is the depth α, and the maximum value of ti is the depth β.

Further, the difference between the depth α and the depth β is a value corresponding to the height of the block portion or a natural multiple of the height of the block portion.

In addition, to obtain information from the depth map, median filtering may be further included to improve image quality by removing noise that may occur during the process of being created through the multiscale deep neural network.

In addition, in the step of binarizing the foreground of the block portion and the background of the base plate in order to improve the contrast of the 3D depth map (S200), the step of converting the inferred depth map to the original image size by up-sampling again Includes.

Further, in the step S300, when the color is mapped to the area of the cube mesh data, the color of the cube mesh data is initially set as a block designation color so that the calculation speed can be improved, and the original corresponding to the cube mesh data is Comparing the average color of the image, and mapping the color to the nearest color.

In addition, in the step S300, the original image having an RGB model is converted to light changes in order to prevent color distortion due to external influences including illuminance and shadow when color is mapped to the area of the cube mesh data. And changing to a Hue-Saturation-Value (HSV) model with relatively little influence, and generating a correction image by selecting a value having a smaller error using the H value of the HSV model and the color average.

In addition, the prefabricated teaching aid using the polyomino is formed in a plate shape arranged horizontally, and is arranged to face the base plate vertically, and an image of an object shape including an animal is formed so that each block portion is formed in the shape of the object on the base plate. It may further include a; guide image portion for guiding to be assembled.

According to a method of performing three-dimensional modeling using a three-dimensional depth map generation and image segmentation technology based on a single camera for the prefabricated teaching aid using polyomino according to the present invention,

First, unlike the method of analyzing feature points using images taken with multiple cameras and reconstructing the depth map through the mapping between them, various equipment is required, which not only increases the cost, but also includes various parameters reflecting the shooting environment. It is possible to create an accurate three-dimensional modeling of the block portion even with a single low-end camera or a single webcam of a smartphone without having the constraints to be measured.

Second, it is possible to link a device capable of interacting with a user who uses a prefabricated teaching aid using polyomino. A more accurate depth map can be generated by using the original image acquired by shooting through a single camera of a smart phone, and further, accurate three-dimensional modeling of the block portion is possible.

Third, when inferring the depth map from the output value of the multiscale deep neural network, the larger the depth value, the larger the weight is set so that the larger the depth value has a wider error range. It is possible to reduce.

Fourth, in order to obtain information from the depth map, it is possible to further improve the image quality by further including median filtering to remove noise that may occur in the process of being created through a multiscale deep neural network.

Fifth, when color mapping for the area of the cube mesh data, the color of the cube mesh data is initially set with the block designation color, and the color average of the original image corresponding to the cube mesh data is compared, and the color is selected with the closest color. By mapping, the computational speed can be improved.

Sixth, by changing the original image with the RGB model to an HSV model that has relatively little influence on changes in light, color distortion due to external influences including illumination and shadows is prevented when color mapping to the area of the cube mesh data. Can be prevented.

1 is a block diagram of a deep neural network network for estimating a depth map based on a single image.
2 is a structure of a network proposed in the present invention.
3 is a diagram illustrating that the size of an input image is reduced by combining initial blocks.
FIG. 4 shows a bottleneck module composed of three convolutional layers.
5 is a diagram for a dysed convolution used in the present invention.
6 is a diagram for an asymmetric convolution used in the present invention.
FIG. 7 is a diagram illustrating a conventional method of uniformly dividing a range of depth and a method of dividing by a method of giving a weight to a large value proposed in the present invention.
8 is a diagram illustrating a process in which an input image is finally generated as a depth map by a deep network according to the present invention.
9 is a diagram illustrating a result of applying a binarization technique in a depth map inferred according to the present invention and generation of cube mesh data.
FIG. 10 is a diagram illustrating mapping of colors to the cube mesh data area generated in FIG. 9.
11 is an image acquired from a single camera, i.e., an RGB model is changed to an HSV model, a standardized color system is used for color matching, and a value with less error is selected using the H value and the color average of the HSV model, It is a diagram explaining the process of creating.
12 is a diagram showing a single image acquired by a general webcam z camera, a depth map inference result, and a modeling result according to the present invention.
13 is an exploded perspective view showing the configuration of a prefabricated parish using polyomino according to a preferred embodiment of the present invention.
14 and 15 are perspective views showing the configuration of a block unit according to a preferred embodiment of the present invention.
16 is a plan view showing various polyomino shapes of a block portion according to a preferred embodiment of the present invention.
17 is a plan view showing a state in which a first guide line and a second guide line are displayed on a guide image part according to a preferred embodiment of the present invention.
18 is a perspective view showing a state in which each block portion is assembled on a base plate according to a preferred embodiment of the present invention.
19 is a perspective view showing a configuration in which a base plate is mounted inside a case according to a preferred embodiment of the present invention.
20 is a perspective view showing a configuration in which a lamp and a control unit are provided in a guide image unit according to a preferred embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, terms or words used in the present specification and claims should not be construed as being limited to their usual or dictionary meanings, and the inventors appropriately explain the concept of terms in order to explain their own invention in the best way. Based on the principle that it can be defined, it should be interpreted as a meaning and concept consistent with the technical idea of the present invention.

Accordingly, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all the technical spirit of the present invention. It should be understood that there may be equivalents and variations.

(Prefabricated parish using polyomino)

The prefabricated teaching aid using polyomino, which is a target of application of a method of generating a three-dimensional depth map based on a single camera and performing a three-dimensional modeling using image segmentation technology according to a preferred embodiment of the present invention, includes a plurality of block parts made of polyomino. Creativity by providing an image (I) designed in the shape of an object including an animal on the base plate 110 by assembling learning in the form of assembling 120 on the base plate 110 to develop creativity and small muscles. As a learning tool that can guide the insufficient infants to easily follow the assembly learning, as shown in Figs. 13 to 16, including a base plate 110, a block unit 120 and a guide image unit 130 do. Here, it is not necessary to include an animal in the image (I), it means that the image (I) itself may be formed in the shape of an animal as shown in the drawing, and there is no limitation on the type of the image to be designed.

First, the base plate 110 is a plate member that provides a space for assembly learning and is formed in a plate shape having a certain area and is horizontally arranged, and a plurality of first coupling units 112 are arranged in a row on the upper surface. .

Here, as shown in FIG. 13, the base plate 110 includes a plurality of unit plate members 111 formed in a square or rectangular plate shape in which a plurality of first coupling portions 112 are formed on an upper surface, and each unit Around the lateral circumference of the plate member 111, a coupling protrusion 113 and a coupling groove 114 for lateral coupling of other adjacent unit plate members 111 are disposed on different sides, so that each unit plate member 111 is Since it can be assembled into a plate shape having a certain area while being laterally combined, the shape or size of the base plate 110 can be determined according to the user's level or preference.

In addition, the block unit 120 is a tool used to perform assembly learning, and as shown in FIG. 14, a polyamino furnace having a shape of a single hexahedron 121 or a shape in which a plurality of hexahedrons 121 are laterally fastened. It is made, and a second coupling portion 122 in a form that meshes with the first coupling portion 112 is formed on the bottom surface. Accordingly, as shown in FIG. 18, a plurality of blocks are assembled on the base plate 110 by the first coupling part 112 and the second coupling part 122 to form various types of block sets.

Here, the first coupling part 112 is made of a groove recessed downwardly in a cross shape on the upper surface of the base plate 110, and the second coupling part 122 is a predetermined length at the center of the bottom surface of each hexahedron 121 Since it is composed of a cross-pillar extending downwardly and can be fastened while being inserted into the first coupling part 112, it is possible to minimize the flow of the block part 120 in the assembled state while being easy to assemble.

In addition, the first coupling portion 112 is made of a cross groove that is recessed like the illustrated second coupling portion 122, and the second coupling portion 122 is a cross that protrudes like the illustrated first coupling portion 112 Consisting of a column shape, the same assembly function can be implemented.In addition, projections and grooves of various shapes such as a combination of a cylindrical protrusion and a circular groove, a combination of a square groove and a square pillar, and a combination of a straight plate-shaped protrusion and a straight groove It can also be made in the shape of.

In addition, the first coupling portion 112 and the second coupling portion 122 are composed of a combination of a magnet having a + polarity and a magnet having a-polarity in addition to the combination of protruding protrusions and grooves. In addition, the first coupling part 112 is formed of a protruding line protruding upward in a line shape on the base plate 110, and the second coupling part 122 is a form in which an edge portion is supported on each protrusion line. Consisting of, it is possible to make a connection between each other. In addition, except for the configuration of the first coupling portion 112 and the second coupling portion 122, each block portion 120 may be seated on the upper surface of the base plate 110, so that assembly learning may be performed.

In addition, as shown in Fig. 14, a plurality of elastic grooves 123 formed in a shape that is concave upwardly spaced apart at regular intervals along the circumference around the second coupling portion 122 at the bottom of each hexahedron 121 By being formed, the second coupling part 122 in the form of a cross can be inclined at a certain angle with respect to the body of the block part 120, so that the block part 120 assembled from the horizontally arranged base plate 110 is constant. It can be easily separated upwards in a direction inclined at an angle.

In addition, on the upper surface of each hexahedron 121, a third coupling portion 124 made of a groove recessed downwardly in a cross shape or a square shape is formed, and on the block portion 120 assembled to the base plate 110 A block set may be formed in a three-dimensional shape while assembling the other block part 120.

In addition, the second coupling portion 122 and the third coupling portion 124 may have various assembly standards that mesh with each other, so that the difficulty of assembling learning may be adjusted.

More specifically, each second coupling portion 122 has a shape of various pillars such as a cross, a triangular pillar, and a square pillar, and each third coupling portion 124 is a cross groove, a triangular groove, and a square groove. There may be various assembly standards that are meshed with each other, such as formed in the shape of, and in the case of forming a three-dimensional block set in the form of stacking each block part 120 into a plurality of layers through these various assembly standards, assembling each layer It is possible to limit the assembly specifications of the specific shape possible.

For example, on the first floor fastened to the base plate 110, only the block portion 120 having the second coupling portion 122 of a cross column may be assembled, and the third coupling portion ( When 124 is in the shape of a triangular groove, only the block portion 120 having the second coupling portion 122 of a triangular column is assembled on the second floor, and the third coupling portion of the block portion 120 assembled on the second floor ( Only the block portion 120 having the second coupling portion 122 having a shape corresponding to the 124) can be assembled.

Meanwhile, the guide image unit 130 is an image member that guides the learner to assemble the block unit 120 into a specific object shape, and is formed of a film or plate shape having an area equal to or relatively small to the base plate 110. It is arranged to face the base plate 110 up and down, and an image (I) in the shape of an object including an animal is formed to guide each block portion 120 to be assembled in the shape of the object.

Here, as shown in FIG. 18, the block unit 120 is made of a plurality of different colors, and the image (I) is made by coloring each color included in the block unit 120 into an object shape. It can increase the image effect of the block set and induce more interest in assembly learning.

In addition, as shown in FIGS. 13, 17 and 18, a first guide line 132 formed in a grid shape having a spacing corresponding to the width of each hexahedron 121 is formed in the guide image unit 130 Since the assembly position of each block unit 120 can be guided, the target of the block unit 120 having a specific shape to be assembled in an empty space can be predicted, and through this, the first guide line 132 is assembled with or without You can adjust the difficulty of learning.

In addition, the guide image unit 130 includes a second guide line 133 formed in a shape corresponding to the planar outline of each block unit 120 to guide the assembly position of the block unit 120 having a specific polyomino shape. Since it may be formed, a position to be assembled in a polyomino shape of each block unit 120 may be indicated, and the difficulty of assembling learning may be more precisely adjusted with the presence or absence of the second guideline 133 through this.

Here, although the drawing illustrates that one second guideline 133 is formed, a plurality of or a plurality of second guidelines ( 133) may be formed.

In addition, the guide image unit 130 has a coloring surface that can be colored with any one of a pencil, colored pencil, crayon, or a color pen on the upper surface, or is made of a material that can be colored with the tool, such as paper, so that the learner can use the guide image Since each block unit 120 can be assembled on the unit 130 in the shape of the object in the pattern by hand, drawing learning can be performed at the same time, and the fun and attachment of assembly learning can be induced.

In addition, the guide image unit 130 has a rectangular center formed by the first guide line 132 in the vertical direction and the first guide line 132 in the horizontal direction, that is, at a position corresponding to the first coupling unit 112. A through hole 134 having a shape corresponding to the planar shape of the first coupling portion 112 or the second coupling portion 122 is formed to assemble the block portion 120 on the base plate 110 while assembling the guide image portion ( A position for inserting the second coupling portion 122 into the first coupling portion 112 of the base plate 110 disposed under the 130) can be accurately indicated.

In addition, the guide image unit 130 is made of a hard material having a predetermined rigidity, and each block unit 120 assembled to the base plate 110 can be separated upward at the same time by lifting the guide image unit 130. have. For this purpose, the guide image unit 130 is preferably made of a high-strength synthetic resin material or a metal material. Therefore, it is possible to reduce the learning preparation time required while separating a plurality or a plurality of assembled block units 120 individually.

At this time, as shown in FIG. 13, the guide image part 130 is equipped with a handle 135 used to separate each block part 120 upwardly from the base plate 110 on one side, and a through hole 134 ) Is formed in a long hole shape with a long length in the direction of one side where the handle 135 is mounted, so that the block part 120 can be easily separated in a manner that is tilted from one side to the other side around the base plate 110. have.

The drawing illustrates a structure in which the guide image part 130 is seated on the upper part of the base plate 110 so that each block part 120 is assembled to the base plate 110 with the guide image part 130 therebetween. It is not limited, and the guide image unit 130 may provide an image I even in a structure that is disposed under or inserted into the base plate 110.

To this end, the base plate 110 is made of a transparent or translucent light-transmitting material, and the guide image unit 130 includes the base plate 110 so that the image I is transmitted upwardly through the surface of the base plate 110. ) Can be placed horizontally underneath. In addition, a slit-shaped insertion groove is formed inside the base plate 110, and the guide image portion 130 is inserted and mounted so that the guide image portion 130 is horizontally disposed therein to provide an image (I).

In addition, when the guide image portion 130 is disposed under the base plate 110 and the second coupling portion 122 of the block portion 120 does not reach the guide image portion 130, the through hole 134 is It may not be formed, and the through hole 134 may be formed only in an image representing a specific object within the entire area of the guide image unit 130. That is, the through hole 134 is formed only in the area where the block unit 120 is assembled within the entire area of the guide image unit 130.

In addition, the guide image unit 130 has a rectangular center formed by the first guide line 132 in the vertical direction and the first guide line 132 in the horizontal direction, that is, at a position corresponding to the first coupling unit 112. A through hole 134 having a shape corresponding to the planar shape of the first coupling portion 112 or the second coupling portion 122 is formed to assemble the block portion 120 on the base plate 110 while assembling the guide image portion ( A position for inserting the second coupling portion 122 into the first coupling portion 112 of the base plate 110 disposed under the 130) can be accurately indicated.

Meanwhile, as shown in FIG. 19, a case 140 for accommodating the base plate 110 may be further included. In addition, when the base plate 110 is made of a light-transmitting material, the guide image unit 130 may be horizontally disposed under the base plate 110 so that the image I is transmitted upwardly through the surface of the base plate 110. have.

In addition, as shown in Fig. 20, the guide image unit 130 is arranged to form a row of rows corresponding to each of the first coupling units 112, and drives light emission according to a control signal, and a plurality of display light is emitted upwardly. The lamp 137 and the control unit 138 for driving and controlling each lamp 137 so that the display light emitted from each lamp 137 forms a designated image I may be further included.

Therefore, each lamp 137 is driven and controlled so that images I for various conventional objects are displayed on the guide image unit 130 without the need to replace the different guide image unit 130 for each image I. It is possible to provide a more diverse type of image (I) and to facilitate conversion of the image (I).

In addition, as shown in the drawing, each guide image unit 130 is vertically disposed between each lamp 137 to block the display light emitted from the lamp 137 from incident sideways to the light emitting region of the adjacent lamp 137 It is preferable to further include a partition wall (139).

(Method of performing 3D modeling using 3D depth map generation and image segmentation technology based on a single camera)

(Generate 3D depth map based on a single camera)

What is proposed in the present invention is a deep network for generating a depth map. In the stereo image-based depth map estimation method, a method of estimating depth information using the displacement of corresponding pixels between two images is used, but in a single image, the spatial structure of objects in the image is determined to estimate the depth information. By analyzing, the depth map can be estimated from the relative perspective between objects.

In order to estimate the depth information using the perspective obtained through the analysis of a single image, more accurate depth information is extracted by estimating the relative perspective of local objects existing in the image and extracting the global perspective information that exists in the entire image. I can.

That is, an accurate depth map through a method of grasping approximate depth information using global spatial information from an input single image, and improving depth information in detail for the previously extracted global depth information using local spatial information. Can be estimated.

1 is a block diagram of a deep neural network network for estimating a depth map based on a single image. As shown in FIG. 1, the present invention relates to a new multiscale deep neural network structure in which an ENet capable of extracting spatial information existing in an image and estimating depth information is modified. 2 is a structure of a network proposed in the present invention.

The basic structure of the network of the present invention is composed of a series of bottleneck modules modified from ResNets, and additionally combines an initial block at the front end of the network to reduce the size of an input image. 3 is a diagram illustrating that the size of an input image is reduced by combining initial blocks.

The bottleneck module consists of three convolutional layers. FIG. 4 shows a bottleneck module composed of three convolutional layers. (1) 1×1 projection: dimension reduction of feature maps (2) convolution, dilated (Fig. 5), asymmetric (Fig. 6), full convolution parallelism. (3) 1×1 expansion. Here, batch normalization and PReLU are applied between all convolutions.

Now, a technique for improving the efficiency of convolutional neural networks will be described.

In order to prevent the loss of pixel-level features of the feature map due to repeated convolution and pooling, the size of the feature map is designed to be maintained in each layer. In order to obtain a feature map of a rich size, as many learning parameters are required, the same effect can be obtained by reducing the parameters by changing the existing convolution operation.

Directed convolution maintains the size of the feature map and achieves the same effect without significantly increasing the parameter. For reference, FIG. 5 is a diagram for a directed convolution used in the present invention.

6 is a diagram for an asymmetric convolution used in the present invention. By setting an interval within a convolutional filter by a certain distance, it is possible to have a large Receptive field using only fewer parameters. As shown in FIG. 6, asymmetric convolution is a technique of performing continuous convolution using an asymmetric kernel. In order to perform general 3×3 convolution, if you perform convolution using a 3×1, 1×3 (=6) type kernel instead of using a 3×3 (=9) type kernel, the same effect with only 6 parameters Can be obtained.

Now, the method of converting the output value to the depth map will be explained. In the approach to infer the depth map using a neural network, the global contextual information given to the image is grasped using convolution, and based on this, the depth is determined through regression analysis. Infer the map. However, in the case of this approach, since it is unsuitable for handling depth data having a continuous property, in the present invention, a classification problem of predicting each range is solved by dividing the continuous depth values by ranges.

Here, in depth map inference, the larger the depth value, the greater the inference error.Therefore, in order to reduce the error in the depth range prediction, the larger the depth value, the larger the weight to have a sufficiently wide error range for a large depth value. Introduced a method of putting.

FIG. 7 is a diagram illustrating a conventional method of uniformly dividing a range of depth and a method of dividing by a method of giving a weight to a large value proposed in the present invention.

When the above equation is described with reference to FIG. 7, it is possible to infer depth uniformly dividing ti in a depth range having a section of depth α and depth β, but taking into account various inference errors, the above equation In other words, as in the last line of FIG. 7, the weight of the data range is lowered for a small value and a large weight is given for a large value. Here, K is a constant value by the designer. By doing this, it is possible to accommodate a large range of errors that a large depth value may have. In addition, the difference between the depth α and the depth β is a value corresponding to the above-described height of the block portion or a natural multiple of the height of the block portion.

The transformed depth map is converted back to the original image size by up-sampling, for example, bilinear interpolation. 8 is a diagram illustrating a process in which an input image is finally generated as a depth map by a deep network according to the present invention.

(How to do 3D modeling)

In order to obtain information from the depth map, noise that may occur in the process of being created through the deep network is removed by median filtering to improve image quality. In order to improve the contrast of the depth map image, a binary method of fragmenting the range of the foreground and background is applied.

Now, cube mesh generation and color mapping for the binarized image will be described. The binarized result is mapped to a block buffer and cube mesh data is generated for each block. Whether to generate a block is determined based on a threshold for a white ratio in each area of the block (eg, 70% of the pixels in the block). 9 is a diagram illustrating a result of applying a binarization technique in a depth map inferred according to the present invention and generation of cube mesh data.

FIG. 10 is a diagram illustrating mapping of colors to the cube mesh data area generated in FIG. 9. In order to improve the computational speed, the cube block is stored in a block-designated color, and the average color of the original image corresponding to the mesh data is compared to map the color to the closest color. In order to ensure robustness from camera distortion, it is also possible to change the color model. This is because a single color image obtained from a camera may cause color distortion due to external influences such as illuminance and shadow.

Since 3D modeling relies only on camera images to extract and color accurate colors, color distortion can have a big impact on modeling accuracy. It is possible to secure robustness against distortion by changing a color image with an RGB model to a Hue-Saturation-Value (HSV) model that has relatively little influence on changes in light. It has a standardized color system for color matching, and it is possible to select a value with less error using the H value and color average of the HSV model. 11 is an image obtained from a single camera, that is, an RGB model is changed to an HSV model, a standardized color system is used for color matching, and a value with a small error is selected using the H value and the color average of the HSV model, and a correction image It is a diagram explaining the process of creating.

Next, the experimental results in the case of applying the present invention will be described. Quantitative performance is evaluated by three measurement methods. The three measurement methods are as follows. (1) Relative Difference: The average of the difference between the value measured by Intel RealSense and the depth information obtained from the system, (2) RMSE (Root Mean Square Error): The value measured by Intel RealSense and the depth information obtained from the system. Mean squared error between, (3) 3D model creation time: It should be run on low-end systems such as smartphones, so performance test only with CPU, not high-performance GPU (Windows 10, RAM 32GB, Intel CPU i7-7700K version)

Here, a database image including a single captured image was obtained. (More than 20 images with a size of 640 x 480 or larger) And, for each image, using an Intel RealSense depth camera, ground-truth data was obtained and used in a comparative experiment, and excellent performance was measured as shown in the table below.

Evaluation item unit Measured performance Relative difference error 0.125 RMSE error 0.225 3D model creation time ms 100

12 is a diagram showing a single image acquired by a general webcam z camera, a depth map inference result, and a modeling result according to the present invention. As shown in FIG. 12, in order to verify the performance of the modeling system proposed by the present invention, various block samples were made and experiments were conducted. Block images captured by a general webcam were transmitted as input to the neural network, and then inferred from the neural network. After post-processing the result and the input image, the final 3D model was well created.

As described above, although the present invention has been described by the limited embodiments and drawings, the present invention is not limited thereto, and the technical spirit of the present invention and the following by those of ordinary skill in the technical field to which the present invention pertains. It goes without saying that various modifications and variations are possible within the equal range of the claims to be described.

Claims (11)

As a method of performing three-dimensional modeling for prefabricated teaching aids using polyomino, using a three-dimensional depth map generation and image segmentation technology based on a single camera,
Prefabricated teaching aids using the polyomino,
A base plate formed in a plate shape having a certain area and arranged horizontally and arranged to form a row with a first coupling portion at an upper portion thereof; And a polymino having a shape of a single hexahedron or a shape in which a plurality of hexahedrons are laterally fastened, and a second coupling portion coupled to the first coupling portion is formed on the bottom surface, and a second coupling portion with the first coupling portion A plurality of blocks are assembled on the base plate by the unit to form a block assembly of various types; including,
The three-dimensional modeling method using the single camera-based three-dimensional depth map generation and image segmentation technology includes the steps of generating a three-dimensional depth map of the block unit and the base plate from an original image obtained from a single camera (S100); Binarizing the foreground of the block portion and the background of the base plate to improve the contrast of the three-dimensional depth map (S200); And based on the binarization result, generating cube mesh data of the block unit, generating a block model based on a pixel whose white ratio is greater than or equal to a threshold value, and assigning the color of the block unit to the block model. Including; mapping to generate a three-dimensional model of the block unit (S300),
In the step (S100) of generating a three-dimensional depth map of the block portion and the base plate from an image obtained from the single camera, spatial information on the block portion and the base plate existing in the image is extracted, and depth information is obtained. Using a multiscale deep neural network that can be estimated,
In the step (S100) of generating a three-dimensional depth map of the block portion and the base plate from the image acquired from the single camera,
The multiscale deep neural network is composed of a bottleneck module that is continuously configured, and the bottleneck module is a dilated layer that does not increase the number of parameters while maintaining the size of the feature map. Convolution is performed to set an interval within the convolution filter by a certain distance to have a receptive field with a smaller number of parameters compared to general convolution. It is composed including the lusion layer,
In the step (S100) of generating a three-dimensional depth map of the block portion and the base plate from the image acquired from the single camera,
When inferring the depth map from the output value of the multiscale deep neural network, when the depth range has a section between the predetermined depth α and the depth β, when inferring the depth map from the output value of the multiscale deep neural network In order to reduce the error, the larger the depth value, the greater the weight to have a wider error range for a larger depth value.
In the step (S100) of generating a three-dimensional depth map of the block portion and the base plate from the image acquired from the single camera,
When inferring the depth map from the output value of the multiscale deep neural network, when the depth range has a section between the predetermined depth α and the depth β (depth α <depth β), a possible value ti as a depth is

And K is a constant value by the designer, i is a natural number, the minimum value of ti is the depth α, and the maximum value of ti is the depth β,
A method of performing three-dimensional modeling for prefabricated teaching aids using polyomino using three-dimensional depth map generation and image segmentation technology based on a single camera.
delete delete delete delete The method of claim 1,
The difference between the depth α and the depth β is a value corresponding to a height of the block portion or a natural multiple of the height of the block portion,
A method of performing three-dimensional modeling for prefabricated teaching aids using polyomino using a three-dimensional depth map generation and image segmentation technology based on a single camera.
The method of claim 6,
It characterized in that it further comprises median filtering to improve image quality by removing noise that may occur when information is obtained from the depth map through the multiscale deep neural network,
A method of performing three-dimensional modeling for prefabricated teaching aids using polyomino using a three-dimensional depth map generation and image segmentation technology based on a single camera.
The method of claim 1,
In the step of binarizing the foreground of the block portion and the background of the base plate to improve the contrast of the 3D depth map (S200), the inferred depth map is again upsampled and converted into an original image size. ,
A method of performing three-dimensional modeling for prefabricated teaching aids using polyomino using a three-dimensional depth map generation and image segmentation technology based on a single camera.
The method of claim 8,
In the step S300, when the color is mapped to the area of the cube mesh data, the color of the cube mesh data is initially set as a block-designated color so that the calculation speed can be improved, and the original image corresponding to the cube mesh data is Comparing the color average, characterized in that it comprises the step of mapping the color to the nearest color,
A method of performing three-dimensional modeling for prefabricated teaching aids using polyomino using a three-dimensional depth map generation and image segmentation technology based on a single camera.
delete The method of claim 1,
The prefabricated teaching aid using the polyomino includes a guide image unit disposed on the base plate and configured to guide each block unit to be assembled in the shape of the object on the base plate by forming an image in the shape of an object including an animal; Included,
A method of performing three-dimensional modeling for prefabricated teaching aids using polyomino using a three-dimensional depth map generation and image segmentation technology based on a single camera.

Images