KR102159236B1 - Terminal and method for surpporting 3d printing, computer program for performing the method - Google Patents

Abstract

A method of supporting 3D printing includes the steps of recognizing a product; Searching and providing a 3D model existing on the web related to the recognized product; Calculating and simulating suitability between the recognized product and the searched 3D model, based on the information of the recognized product and the searched 3D model; And transmitting the 3D model information to the 3D printer to produce the 3D model. Accordingly, it is possible to propose a technology that can actively utilize 3D printing technology for DIY and business models in various fields.

Description

A terminal and method that supports 3D printing, and a computer program for performing the method {TERMINAL AND METHOD FOR SURPPORTING 3D PRINTING, COMPUTER PROGRAM FOR PERFORMING THE METHOD}

The present invention relates to a terminal and method supporting 3D printing, and a computer program for performing the method, and more particularly, to a terminal and method supporting 3D printing using augmented reality in a mobile environment, and a method for performing the method. It's about computer programs.

Recently, 3D printing technology has caused a monumental change in product development. Open source hardware is attracting attention as a third industrial revolution leading the popularization of product design/production by the general public along with additive manufacturing technology. Users who played the role of unilateral consumers in the past use 3D printers to design and produce new products or parts for maintenance. In addition, the number of 3D printable models that can be used for free has already exceeded 120,000 and is increasing dramatically. As such, with the spread of 3D printers and open source hardware, do it yourself (DIY) by consumers is becoming popular.

In addition, the web has been playing a role of a platform for sharing knowledge about open source hardware as well as knowledge about open source software. The formation of a consensus on the value of the sharing economy along with information sharing is realizing the value of sharing 3D models of products that can be used in real life with open source hardware along with the popularization of 3D printers. Accordingly, it is possible to form a consensus for the web, crowd sourcing, social networks, and sharing economy.

Meanwhile, with the spread of smartphones, tasks that were possible in conventional workstations have become possible with smartphones in the hand. Google is showing a prototype of a smartphone and tablet computer equipped with a 3D camera (depth camera) through the Tango project. These changes are accelerating the realization of augmented reality technology and enable mobile terminals to more easily and quickly acquire information about the real world and recognition of objects. In addition, it is possible to realize product recognition for DIY and tracking of 6-degree-of-freedom parts for augmented reality in mobile devices.

However, the problem of productive utilization remains behind the popularization of 3D printers. The majority of consumers who can buy a printer are not interested in the technical understanding of the modeling process that will generate the 3D model to feed the printer and want a user-friendly way. Therefore, the demand for personalized custom products that meet individual needs is increasing, but there is a need for a supply and demand method of a 3D model to meet the demand.

In addition, most of the simple DIY demands that occur in real life are related to maintenance and remodeling of products owned by consumers. In order to produce DIY parts that suit the user's intention, appropriate DIY parts are presented according to the physical information of the target product. A simple service is required.

In addition, with the spread of smart mobile devices, modeling methods for touch pads and tablets have been developed, but since consumers do not pay attention to these technical modeling methods, the alternative design method of the knowledge sharing method that solves the modeling itself or replaces it. Is required.

Furthermore, despite the popularization of 3D printers, the initial investment cost hinders the actual use of 3D printers by consumers. In order to induce easy DIY by providing a production method without initial investment cost, it is required to create a 3D printing environment in the form of a service that can be easily used at a low cost.

Accordingly, the technical problem of the present invention is conceived in this respect, and an object of the present invention is to provide a terminal supporting 3D printing that a user can easily and conveniently use.

Another object of the present invention is to provide a method for supporting 3D printing that a user can use easily and conveniently.

Another object of the present invention is to provide a computer program stored in a recording medium in order to perform the method of supporting 3D printing in combination with hardware.

A terminal supporting 3D printing according to an embodiment for realizing the object of the present invention includes: a product recognition unit for recognizing a product; A web resource utilization unit that searches and provides 3D models existing on the web related to the recognized product; A suitability calculation unit that calculates and simulates a suitability between the recognized product and the searched 3D model based on the information of the recognized product and the searched 3D model; A transmission unit for transmitting the 3D model information to a 3D printer to produce the 3D model; And a display unit displaying the recognized product, the searched 3D model, and the simulation result.

In an embodiment of the present invention, the terminal supporting 3D printing may further include an augmented reality visualization unit that displays the recognized product and the searched 3D model in augmented reality using a depth camera.

In an embodiment of the present invention, the depth camera may be formed integrally with the terminal or may be formed separately from the terminal.

In an embodiment of the present invention, the augmented reality visualization unit includes: a download unit for downloading the searched 3D model; A 3D viewer for analyzing the searched 3D model in 3D; And an augmented reality controller that provides the 3D model together with the searched product as augmented reality.

In an embodiment of the present invention, the augmented reality control unit includes: a preprocessor for removing noise from a color-depth image provided by the depth camera; A product detection unit extracting a foreground from the color-depth image; And a pose analysis unit that estimates a pose of the product and the 3D model.

In an embodiment of the present invention, the product recognition unit may recognize a barcode of the product using the camera of the terminal or recognize a product from an image of the product.

In an embodiment of the present invention, the transmission unit may transmit the 3D model information to a local 3D printer or to a cloud 3D printing agency.

In an embodiment of the present invention, the display unit may provide a user interface for supporting 3D printing.

A method of supporting 3D printing using a terminal according to an embodiment for realizing another object of the present invention described above includes the steps of recognizing a product; Searching and providing a 3D model existing on the web related to the recognized product; Calculating and simulating a suitability between the recognized product and the searched 3D model based on the information of the recognized product and the searched 3D model; And transmitting the 3D model information to a 3D printer to produce the 3D model.

In an embodiment of the present invention, the method of supporting 3D printing using the terminal may further include displaying the recognized product and the searched 3D model in augmented reality.

In an embodiment of the present invention, displaying the recognized product and the searched 3D model in augmented reality may include downloading the searched 3D model; Analyzing the searched 3D model in three dimensions; And providing the 3D model together with the searched product in augmented reality.

In an embodiment of the present invention, the providing of the 3D model together with the searched product as augmented reality may include removing noise from a color-depth image provided by a depth camera; Extracting a foreground from the color-depth image; And estimating a pose of the product and the 3D model.

In an embodiment of the present invention, in the step of recognizing the product, the barcode of the product may be recognized using the camera of the terminal, or the product may be recognized from an image of the product.

In an embodiment of the present invention, the step of transmitting the 3D model information to a 3D printer may transmit the 3D model information to a local 3D printer or to a cloud 3D printing agency.

In an embodiment of the present invention, the method of supporting 3D printing using the terminal may further include providing an interface for supporting 3D printing.

A computer program stored in a recording medium according to an embodiment for realizing another object of the present invention may perform a method of supporting 3D printing by using a terminal.

According to the terminal and method that supports such 3D printing, and a computer program for performing the method, by presenting a methodology that can solve the challenges in the DIY market, it presents the possibility of realizing DIY using a 3D printer. We present product recognition technology using camera images, parts recognition technology using depth cameras, and 6 degree of freedom tracking technology that can augment 3D printable models in real time.

Through this, integrated DIY maintenance that connects product recognition, augmented reality, and 3D printing is possible, and it can be applied to a product family used by consumers in daily life, and the value of sharing open source hardware knowledge and augmented reality and 3D printing technology It is practical and can be commercialized through synergy.

In addition, the present invention can serve as a platform for discovering additional services, such as extending the product life cycle by replacing the gap in the after-sales service, as well as sharing functions among other manufacturers.

1 is a block diagram of a terminal supporting 3D printing according to an embodiment of the present invention.
2 is a block diagram of a terminal supporting 3D printing according to another embodiment of the present invention.
3 is a detailed block diagram of the augmented reality visualization unit of FIG. 2.
4 is a detailed block diagram of the augmented reality control unit of FIG. 3.
5 is an example of a depth image and images of a 3D model according to the augmented reality controller of FIG. 3.
6 is a block diagram showing an embodiment of a system supporting 3D printing of the present invention.
7 is a flowchart illustrating a method of supporting 3D printing using a terminal according to an embodiment of the present invention.
8 is a flowchart illustrating in detail the augmented reality display step of FIG. 7.
9 is a sequence diagram illustrating a menu configuration of a user interface provided to implement a method for supporting 3D printing according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It is to be understood that the various embodiments of the present invention are different from each other but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scopes equivalent to those claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions over several aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a terminal supporting 3D printing according to an embodiment of the present invention. 2 is a block diagram of a terminal supporting 3D printing according to another embodiment of the present invention.

The terminals (10, hereinafter referred to as terminals) supporting 3D printing are various types of mobile devices such as smartphones, mobile phones, tablet computers, netbooks, PDAs, PMPs, and wearable computers. Can include all. In addition, the terminal 10 includes a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), a user terminal (UT), and a subscriber (SS). station), a wireless device, a personal digital assistant (PDA), a wireless modem, a handheld device, and a wearable device.

In addition, the terminal 10 may execute various application programs based on an operating system (OS). The operating system is a system program for allowing an application program to use the hardware of a computer, and includes all operating systems that can be mounted on mobile terminals such as iOS, Android OS, Windows Mobile OS, Bada OS, Symbian OS, and Blackberry OS. I can.

The application program is a program developed to perform a specific task using a computer, as well as various applications, various multimedia contents such as games, videos, and photos, or an image viewer that executes the multimedia contents, a video player, etc. It can contain all of the executable programs.

However, differently, the terminal 10 may be in the form of a fixed desktop computer, a monitor, a smart TV, a server, or an engine.

Referring to FIG. 1, the terminal 10 according to the present invention automatically recognizes a target product, searches for and recommends a printable model, evaluates suitability, and simulates with augmented reality, enabling 3D printing orders with one click. Do. To this end, the terminal 10 includes a product recognition unit 110, a web resource utilization unit 130, a suitability calculation unit 150, a transmission unit 190, and a display unit 200. As another embodiment of the present invention, referring to FIG. 2, an augmented reality visualization unit 170 may be further included.

Software (application) supporting 3D printing may be installed and executed in the terminal 10 of the present invention, and the configuration of the product recognition unit 110 and the like is configured to support the 3D printing executed in the terminal 10. It can be controlled by software. In addition, the configuration of the product recognition unit 110 and the like may be formed by being integrated into one module, or differently, may be formed as separate modules.

The display unit 200 displays a series of processes in which 3D printing software is performed in the terminal 10, and provides a user interface including an input menu, a selection menu, and an additional menu related thereto.

The display unit 200 includes all of a liquid crystal display (LCD) panel, a plasma display panel (PDP), an organic light-emitting diode (OLED) display panel, a glasses-type display device, etc. can do. In addition, it may be a mobile display device such as a mobile pad, a smart phone, or a wearable computer.

In an embodiment, a user interface that enables a user's input may be provided on the display unit 200 including a touch screen function. Accordingly, the user may directly input on the image displayed on the display unit 200 using a hand or an electronic pen. In addition, various input means such as a keypad, a keyboard, a mouse, a joystick, a touch pad, and a button may be implemented on the display unit 200.

The product recognition unit 110 recognizes a product requested by a user. The product recognition unit 110 may recognize a product in various ways by using a function included in the terminal 10.

For example, the product recognition unit 110 may recognize a product through a barcode attached to the product using a camera included in the terminal 10. Most of the products currently in circulation can recognize product information through the barcode system used in the industry.

The product recognition unit 110 may recognize a barcode from the camera of the terminal 10 and obtain an accurate product name of a corresponding product by interlocking the recognized barcode with a Product DB server. Product names for barcodes can be constructed by mashing up vast amounts of product information through web APIs such as Google API, Amazon API, and SearchUPC.

As another example, a product name may be directly acquired from an image by a camera. Alternatively, the user may directly input a product's unique number or product name into the terminal 10 to allow the product recognition unit 110 to recognize the product. As another example, the product recognition unit 110 may recognize product information by directly recognizing image or shape information of a product from the camera of the terminal 10.

In addition, the product recognition unit 110 may search for printable 3D models of the recognized product by using the product name in the product DB as a keyword, and display an item list on the display unit 200. Through this, the user can easily check the search history and search results.

The web resource utilization unit 130 searches for and provides a 3D model existing on the web related to the recognized product. In other words, by using the product name acquired by the product recognition unit 110, models capable of 3D printing related to the product are collected from web resources.

The web resource utilization unit 130 may provide a proxy server to provide additional services to users while utilizing 3D model resources existing on the web. For example, through a warehouse search of 3D printable models such as Thingiverse and Yeggi, it is possible to provide information on 3D models related to a product requested by a user among about 120,000 models that currently exist.

In this case, the user's waiting time may be reduced by storing the result of the search word searched by the user in the proxy server, and caching the image and the STL file included in the search result in advance. In addition, the web resource utilization unit 130 provides only necessary information briefly to reduce overload.

In addition, the web resource utilization unit 130 may provide recognition information of a product existing on the web. In other words, the product recognition unit 110 may collect and provide product-specific information necessary for recognizing a product from information existing on the web.

The suitability calculation unit 150 calculates and simulates the suitability between the recognized product and the searched 3D model based on the information of the recognized product and the searched 3D model. Accordingly, the user may be provided with an additional service that determines whether or not the 3D model to be printed has compatibility with the product to be DIY.

The suitability between the product and the 3D model consists of determining whether assembly is possible based on the shape information of the product and the 3D model. The suitability determination result is transmitted to the proxy server, and the compatibility of the 3D model to be printed can be returned to the user in three states, for example, good, normal, and bad. The proxy server may sort the returned compatibility determination result according to the degree of superiority, and return the result according to the recognized product in the form of JSON (JavaScript Object Notation), a lightweight data exchange format shown in Table 1 below.

Field name Field description Title title Desc detailed description Thumb Thumbnail image Like Other users have preferred the 3D model
Count Collect Other uploaded users
3D model collection Fit Conformity evaluation diagram Target Information about the target URL URL to receive 3D model information Feasibility Whether the uploaded 3D model file can be executed ID ID of the uploaded file Image Uploaded photo File_name Uploaded 3D model file name Model Uploaded 3D model file Filesize Uploaded 3D model file
size

The augmented reality visualization unit 170 displays a product recognized by the product recognition unit 110 and a 3D model searched by the web resource utilization unit 130 as augmented reality using a depth camera. The augmented reality visualization unit 170 visualizes a user's product and a 3D printable model together using augmented reality to help select a final model from the 3D model selected for the product. Accordingly, the user can efficiently acquire information on the selected 3D model.

To this end, referring to FIG. 3, a download unit 171 for downloading the searched 3D model, a 3D viewer unit 173 for analyzing the searched 3D model in three dimensions, and augmenting the 3D model together with the searched product It includes an augmented reality control unit 175 that provides reality.

The download unit 171 may load a 3D model file including an STL file, which is a 3D printer standard file format, in real time through a web URL. STL file loader supports both binary method and ASCII method, and can render in 3D using loaded file information.

The 3D viewer unit 173 analyzes a 3D image capable of simultaneously utilizing color and depth using a depth camera. The augmented reality control unit 175 provides information on a suitability with a product and an application method by showing the 3D model to be printed together with the product in augmented reality to the user. This makes it possible to select a final 3D model by considering predictable problems that may arise from the 3D printing result in advance.

The 3D viewer unit 173 and the augmented reality controller 175 use a color-depth (RGB-D) image of a depth camera. The 3D viewer unit 173 is a color-depth (RGB-D) mounted on the terminal 10 to augment a 3D model to be printed with the original product through the augmented reality control unit 175 to augment the product. You can use a camera. The depth camera functions to acquire 3D geometric information of a product through infrared rays.

Such a depth camera may be formed integrally with the terminal 10 or may be formed separately from the terminal 10. When the depth camera is formed as a separate module or device from the terminal 10, an image can be transmitted and received through wired or wireless communication between both.

In contrast, the color-depth camera is independently mounted on the terminal 10 and can be used conveniently without a separate connection. The present invention based on such a color-depth camera device has the following advantages compared to a conventional camera-based system.

First, in conventional computer vision, three-dimensional information estimation has to go through a complex series of processes (feature extraction and matching, rigid body/non-rigid body conversion) from one or more two-dimensional color images, but when using a depth camera as in the present invention, It is easy for 3D analysis because 3D information is acquired in real time without separate processing.

Second, since the present invention uses a camera mounted on a mobile terminal, there is no restriction in mobility unlike a conventional desktop camera-based system. In addition, since input/output processing for input data is independently operated in the mobile device, the result of 3D data analysis can be checked in real time without any separate processing.

Third, most of the conventional 3D scan and 3D model reconstruction cases applied to color-depth images are 3D CAD modeling such as surface scanning of a subject using high-level hardware resources. The present invention is a new type of production model that enables a 3D model to be checked before 3D printing through an augmented reality technology reflecting color-depth.

4 and 5, the augmented reality control unit 175 augments a product and a 3D model appropriate thereto and provides a pre-processing unit 1751, a product detection unit 1753, and a pose analysis unit 1755 to the user in real time. ).

The preprocessor 1751 removes noise from a color-depth image provided by the depth camera. Preprocessing is performed to obtain improved results in the subsequent product detection and pose analysis steps through noise removal and image improvement for the input color-depth image. To this end, improved data may be generated through smoothing and hole filling operations.

5(a) is an example of a depth image of a product, and FIG. 5(b) is an example of a color image of a product. 5(c) shows the depth image preprocessed from FIG. 5(a).

The product detection unit 1753 extracts a foreground from the color-depth image. Using the acquired color-depth image, a 3D area for a product is segmented and acquired. For this, geometric information including the curvature of the product surface to be detected and a product located in a short distance from the camera are detected. The color information is simultaneously used for segmentation of the depth image to accurately detect a 3D product area. 5(d) shows the product detection result.

The pose analysis unit 1755 estimates a pose of the product and the 3D model. A pose of a product is estimated from a three-dimensional product area defined by the product detection unit 1753 and a pose of a 3D model is defined based on this. The product pose is defined as a relative coordinate system calculated through a three-dimensional product area analysis based on the camera coordinate system. The product coordinate system moved to the combination part of the predefined product and model is defined as a pose of the 3D model, and this is used to augment the 3D model of the augmented reality control unit 175.

5(e) is an example of a product pose estimation result, and FIG. 5(f) is an example of a pose estimation result of a 3D model.

The transmission unit 190 transmits the 3D model information to a 3D printer to produce the 3D model. For example, the transmission unit 190 may transmit the 3D model information to a cloud 3D printing agency. When a user selects a cloud 3D printing agency service through the transmission unit 190, it may be connected to a cloud 3D printing agency service such as i.materialise, sculpteo, shapeways, and 3D HUBS.

Accordingly, according to the present invention, a model capable of 3D printing can be delivered through a simple connection with a 3D printing agency service, and a user can be provided with a printed 3D model.

As another example, the transmission unit 190 may transmit the 3D model information to a local 3D printer. When the user selects a local 3D printer, the model is produced by sending the 3D model and necessary information to the selected printer. The connection between the present invention and a local 3D printer can be conveniently made anywhere, such as cloud printing, using wireless communication.

6 is an example of a configuration diagram of a system 1 supporting 3D printing according to the present invention.

Referring to FIG. 6, as three layers of the system 1 configuration diagram, a web-based augmented reality platform 300, a block 100 showing processes performed at run time of the method for supporting 3D printing of the present invention, and Represents a block 500 present on the web.

The method of supporting 3D printing of the present invention can be implemented in the form of an augmented reality web application (AR Web App) of an augmented reality content authoring platform based on HTML5. Since the HTML5-based augmented reality platform was developed with Web-based technologies such as HTML, CSS, and JavaScript, it has the advantages of simplicity, modularity, and decentralization, which are the characteristics of the web, and at the same time, it is possible to reuse various media resources with hyperlinks and URIs. .

The present invention increases the accuracy of product recognition by adding a precise 6-degree-of-freedom tracking function using a structure sensor by expanding the tracking engine compared to the conventional invention, and by using the Situated Renderer's WebGL renderer to show the user as an augmented reality model It helps users understand the authoring intention and application method.

7 is a flowchart illustrating a method of supporting 3D printing using a terminal according to an embodiment of the present invention. 8 is a flowchart illustrating in detail the augmented reality display step of FIG. 7.

The method of supporting 3D printing using the terminal according to the present embodiment may be performed in substantially the same configuration as the terminal 10 of FIG. 1 or 2. Accordingly, the same components as those of the terminal 10 of FIG. 1 or 2 are given the same reference numerals, and repeated descriptions are omitted.

Alternatively, the method of supporting 3D printing using a terminal according to the present embodiment may be executed by software (application) for supporting 3D printing using a terminal.

Referring to FIG. 7, in the method of supporting 3D printing using a terminal according to the present embodiment, a product is recognized according to a user's request (step S10). Recognition of the product may be performed through a barcode displayed on the product using a camera included in the terminal or connected to the terminal. Also, the recognition of the product may be made from an image of the product using a camera included in the terminal or connected to the terminal.

When a product is recognized, a 3D model existing on the web related to the recognized product is searched and provided (step S30). In this case, a proxy server may be provided to provide additional services to users while utilizing 3D model resources existing on the web.

When a product is searched, a suitability between the recognized product and the searched 3D model is calculated and simulated based on the information of the recognized product and the searched 3D model (step S50). Accordingly, the user may be provided with an additional service that determines whether or not the 3D model to be printed has compatibility with the product to be DIY.

In addition, by displaying the recognized product and the searched 3D model in augmented reality (step S70), it is possible to help the user to select a final model from the 3D model selected for the product.

Referring to FIG. 8, the step of displaying the recognized product and the searched 3D model in augmented reality (step S70) includes downloading the searched 3D model (step S71), and analyzing the searched 3D model in three dimensions. And providing the 3D model in augmented reality together with the searched product (step S75).

Providing the 3D model together with the searched product as augmented reality (step S75) includes removing noise from a color-depth image provided by a depth camera, extracting a foreground from the color-depth image, and the product And estimating a pose of the 3D model.

In the simulation step (step S50) or the augmented reality display step (step S70), the user can select a 3D model to be printed, and transmits the 3D model information to a 3D printer to produce the 3D model (step S90). .

At this time, depending on the user's selection, the 3D model information may be transmitted to a local 3D printer or to a cloud 3D printing agency.

The method for supporting 3D printing according to the present invention is driven in a terminal, especially in a mobile environment, so that a user can easily search for a desired product, as well as accurately recognize and track it. In addition, it is possible to increase the usability and functionality of 3D printing by proceeding from product recognition to the process of transmitting the selected 3D model to a 3D printer or agency.

A method of supporting such 3D printing may be implemented by software (application).

9 is a sequence diagram illustrating an embodiment of a menu configuration of a user interface provided to implement a method for supporting 3D printing according to the present invention. For example, when the intake port of the company A's vacuum cleaner is damaged, and the inlet port of the company B's vacuum cleaner is to be compatible with the body of the company A's cleaner, a connector for compatibility between the two components is 3D printed.

Referring to FIG. 9, a product is searched through a barcode (step S100). As the first step of running an application program (application) that supports 3D printing, it recognizes barcodes to search for 3D models related to products of interest to the user. An auxiliary line or guide can be used to guide the user to easily scan the product by bringing the barcode to the proper location. In addition, the sound effect when recognizing the barcode provides the user with an experience similar to that of using an actual barcode reader, so that even a beginner can easily access and search for a product.

In this case, a search history menu may be provided (step S200). The history menu on the right side of the display unit 200 of the terminal 10 may provide information on barcodes and product names searched by a user in the past. Since the information access path is shortened through such historical information, the user can save the trouble of recognizing barcodes for previously searched products.

When the barcode is searched, the barcode search result is displayed (step S300). When a product name that matches the barcode is searched, the searched product is displayed on the upper part, and at the same time, you can select whether to search for 3D-printable models or other products for that product. If you press the button to search for models capable of 3D printing, you can show through the progress bar that the process of searching for resources existing on the web is currently in progress.

Models capable of 3D printing for the product are searched, and the result is displayed (step S400). When receiving information from the server is completed, the item menu is activated and the search results can be displayed in a list. If you select one item from the list, detailed information about it can be provided.

In addition, it is possible to provide a service for determining the suitability of the 3D model and the augmented object. Each item in the list provides information on the model name, popularity (e.g., heart mark), and fit, and the determined fit is information on how accurately it will fit the target object when the 3D model is printed. . Through this information, the user can receive the 3D model information determined to be suitable, and it is possible to reduce the 3D printing cost by preventing incorrect 3D model printing in advance.

Then, detailed information of the selected 3D model is provided (step S500). This is a step of providing detailed information of a 3D model through crowd sourcing. In the detailed information, information on other 3D models existing in the same item can be checked in addition to the representative 3D shown on the list. In addition, it is possible to provide ratings of other users for the selected item together.

In this case, information on the 3D model can be provided in various ways. For example, a user who wants to view the selected 3D model while rotating 360 degrees instead of in an augmented state can view the 3D model through touch interaction after pressing the 3D View button. In addition, by pressing the AR View button, you can augment and show a desired 3D model among several 3D models that exist in the item. In addition, for users who want to know the source of detailed product information, a button to move to the web page where the 3D model is imported can be provided.

Further, it is possible to check a realistic 3D model using augmented reality (step S600). On the screen showing the augmented 3D model on the product, a button can be provided on the upper left to go to the screen of the first stage of execution of an application program (application) that supports 3D printing to search for barcodes of other products. In addition, in the upper middle part, the name of the 3D model that is currently being augmented and the fitness information with the target are expressed step by step so that it can be recognized at a glance, and a button for printing the 3D model currently being viewed is provided in the lower left. I can.

Finally, a menu for printing the 3D model finally checked using augmented reality is provided (step S700). The photo information and name of the 3D model to be printed can be notified to the user once again on the left, and the 3D printers that can be connected and the service companies that can request printing can be selected on the right. It also provides an edit button to allow you to edit settings for 3D printing or service companies.

However, since the user interface provided to implement the method for supporting 3D printing presented in FIG. 9 and the description thereof is only an example, it is not limited thereto and may be configured and changed as necessary.

Such a method of supporting 3D printing in a terminal may be implemented as an application or implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention, and may be known and usable to those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although the above has been described with reference to embodiments, it is understood that those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the following claims. You can understand.

Product recognition and the search and enhancement of compatible 3D printable models provided by the present invention are not limited to a specific product group and can provide a variety of useful values to consumers. From the consumer's perspective, it can be used not only to help DIY for home appliances, but also to DIY for various fields such as automobiles and leisure by using printable 3D models and fitness information in various fields existing in the 3D printable warehouse.

On the one-man enterprise side, 3D model developers can upload their own 3D models as paid models to a 3D printable warehouse. By providing custom parts for specific purposes that are difficult to mass-produce and models that are produced in consideration of the characteristics of individual consumers, a business model for multi-species small quantity production is established.

From the point of view of the industry, it can help to reduce the cost of maintenance of facilities, and provides a visualized manual for problem solving by utilizing parts recognition and augmented reality technology in situations where maintenance is required. Therefore, even an unskilled user can easily maintain the product.

In addition, product information DB and suitability information for various fields such as household appliances, living, home decor, office, automobile, leisure, fashion, etc. are provided in the form of an in-app purchase to users who want to provide service and DIY for each field Real-time assembly inspection function can be provided to customers through in-app purchase, and paid 3D model developers sell the created 3D models in a 3D printable warehouse, allowing users to easily purchase in-app purchases. Service can be created.

10: terminal 110: product recognition unit
130: web resource utilization unit 150: fitness calculation unit
170: augmented reality visualization unit 171: download unit
173: 3D viewer unit 175: augmented reality control unit
1751: preprocessing unit 1753: product detection unit
1755: pose analysis unit 190: transmission unit
200: display unit 1: system supporting 3D printing

Claims (16)

A camera that acquires image information related to an object;
A product recognition unit that recognizes an object based on the image information and recognizes an already existing product matching the recognized object;
A web resource utilization unit that searches and provides 3D models related to the recognized product on the web;
A fitness calculation unit that calculates and simulates suitability for assembling between the recognized product and the searched 3D model based on the recognized product and shape information of the searched 3D models.- The 3D models used for calculating the suitability are Includes 3D models of assemblyable parts in recognized products;
A display unit for displaying the recognized product, the 3D model of the searched part, and the simulation result based on the calculated suitability so that the user can select the displayed 3D model;
By moving the coordinate system of the recognized product to the coupling part of the recognized shape of the product and the 3D model of the selected part, the pose of the 3D model of the selected part is above the recognized product so that the pose of the recognized product is tracked. An augmented reality visualization unit that provides a 3D model of the selected part as augmented reality; And
Including a transmission unit for transmitting the selected 3D model information to a 3D printer to produce a 3D model of the selected part,
A terminal supporting 3D printing, characterized in that the shape of the recognized product to which the 3D model of the selected part is combined for the augmented reality is based on 3D geometric information of the recognized product acquired from a color-depth image .
The method of claim 1,
The augmented reality visualization unit,
A download unit for downloading a 3D model of the selected part;
A 3D viewer for analyzing the 3D model of the selected part in 3D using a depth camera; And
A terminal supporting 3D printing comprising an augmented reality control unit that provides a 3D model of the selected part as an augmented reality together with the recognized product.
The method of claim 2,
The depth camera,
A terminal that is integrated into the terminal or formed separately from the terminal, supporting 3D printing.
delete The method of claim 2,
The augmented reality control unit,
A preprocessor for removing noise from a color-depth image provided by the depth camera;
A product detection unit extracting a foreground from the color-depth image; And
A terminal supporting 3D printing, comprising a pose analysis unit for estimating a pose of the recognized product and a 3D model of the selected part.
The method of claim 1,
The transmission unit,
A terminal supporting 3D printing that transmits the 3D model information of the part to a local 3D printer or to a cloud 3D printing agency.
The method of claim 1,
The display unit,
A terminal that supports 3D printing, providing a user interface to support 3D printing.
Recognizing an already existing product matching the object based on image information related to the object;
Searching and providing 3D models existing on the web related to the recognized product;
Calculating and simulating suitability for assembling between the recognized product and the searched 3D model based on the recognized product and shape information of the searched 3D model-The 3D models used for calculating the suitability are the recognized Includes 3D models of parts that can be assembled into the product;
Displaying the recognized product, the 3D model of the searched part, and the simulation result based on the calculated suitability so that the user can select the displayed 3D model;
By moving the coordinate system of the recognized product to the coupling part of the recognized shape of the product and the 3D model of the selected part, the pose of the 3D model of the selected part is above the recognized product so that the pose of the recognized product is tracked. Providing a 3D model of the selected part as augmented reality; And
Transmitting the selected 3D model information to a 3D printer to produce a 3D model of the selected part,
3D printing using a terminal, characterized in that the shape of the recognized product to which the 3D model of the selected part is combined for the augmented reality is based on 3D geometric information of the recognized product acquired from a color-depth image How to support it.
The method of claim 8,
Providing the 3D model of the selected part on the recognized product as augmented reality,
Downloading a 3D model of the selected part from the web to the terminal;
Analyzing the 3D model of the selected part in 3D using the color-depth image; And
A method for supporting 3D printing using a terminal comprising the step of providing a 3D model of the selected part together with the recognized product as augmented reality.
The method of claim 9,
Providing the 3D model of the selected part together with the recognized product in augmented reality,
Removing noise from the color-depth image provided by the depth camera;
Extracting a foreground from the color-depth image; And
And estimating a pose of the recognized product and the 3D model of the selected part.
The method of claim 8,
Recognizing the product,
Recognizing a barcode of the object using a camera of the terminal, or recognizing a product from an image of the object.
The method of claim 8,
Transmitting the 3D model information to a 3D printer,
A method of supporting 3D printing using a terminal, comprising transmitting the 3D model information of the part to a local 3D printer or to a cloud 3D printing agency.
The method of claim 8,
A method for supporting 3D printing using a terminal, further comprising the step of providing an interface for supporting 3D printing.
A computer program stored in a recording medium to perform the method of supporting the 3D printing by being combined with hardware according to any one of claims 8 to 13. delete delete

Images