KR102616306B1 - Method for controlling 3d printer and system therefor - Google Patents

Abstract

A 3D printer control method and system are provided. A 3D printer control method according to some embodiments includes obtaining design data for each layer of a product, receiving a captured image of the current layer of a product output from a laminated 3D printer in real time, and designing each layer. Comparing the design data of the current layer obtained from the data and the captured image, performing a check for an output error in the current layer, and stopping the output of the 3D printer based on the result of the check that an output error exists in the current layer. may include. According to this method, problems such as unnecessary material consumption and process delays caused by delays in determining defective products can be minimized.

Description

3D printer control method and system {METHOD FOR CONTROLLING 3D PRINTER AND SYSTEM THEREFOR}

The present invention relates to a method and system for controlling a 3D printer, and more specifically, to a method and system for monitoring and controlling the output process of a 3D printer in real time.

A stacked 3D printer refers to a printer that prints products by stacking very thin layers one by one. The product manufacturing process based on these 3D printers includes a modeling process to design the product, an output (printing) process to create a product by laminating materials (e.g., filament, resin, etc.), and a post-processing process to dry, polish, or dye the printed product. It can be divided into

Meanwhile, when mass producing products using a 3D printer, quality evaluation based on a 3D scanner is usually performed only on the finished product. In other words, even when an error occurs in the output process, a defective product is determined after the product is completed. In this case, the delay in determining a defective product causes various inefficiency problems such as unnecessary material consumption and process delays. Therefore, technology that can detect output errors in real time during the printing process is required.

Korean Patent Publication No. 10-2023-0038839 (published on March 21, 2023)

The technical problem to be solved through some embodiments of the present disclosure is to provide a method and system for detecting output errors in real time during the output process of a 3D printer.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a method and system that can accurately detect various output errors of a 3D printer.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

In order to solve the above technical problem, a 3D printer control method according to some embodiments of the present disclosure is a method performed by at least one computing device, comprising the steps of acquiring design data for each layer of the product, Receiving in real time a captured image of the current layer of the product output from a 3D printer, comparing the design data of the current layer obtained from the design data for each layer with the captured image, thereby detecting an output error of the current layer. It may include performing an inspection and stopping output of the 3D printer based on a result of the inspection indicating that an output error exists in the current layer.

In some embodiments, performing the checking for an output error of the current layer may include further comparing design data of the current layer and a depth map of the current layer.

In some embodiments, the captured image includes a first captured image received at a first time point and a second captured image received at a second time point, and performing a check for an output error of the current layer includes: Comparing each of the first captured image and the second captured image with design data of the current layer, wherein the interval between the first viewpoint and the second viewpoint may be determined based on the complexity of the current layer. there is.

In some embodiments, performing a check for an output error of the current layer includes calculating a complexity of the current layer, adjusting a tolerance to be applied to the current layer based on the complexity, and adjusting the tolerance to be applied to the current layer. It may include performing a check for the output error based on the established tolerance.

In some embodiments, calculating a difference between the current layer and the previous layer based on the design data for each layer and checking output errors regarding missing layers based on a determination that the calculated difference is greater than or equal to a reference value. Additional steps may be included.

In some embodiments, calculating a difference between the current layer and the previous layer based on the design data for each layer and checking an output error regarding a layer shift based on a determination that the calculated difference is less than a reference value. Additional steps may be included.

In order to solve the above-described technical problem, a 3D printer control system according to some embodiments of the present disclosure includes one or more processors and a memory that stores a computer program executed by the one or more processors, and the computer program includes An operation of acquiring design data for each layer of a product, an operation of receiving a captured image of the current layer of the product output from a lamination type 3D printer in real time, design data of the current layer obtained from the design data of each layer, and Instructions for performing a check for an output error in the current layer by comparing the captured image and an operation for stopping the output of the 3D printer based on the result of the check that an output error exists in the current layer. It can be included.

In order to solve the above-mentioned technical problem, a computer program according to some embodiments of the present disclosure is combined with a computing device to acquire design data for each layer for a product, the product output from a 3D printer in a layered manner. receiving a captured image for the current layer in real time, performing a check for output errors of the current layer by comparing the captured image with design data of the current layer obtained from the design data for each layer, and It may be stored in a computer-readable recording medium to execute the step of stopping the output of the 3D printer based on the result of the inspection that an output error exists in the current layer.

According to some embodiments of the present disclosure, output errors of the 3D printer may be detected in real time by comparing the captured image of the current layer and design data. And, in response to detecting an output error, the output (operation) of the 3D printer may be immediately stopped. In this case, problems such as unnecessary material consumption and process delays caused by delays in determining defective products can be minimized.

Additionally, output error checking may be performed in a differential manner depending on the complexity of the layer. In this case, output error checking can be performed more intensively in complex layers, and as a result, both performance efficiency and accuracy of output error checking can be improved.

Additionally, an output error check regarding layer shift or layer missing may be selectively performed by considering the difference between the current layer and the previous layer. In this case, there is no need to check for two output errors in all layers, so the performance efficiency of output error checking can be further improved. In addition, by performing inspection for various output errors, all output errors present in the product can be detected.

Additionally, by training a deep learning model using a number of tasks related to depth map prediction, the depth map prediction performance of the deep learning model can be greatly improved. Additionally, by using this deep learning model, an accurate and sophisticated depth map can be obtained even when there is no depth camera module.

The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below.

1 and 2 are exemplary configuration diagrams for schematically explaining a 3D printer control system according to some embodiments of the present disclosure.
Figure 3 is an example flowchart showing a 3D printer control method according to some embodiments of the present disclosure.
FIG. 4 is an exemplary diagram illustrating a depth map-based output error checking method according to some embodiments of the present disclosure.
5 and 6 illustrate monitoring screens of a control system that may be referenced in some embodiments of the present disclosure.
Figure 7 is an example flowchart showing a 3D printer control method according to some other embodiments of the present disclosure.
FIG. 8 is an exemplary diagram for explaining a layer complexity calculation method according to some embodiments of the present disclosure.
FIG. 9 is an exemplary diagram for explaining a layer complexity calculation method according to some other embodiments of the present disclosure.
Figure 10 is an example flowchart showing a 3D printer control method according to some other embodiments of the present disclosure.
FIGS. 11 and 12 are exemplary diagrams for further explaining the detailed process of the output error checking step considering the layer difference shown in FIG. 10.
13 and 14 are exemplary diagrams for explaining the structure and learning method of a deep learning model (i.e., a deep learning model for generating a depth map) according to some embodiments of the present disclosure.
15 and 16 are exemplary diagrams for explaining a data augmentation method for learning a deep learning model according to some embodiments of the present disclosure.
17 illustrates an example computing device that can implement a control system according to some embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the technical idea of the present disclosure is not limited to the following embodiments and may be implemented in various different forms. The following examples are merely intended to complete the technical idea of the present disclosure and to be used in the technical field to which the present disclosure belongs. It is provided to fully inform those skilled in the art of the scope of the present disclosure, and the technical idea of the present disclosure is only defined by the scope of the claims.

When adding reference numerals to components in each drawing, it should be noted that identical components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined. The terminology used herein is for the purpose of describing embodiments and is not intended to limit the disclosure. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context.

Additionally, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being “connected,” “coupled,” or “connected” to another component, that component may be directly connected or connected to that other component, but there is another component between each component. It will be understood that elements may be “connected,” “combined,” or “connected.”

As used in this disclosure, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element that includes one or more other components, steps, operations and/or elements. Does not exclude presence or addition.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

1 is an exemplary configuration diagram schematically illustrating a 3D printer control system according to some embodiments of the present disclosure.

As shown in FIG. 1, the 3D printer control system according to embodiments is a system that can efficiently manufacture a desired product 14 through appropriate control of the 3D printer 11. The 3D printer control system may include a control system 10, a 3D printer 11, and a camera module 12. However, in some cases, the control system 10 and the camera module 12 may be called a '3D printer control system', and the control system 10 may be called a '3D printer control system' alone. .

The control system 10 is a computing device/system that performs real-time monitoring and control of the 3D printer 11. For example, the control system 10 receives captured images (videos) of the product 14 output from the 3D printer 11 in real time from the camera module 12 and analyzes these captured images to control the 3D printer 11. Output errors (i.e., output errors present in product 14) can be detected. Since the 3D printer 11 outputs the product 14 in a layered manner, the control system 10 can receive a captured image for the current layer in real time and detect output errors existing in the current layer in real time. In addition, the control system 10 may immediately stop output (i.e., operation) of the 3D printer 11 in response to detecting an output error. By doing so, problems such as unnecessary material exhaustion and process delays can be minimized.

The specific monitoring and control method of the control system 10 will be described in detail later with reference to the drawings of FIG. 3 and below.

The above-described control system 10 may be implemented with at least one computing device. For example, all functions of control system 10 may be implemented in a single computing device, with a first function of control system 10 implemented in a first computing device and a second function implemented in a second computing device. It could be. Alternatively, certain functions of control system 10 may be implemented in multiple computing devices.

Computing devices may include all types of devices with computing (processing) capabilities, and see FIG. 17 for an example of such a device.

For reference, a computing device is a collection of interacting various components (e.g. memory, processor, etc.), so in some cases, it may be called a ‘computing system’. Of course, the term computing system can also encompass the concept of a collection of interacting computing devices.

Next, the camera module 12 is a module that photographs the product 14 output from the 3D printer 11 in real time. For example, the camera module 12 may be placed (installed) above the product 14 and photograph the layers constituting the product 14 directly downward. Figure 1 shows an example in which one camera module 12 is installed on the top of the 3D printer 11, but the scope of the present disclosure is not limited thereto. For example, inside the 3D printer 11, there is a camera module (not shown) that photographs the layers of the product 14 from the side, a camera module (not shown) that photographs the product 14 from an angle other than directly below, etc. It may be installed. Additionally, in some cases, depth camera modules (not shown) that can generate depth maps (i.e., depth images) for the product 14 from various angles may be further installed.

The camera module 12 can transmit captured images (videos) to the control system 10 in real time. By doing so, output errors of the 3D printer 11 can be detected and controlled in real time.

Next, the 3D printer 11 is a printing equipment equipped with manufacturing capabilities for 3D products 14. As shown, the 3D printer 11 may be a printing equipment that manufactures the product 14 in a layered manner. For example, the 3D printer 11 may be a printing device that operates in a Stereo Lithography Apparatus (SLA) method or a Fused Deposition Modeling (FDM) method, but the scope of the present disclosure is not limited thereto.

For reference, FIG. 1 shows an example in which the 3D printer 11 manufactures the product 14 by stacking layers on the bed 13.

In some embodiments, as shown in FIG. 2, the manufacturing system 10 and the 3D printer 11 may be manufactured in the form of an integrated mass production chamber. For example, an integrated mass production chamber may be manufactured in a form in which the output process of the 3D printer 11 can be visually confirmed on the screen (display) of the control system 10 (e.g., an output error in the current layer 22 of the product may occur). existence can be confirmed with the naked eye and on the screen), but the scope of the present disclosure is not limited thereto.

So far, a 3D printer control system according to some embodiments of the present disclosure has been schematically described with reference to FIGS. 1 and 2 . Below, various methods that can be performed in the control system 10 will be described in detail.

Hereinafter, in order to provide convenience of understanding, the description will be continued assuming that all steps/operations of the methods to be described later are performed in the control system 10 in the environment illustrated in FIGS. 1 and/or 2. Accordingly, when the subject of a specific step/action is omitted, it can be understood as being performed in the control system 10. However, in a real environment, some steps/operations of the methods to be described later may be performed on other computing devices.

In addition, hereinafter, when the drawings are not directly referred to, reference numbers may be omitted, and for clarity of the present disclosure, reference numbers for the same components may be changed depending on the embodiment.

Figure 3 is an example flowchart showing a 3D printer control method according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 3, these embodiments may begin at step S31 of acquiring design data for each layer for the product.

Design data by layer is data that can be classified by layer and can include various data/information that can be obtained during the design (modeling) stage (process) of the product. For example, design data by layer is a G-code file obtained through a slicer program, 3D design data of the product (e.g., design image (blueprint), design dimensions, etc.) analyzed by layer (e.g., by layer). It may include, but is not limited to, design images (blueprints), design dimension data for each layer, etc.). If you are working in the relevant technical field, you will already be familiar with ‘G code’, so explanation of it will be omitted.

In step S32, a captured image of the current layer of the product output from the 3D printer may be received in real time. For example, the control system 10 may receive an image taken directly downward of the current layer of the product from the camera module 12 disposed above the product. Please refer to FIGS. 5 and 6 for examples of images captured in this way.

In step S33, the design data of the current layer and the captured image may be compared. In other words, output error checking can be performed by comparing the design data of the current layer obtained from the design data for each layer and the captured image. For example, the control system 10 may compare the design image of the current layer with the captured image or compare the design dimensions of the current layer with manufacturing dimensions obtained from the captured image. Any specific comparison method may be used.

Meanwhile, the timing and/or cycle for performing the output error check can be set in various ways. For example, the control system 10 may perform an output error check (see S33) for the current layer at the time the output of the current layer is completed, or repeatedly check the output error according to a preset cycle while the current layer is being output. (Refer to S33) can also be performed. At this time, the preset period may be dynamically determined based on the complexity of the current layer. For this, please refer to the descriptions of FIGS. 7 to 9.

In some embodiments, output error checking for the current layer may be performed based further on the depth map for the current layer. For example, as shown in FIG. 4, assume that a depth map (41, a bright color means shallow depth (close distance)) corresponding to the current layer image taken directly downward has been obtained. This depth map 41 may be generated through a deep learning model (see FIG. 13), which will be described later, or through a depth camera module installed on the top of the product. In this case, the control system 10 may check the output error of the current layer by further comparing the depth map 41 and the design data (e.g., design dimensions for depth) of the current layer (e.g., depth map 41 ), compare the depth value of the product area (42) with the corresponding design dimension (see 43) to determine if the manufacturing error exceeds tolerance). By doing so, checking for output errors can be performed more thoroughly.

In some cases, a depth map-based output error check may be performed based on the determination that the complexity of the current layer is greater than the reference value. That is, the control system 10 may additionally perform depth map-based output error checking only for complex layers. In this case, since depth map-based output error checking is selectively performed only for required layers, the performance efficiency of output error checking can be further improved.

In step S34, it may be determined whether an output error of the 3D printer exists in the current layer based on the comparison result. For example, if the manufacturing error according to the comparison result exceeds a preset tolerance, the control system 10 may determine that an output error exists (occurred) in the current layer. At this time, the tolerance may be set to the same value for each layer or may be set differently. For example, the tolerance may be set (determined) to different values based on the complexity of the layer. For this, refer to the descriptions of FIGS. 7 to 9. In some cases, the tolerance may be set (determined) to a value inversely proportional to the height of the layer (e.g., as the height of the layer is set to a smaller value, the tolerance is also set to a smaller value to ensure the precision of the printed product. to be determined). Depending on the determination result of this step, step S35 or step S36 may be performed.

In step S35, output (operation) of the 3D printer may be stopped. For example, the control system 10 may immediately transmit a control command for stopping output to the 3D printer 11 based on a determination (i.e., inspection result) that an output error exists in the current layer. By doing so, problems such as unnecessary material exhaustion and process delays can be minimized.

Additionally, as shown in FIG. 5, the control system 10 may display information about output errors on the monitoring screen 52. For example, the control system 10 may display information about the output error occurrence area 51 of the current layer, the type of output error, cause of occurrence, action, etc. on the monitoring screen 52.

For reference, Figure 6 illustrates the monitoring screen 61 when no output error occurs.

Description will be made again with reference to FIG. 3 .

In step S36, it may be determined whether the current layer is the last layer of the product. If the current layer is not the last layer, the above-described steps S32 to S34, etc. may be repeatedly performed for the next layer.

So far, a 3D printer control method according to some embodiments of the present disclosure has been described with reference to FIGS. 3 to 6. According to the above, output errors of the 3D printer can be detected in real time by comparing the captured image of the current layer with the design data. And, in response to detecting an output error, the output (operation) of the 3D printer may be immediately stopped. In this case, problems such as unnecessary material consumption and process delays caused by delays in determining defective products can be minimized.

Hereinafter, a 3D printer control method according to some other embodiments of the present disclosure will be described with reference to FIGS. 7 to 9. However, for clarity of the present disclosure, description of content that overlaps with the previous embodiments will be omitted.

Figure 7 is an example flowchart showing a 3D printer control method according to some other embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 7, these embodiments relate to a method of performing output error checking considering the layer complexity of the product. Hereinafter, the present embodiments will be described in detail.

In step S71, layer-by-layer design data for the product may be obtained. For this, please refer to the description of step S31 described above.

In step S72, a captured image of the current layer of the product output from the 3D printer may be received in real time. For this, please refer to the description of step S32 described above.

In step S73, the complexity of the current layer can be calculated. However, the specific complexity calculation method may vary depending on the embodiment.

In some embodiments, the complexity of the current layer may be calculated based on the number of G-code instructions. For example, as shown in FIG. 8, the control system 10 counts the number of G-code commands (e.g., 82 to 84, etc., if the current layer is 'layer 2') for the current layer in the G-code file 81. The complexity of the current layer can be calculated by counting and using a value proportional to the number (that is, the greater the number of G-code instructions, the higher the complexity is calculated). In some cases, the control system 10 may assign different weights depending on the type of G-code command and calculate the complexity of the current layer based on these weights. In this case, the complexity of the current layer can be calculated more accurately.

In some other embodiments, the complexity of the current layer may be calculated based on the complexity of the design image. For example, the control system 10 may calculate the complexity of the current layer based on the degree of change in pixel values of the design image of the current layer, the number of pixels with different values, etc. The complexity of the current layer can be calculated as a value proportional to the complexity of the design image.

In some other embodiments, the complexity of the current layer may be calculated based on the shape complexity of the current layer. For example, the control system 10 may calculate the complexity of the current layer as a value proportional to the number of vertices, sides, and/or faces related to the current layer, the number of supports existing in the current layer, the number of holes, etc. .

In some other embodiments, the final complexity of the current layer may be calculated through a weighted sum of the complexity of the first part belonging to the product and the complexity of the second part belonging to the support in the current layer. At this time, the weight of the first part may be set to a higher value than the weight of the second part. For example, as shown in Figure 9, let's assume that there is a first product 91 and a second product 92, and the current layer is 'layer 2'. In this case, the current layer complexity of the first product 91 may be calculated to be lower than that of the second product 92. This is because the current layer of the first product 91 contains more support parts (e.g., 93) than the second product 92 (compare 92 and 94, compare 93 and 95).

In some other embodiments, the complexity of the current layer may be calculated based on various combinations of the above-described embodiments. For example, the control system 10 aggregates the first complexity (e.g., based on the number of G-code instructions) calculated according to the first embodiment and the second complexity (e.g., based on shape complexity) calculated according to the second embodiment. (e.g., weight sum) may be used to calculate the final complexity for the current layer.

This will be described again with reference to FIG. 7 .

In step S74, an output error check for the current layer may be performed considering the complexity of the current layer. However, the specific output error checking method may vary depending on the embodiment.

In some embodiments, the performance cycle (interval) of output error checking may be determined based on the complexity of the current layer. For example, let us assume that while the current layer is being output, a first captured image is received at a first time point and a second captured image is received at a second time point after the first time point. Also, assume that the control system 10 performs an output error check by comparing the first captured image with the design data of the current layer and then comparing the second captured image with the design data of the current layer. In this case, the interval (i.e., period) between the first view and the second view may be determined based on the complexity of the current layer. For example, the higher the complexity of the current layer, the smaller the interval between two viewpoints can be determined. By doing so, output error checking can be performed more thoroughly for complex layers.

In some other embodiments, the tolerance to be applied to the current layer may be adjusted based on the complexity of the current layer, and output error checking for the current layer may be performed based on the adjusted tolerance. For example, the control system 10 may adjust the tolerance to a smaller value as the complexity of the current layer increases. By doing so, more stringent error checking criteria can be applied to complex layers. In some cases, the control system 10 may count the number of layers whose complexity is higher than the standard value among all the layers that make up the product and adjust the tolerance to a value inversely proportional to the number (e.g., the more complex layers there are, the more sophisticated the product is. , so the tolerance is adjusted to a smaller value).

In some other embodiments, a depth map-based output error check may be further performed based on a determination that the complexity of the current layer is greater than the reference value. For this, please refer to the description of FIG. 4.

In some other embodiments, output error checking may be performed based on various combinations of the above-described embodiments.

In steps S75 and S76, output (operation) of the 3D printer may be stopped based on a determination that an output error exists in the current layer. For this, please refer to the description of steps S34 and S35 described above.

In step S77, it may be determined whether the current layer is the last layer of the product. For this, please refer to the description of step S36 described above.

So far, a 3D printer control method according to several other embodiments of the present disclosure has been described with reference to FIGS. 7 to 9. According to the above, output error checking can be performed in a differential manner depending on the complexity of the layer. In this case, output error checking can be performed more intensively in complex layers, and as a result, both performance efficiency and accuracy of output error checking can be improved.

Hereinafter, a 3D printer control method according to some other embodiments of the present disclosure will be described with reference to FIGS. 10 to 12. However, for clarity of the present disclosure, description of content that overlaps with the previous embodiments will be omitted.

Figure 10 is an example flowchart showing a 3D printer control method according to some other embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 10, these embodiments relate to a method of performing output error checking based on the difference between the current layer and the previous layer. Hereinafter, the present embodiments will be described in detail.

In step S101, layer-by-layer design data for the product may be obtained. For this, please refer to the description of step S31 described above.

In step S102, a captured image of the current layer of a product output from a 3D printer may be received in real time. For this, please refer to the description of step S32 described above.

In step S103, the difference between the current layer and the previous layer may be calculated. For example, the control system 10 may calculate the difference between two layers based on differences in shape, area, complexity, number of supports, average depth (e.g., depth measured from the side), etc. between the two layers. However, the scope of the present disclosure is not limited thereto.

In step S104, output error checking may be performed considering the difference between the two layers. However, the specific output error checking method may vary depending on the embodiment.

In some embodiments, output error checking regarding layer shift may be performed based on a determination that the difference between two layers is less than a reference value. Anyone working in the relevant technical field will already be familiar with this ‘layer shift error’, so description of this will be omitted. For example, as shown in FIG. 11, the control system 10 operates when the difference between the two layers 113 and 114 constituting the product 111 is less than the reference value (if the current layer is layer 3 113). Assumption) An output error check regarding layer shift can be performed on the two layers 113 and 114. Specifically, the control system 10 acquires a side depth map (i.e., a depth map corresponding to the side captured image) for the current layer 113 and the previous layer 114, and determines the current layer 113 in the side depth map. By comparing the depth value with the depth value of the previous layer 114, it is possible to determine whether a layer shift error exists (e.g., determine whether the difference between the two depth values matches the difference in design dimensions). For reference, the side depth map may be received from a depth camera module or generated from a side captured image through a deep learning model (see FIG. 13). For this, please refer to the descriptions of FIGS. 13 to 16.

In some other embodiments, output error checking for missing layers may be performed based on a determination that the difference between the two layers is greater than or equal to a reference value. Anyone working in the relevant technical field will already be familiar with this ‘missing layer error’, so explanation of this will be omitted. For example, as shown in FIG. 11, when the difference between the two layers 112 and 113 constituting the product 111 is greater than the reference value (assuming the current layer is layer 4 (112)), the control system ( 10) may perform an output error check regarding missing layers on previous layers (e.g., 113, 114) of the current layer 112. Specifically, as shown in FIG. 12, the control system 10 receives images taken from the side of the previous layers (e.g., 113, 114) of the current layer 112 from the camera module 121, and By analyzing the image, the number of previous layers (e.g., 113, 114) can be counted. And, the control system 10 checks the number of previous layers (e.g., 113, 114) based on the design data for each layer (i.e., checks whether it matches the number of layers in the design data), thereby preventing a missing layer error. It is possible to determine whether exists.

In some other embodiments, output error checking may be performed based on various combinations of the above-described embodiments.

Description will be made again with reference to FIG. 10 .

In steps S105 and S106, output (operation) of the 3D printer may be stopped based on a determination that an output error exists. For this, please refer to the description of steps S34 and S35 described above.

In step S107, it may be determined whether the current layer is the last layer of the product. For this, please refer to the description of step S36 described above.

So far, a 3D printer control method according to several other embodiments of the present disclosure has been described with reference to FIGS. 10 to 12. According to the above, output error check regarding layer shift or missing layer can be selectively performed depending on the difference between the current layer and the previous layer. In this case, there is no need to check for two output errors in all layers, so the performance efficiency of output error checking can be further improved. In addition, by performing inspection for various output errors, all output errors present in the product can be detected.

Various embodiments of the 3D printer control method described so far with reference to FIGS. 3 to 12 may be combined in various forms. For example, the control system 10 may perform an output error check in consideration of the complexity of the current layer and, at the same time, perform an output error check for layer shift and/or layer disappearance by considering the difference between the current layer and the previous layer. .

Hereinafter, a deep learning-based depth map generation method according to some embodiments of the present disclosure will be described with reference to FIGS. 13 to 16.

As shown in FIG. 13, etc., the depth map generation method to be described later relates to a method of generating a depth map (e.g., 137) from a two-dimensional image (e.g., 136) using a deep learning model. Although learning of such a deep learning model may be performed on another computing device, for convenience of understanding, the following description will be continued assuming that learning of the deep learning model is also performed by the control system 10.

First, the structure and operating principles of deep learning models according to embodiments will be described with reference to FIG. 13.

As shown in FIG. 13, a deep learning model according to embodiments may be configured to include an encoder 131, a decoder 132, and a plurality of predictors 133 to 135.

The encoder 131 is a neural network module that extracts features (e.g., feature maps) from the input image 136. The encoder 131 may be implemented based on a neural network, such as a convolutional neural network (CNN) or a transformer, but the scope of the present disclosure is not limited thereto. The encoder 131 may be implemented in any way. In some cases, a pretrained feature extractor (e.g., a pretrained VGG series neural network, etc.) may be used as the encoder 131. Since the encoder 131 is shared by multiple predictors 133 to 135, it may be referred to as a ‘shared encoder’.

Next, the decoder 132 is a neural network module that decodes the features extracted from the encoder 131. The decoder 132 may increase the size of a feature (e.g., feature map) by performing an up-sampling operation on the input features. The decoder 131 may be implemented based on, for example, a convolutional neural network, but the scope of the present disclosure is not limited thereto. Since the decoder 132 is also shared by multiple predictors 133 to 135, it may also be referred to as a ‘shared decoder’. In some cases, the deep learning model may be configured in such a way that the function of the decoder 132 is mounted on the predictors 133 to 135 (that is, the decoder 132 may be omitted from the components of the deep learning model) has exist).

As shown, a skip-connection that transfers features may exist between the encoder 131 and the decoder 132. For example, like U-Net, the encoder 131 and the decoder 132 are each composed of encoding blocks and decoding blocks of different levels, and a skip functions as a feature transfer path between the encoding blocks and decoding blocks of the same level. -A connection may exist. However, the scope of the present disclosure is not limited thereto.

The plurality of predictors 133 to 135 are neural network modules that perform different tasks. A plurality of predictors 133 to 135 may receive decoded features and perform corresponding tasks. Since the predictors 133 to 135 are layers that perform tasks, they may be referred to as ‘task specific layers/modules’ in some cases. The predictors 133 to 135 may be implemented based on a neural network such as a fully-connected layer, but the scope of the present disclosure is not limited thereto.

The first predictor 133 may predict and generate a depth map 137 for the input image 136 based on the decoded features. That is, the first predictor 133 can perform a depth map prediction task.

Next, the second predictor 134 may predict the class label of the input image 136 based on the decoded features. That is, the second predictor 134 may perform a class prediction task (ie, a classification task).

Next, the third predictor 135 may predict and generate a segmentation map 138 for the input image 136 based on the decoded features. That is, the third predictor 135 may perform a semantic segmentation task of predicting the class of the input image 136 in pixel units. If you are working in the relevant technology field, you will already be familiar with the deep learning task called ‘semantic segmentation’, so description of this will be omitted.

As shown, each of the predictors 133 to 135 may perform prediction using data from other predictors (e.g., prediction results, data inside the predictor) (see arrows).

For reference, Figure 13 shows an example where all three predictors 133 to 135 are used for learning a deep learning model, but in some cases, the second predictor 134 and the third predictor 135 ) at least one of them may not be used for learning the deep learning model.

Below, the learning method of the deep learning model will be described with reference to FIG. 14.

As shown in FIG. 14, a deep learning model can be learned by performing a multi-task through a plurality of predictors 133 to 135. Specifically, each training sample constituting the training set may be composed of a two-dimensional image (141), a correct answer depth map (143, i.e., ground truth), a correct answer class label, and a correct answer segmentation map (145), and the control system ( 10) can learn a deep learning model using a large number of training samples constructed in this way.

More specifically, the control system 10 can input (feed) the two-dimensional image 141 to the encoder 131 and predict the depth map 142 for the input image 141 through the first predictor 133. there is. And, the control system 10 may calculate the first loss L1 based on the difference between the predicted depth map 142 and the correct answer depth map 143. Any method may be used to calculate the first loss (L1). In some cases, the control system 10 may calculate the first loss L1 by assigning higher weight to the difference (loss) in the internal area of the object than to the external area. In this case, the deep learning model can be trained to perform predictions by focusing more on the internal area of the object. This technical principle can also be applied to the third loss (L3).

Additionally, the control system 10 may input (feed) the two-dimensional image 141 to the encoder 131 and predict the class label of the input image 141 through the second predictor 134. And, the control system 10 may calculate the second loss (L2) based on the difference between the predicted class label and the correct class label. The second loss L2 may be calculated based on, for example, a cross-entropy loss function, but the scope of the present disclosure is not limited thereto.

Additionally, the control system 10 may input (feed) the two-dimensional image 141 to the encoder 131 and predict the segmentation map 144 for the input image 141 through the third predictor 135. Additionally, the control system 10 may calculate the third loss L3 based on the difference between the predicted segmentation map 144 and the correct segmentation map 145. The third loss L3 may be calculated based on, for example, a cross-entropy loss function, but the scope of the present disclosure is not limited thereto.

Next, the control system 10 calculates the total loss by aggregating (e.g., weighted sum) the first loss (L1) to the third loss (L3), and encoder 131 and decoder 132 based on the total loss. and the parameters of the predictors 133 to 135 may be updated. At this time, considering the importance of the task, the highest weight may be assigned to the first loss (L1) and the lowest weight may be assigned to the second loss (L2), but the scope of the present disclosure is limited thereto.

The above-described processes can be performed repeatedly for other training samples. By doing so, the deep learning model (e.g., first predictor 133) can be equipped with the ability to accurately generate a depth map corresponding to the input image (e.g., 141).

Meanwhile, in some embodiments, as shown, additional losses (e.g., L4 to L6) may be used in training the deep learning model.

For example, the control system 10 may apply a fourth loss (L4 ) can be further calculated. Any method may be used to calculate the borderline difference. And, the control system 10 may update the parameters of the deep learning model further based on the fourth loss (L4). In this case, the deep learning model can be trained to perform predictions by focusing more on the vicinity of the object's border.

As another example, the control system 10 may apply a fifth loss (L5) based on the difference between the edge extracted from the predicted segmentation map 144 and the edge extracted from the input image 141 (or the correct segmentation map 145) ) can be further calculated. And, the control system 10 may update the parameters of the deep learning model further based on the fifth loss (L5). Even in this case, the deep learning model can be trained to perform predictions by focusing more on the vicinity of the object's border.

As another example, the control system 10 may further calculate the sixth loss L6 based on the difference between the boundary line extracted from the predicted depth map 142 and the boundary line extracted from the predicted segmentation map 144. there is. And, the control system 10 may update the parameters of the deep learning model further based on the sixth loss (L6). Even in this case, the deep learning model can be trained to perform predictions by focusing more on the vicinity of the object's border.

As another example, control system 10 may calculate losses based on various combinations of the examples described above. For example, the control system 10 may further calculate the fourth loss (L4) to the sixth loss (L6) and update the parameters of the deep learning model based on these losses (L4 to L6).

When learning of the deep learning model is completed, the control system 10 can generate a depth map for the captured image through the learned deep learning model. For example, the control system 10 may generate a first depth map by inputting a first image taken directly below the current layer of the product into a learned deep learning model, and a first image taken from the side of various layers of the product. 2 A second depth map may be generated by inputting the image into the learned deep learning model.

Meanwhile, in some embodiments, the control system 10 may generate new training samples through a data augmentation technique and further use them to learn a deep learning model. By doing so, the performance of the deep learning model can be further improved. These embodiments will be further described with reference to FIGS. 15 and 16 below.

15 and 16 are exemplary diagrams for explaining a data enhancement method according to some embodiments of the present disclosure.

As shown in FIG. 15 , the control system 10 may generate a new image 153 by extracting the object area 152 from the original image 151. Here, the original image 151 refers to the original two-dimensional image existing in the training set, and the new image 153 refers to an object image with the non-object area (i.e., background area) removed, for example. can do.

Next, the control system 10 may generate a new correct answer depth map 156 for the new image 153 by extracting the object area 155 from the correct answer depth map 154 of the original image 151. At this time, the control system 10 may generate a new correct answer depth map 156 by replacing the depth value corresponding to the non-object area in the correct answer depth map 154 with a constant.

If a deep learning model is trained using the new image 153 and the new correct answer depth map 156 as above, the deep learning model can be trained to predict the depth value by focusing more on the internal area of the object.

In some cases, the control system 10 selects a training sample in which the prediction loss (e.g., prediction loss for the object area) of the predictor (e.g., 133, 134, 135) in the training set is above the reference value (or the prediction accuracy is below the reference value). It is also possible to select samples (e.g., samples consisting of a two-dimensional image and a depth map of the correct answer) and create new training samples using the selected training samples.

Additionally, in some cases, the control system 10 may determine the number of new training samples generated based on the prediction loss (e.g., prediction loss for the object area) of the predictors (e.g., 133, 134, 135) (e.g. , for training samples with high prediction loss, more new training samples are generated).

Additionally, in some cases, the control system 10 may further generate a second new image by changing the color, shape and/or position of the object area (e.g., 152) in the first new image (e.g., 153). It may be possible.

Hereinafter, another example of a data enhancement method will be described with reference to FIG. 16.

As shown in FIG. 16 , the control system 10 may generate a new image 163 by extracting only a partial area 163 from the object area 152 of the original image 161. Here, the partial area 163 may refer to an area in the object area 162 where the prediction loss of the first predictor (i.e., the difference between the predicted depth value and the correct depth value) is greater than or equal to the reference value.

Next, the control system 10 extracts some areas 167 within the object area 166 from the correct answer depth map 165 of the original image 161 to create a new correct answer depth map 168 for the new image 164. can be created. At this time, the control system 10 may generate a new correct answer depth map 168 by replacing the depth value corresponding to the non-object area in the correct answer depth map 165 with a constant.

If a deep learning model is trained using the new image 164 and the new correct answer depth map 168 as above, the deep learning model can be trained to more accurately predict the depth value of the inner area of the object.

So far, a deep learning depth map generation method according to some embodiments of the present disclosure has been described with reference to FIGS. 13 to 16. According to the above, by training a deep learning model using a number of tasks related to depth map prediction, the depth map prediction performance of the deep learning model can be greatly improved. Additionally, by using this deep learning model, an accurate and sophisticated depth map can be obtained even when there is no depth camera module.

Hereinafter, an exemplary computing device 170 capable of implementing the control system 10 according to some embodiments of the present disclosure will be described with reference to FIG. 17.

17 is an exemplary hardware configuration diagram showing the computing device 170.

As shown in FIG. 17, the computing device 170 loads one or more processors 171, a bus 173, a communication interface 174, and a computer program 176 performed by the processor 171. It may include a memory 172 that stores a computer program 176 and a storage 175 that stores a computer program 176. However, only components related to the embodiment of the present disclosure are shown in FIG. 17. Accordingly, a person skilled in the art to which this disclosure pertains can recognize that other general-purpose components may be further included in addition to the components shown in FIG. 17 . That is, the computing device 170 may further include various components in addition to those shown in FIG. 17 . Additionally, in some cases, the computing device 170 may be configured with some of the components shown in FIG. 17 omitted. Hereinafter, each component of the computing device 170 will be described.

The processor 171 may control the overall operation of each component of the computing device 170. The processor 171 is at least one of a Central Processing Unit (CPU), Micro Processor Unit (MPU), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), or any type of processor well known in the art of the present disclosure. It can be configured to include. Additionally, the processor 171 may perform operations on at least one application or program to execute operations/methods according to embodiments of the present disclosure. Computing device 170 may include one or more processors.

Next, memory 172 may store various data, commands and/or information. Memory 172 may load a computer program 176 from storage 175 to execute operations/methods according to embodiments of the present disclosure. The memory 172 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

Next, bus 173 may provide communication functionality between components of computing device 170. The bus 173 may be implemented as various types of buses, such as an address bus, data bus, and control bus.

Next, the communication interface 174 may support wired or wireless Internet communication of the computing device 170. Additionally, the communication interface 174 may support various communication methods other than Internet communication. To this end, the communication interface 174 may be configured to include a communication module well known in the technical field of the present disclosure.

Next, storage 175 may non-transitory store one or more computer programs 176. The storage 175 may be a non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or a device well known in the art to which this disclosure pertains. It may be configured to include any known type of computer-readable recording medium.

Next, the computer program 176, when loaded into the memory 172, may include instructions that cause the processor 171 to perform operations/methods according to various embodiments of the present disclosure. That is, the processor 171 may perform operations/methods according to various embodiments of the present disclosure by executing loaded instructions.

For example, the computer program 176 includes an operation of acquiring design data for each layer of a product, an operation of receiving a photographed image of the current layer of a product output from a laminated 3D printer in real time, and an operation of receiving design data for each layer in real time. By comparing the obtained design data of the current layer with the captured image, perform an operation to check for an output error in the current layer and an operation to stop the output of the 3D printer based on the inspection result that an output error exists in the current layer. It can include instructions that do.

As another example, computer program 176 may include instructions to perform at least some of the steps/operations described with reference to FIGS. 1-16.

In the case illustrated, control system 10 according to some embodiments of the present disclosure may be implemented via computing device 170.

So far, an exemplary computing device 170 capable of implementing the control system 10 according to some embodiments of the present disclosure has been described with reference to FIG. 17 .

The technical ideas of the present disclosure described so far with reference to FIGS. 1 to 17 may be implemented as computer-readable codes on a computer-readable recording medium. Computer-readable recording media may be, for example, removable recording media (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or fixed recording media (ROM, RAM, computer-equipped hard disk). A computer program recorded on a computer-readable recording medium can be transmitted to and installed on another computing device through a network such as the Internet, and thus can be used on another computing device.

In the above, even though all the components constituting the embodiments of the present disclosure have been described as being combined or operated in combination, the technical idea of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the purpose of the present disclosure, all of the components may be operated by selectively combining one or more of them.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it exists.

Although various embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing the technical idea or essential features. I can understand that it is possible. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of this disclosure should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of the technical ideas defined by this disclosure.

Claims (10)

A method performed by at least one computing device, comprising:
Obtaining design data for each layer for the product;
Receiving in real time a captured image of the current layer of the product output from a laminated 3D printer;
Comparing the design data of the current layer obtained from the design data for each layer with the captured image, thereby performing an inspection for an output error of the current layer; and
Stopping the output of the 3D printer based on the result of the inspection that an output error exists in the current layer,
The step of performing a check for an output error of the current layer includes further comparing design data of the current layer and a depth map of the current layer,
The step of further comparing the design data of the current layer and the depth map of the current layer includes comparing the design data of the current layer and the depth map of the current layer based on a determination that the complexity of the current layer is greater than or equal to a reference value. A 3D printer control method comprising:
delete delete According to claim 1,
The step of checking for output errors of the current layer is:
calculating the complexity of the current layer;
adjusting a tolerance to be applied to the current layer based on the complexity; and
A 3D printer control method comprising performing a check for the output error based on the adjusted tolerance.
According to claim 1,
calculating a difference between the current layer and the previous layer based on the design data for each layer; and
Further comprising performing an output error check regarding missing layers based on a determination that the calculated difference is greater than or equal to a reference value,
The steps to perform output error checking for missing layers are:
Receiving images taken from the side of layers previous to the current layer, including the immediately preceding layer;
analyzing the image and counting the number of previous layers; and
A 3D printer control method comprising determining whether an output error related to the missing layer exists by checking the number of previous layers based on the design data for each layer.
According to claim 1,
calculating a difference between the current layer and the previous layer based on the design data for each layer; and
Further comprising performing an output error check regarding layer shift based on a determination that the calculated difference is less than a reference value,
How to control a 3D printer.
According to clause 6,
The step of performing an output error check for the layer shift is:
Obtaining a side depth map for the current layer and the previous layer; and
A 3D printer control method comprising determining whether an output error related to the layer shift exists by comparing the depth value of the current layer and the depth value of the immediately preceding layer in the side depth map.
In clause 7,
The step of acquiring the side depth map is,
Receiving side images for the current layer and the immediately preceding layer; and
Generating the side depth map from the side image through a deep learning model,
The deep learning model is:
a shared encoder that extracts features from the input image;
a shared decoder that decodes the extracted features;
a first predictor that receives the decoded features and predicts a depth map for the input image;
a second predictor that receives the decoded features and predicts a class label of the input image; and
A third predictor that receives the decoded features and predicts a segmentation map for the input image,
The shared encoder and the shared decoder are learned based on prediction losses of the first to third predictors,
How to control a 3D printer.
According to clause 8,
The shared encoder and the shared decoder,
A first loss calculated from the difference between the boundary line between the input image and the predicted depth map,
A second loss calculated from the boundary line difference between the input image and the predicted segmentation map, and
Learned further based on a third loss calculated from the boundary line difference between the predicted depth map and the predicted segmentation map,
How to control a 3D printer.
According to clause 9,
The deep learning model is,
The original image and the depth map of the correct answer for the original image,
A first image generated by extracting an object area from the original image and a first depth map generated by replacing a depth value corresponding to a non-object area in the correct answer depth map with a constant, and
It is learned using a second image generated by extracting a partial area within the object area of the original image and a second depth map for the second image generated from the first depth map,
The partial area is an area in which the difference between the predicted depth value obtained through the first predictor and the correct answer depth value within the object area is greater than or equal to a reference value.