KR20220008184A - Method for training a model to identify hull blocks based on a convolutional neural network - Google Patents

Abstract

The present invention relates to a hull block identification model learning method based on a convolutional neural network (CNN) which models and learns a plurality of images obtained by rotating a hull block object in 3D CAD data by multi-views based on a CNN, thereby automatically identifying and classifying a hull block from data obtained from a camera.

Description

Method for learning hull block identification model based on convolutional neural network

The present invention relates to a method for learning a hull block identification model based on a convolutional neural network, and more specifically, to a convolutional neural network ( By modeling and learning based on CNN), it relates to a method for learning a hull block identification model based on a convolutional neural network that can automatically identify and classify hull blocks from data acquired from a camera.

In general, the part that takes the most time according to the schedule in the ship's production process is the stage before docking. Because large ships built at large shipyards are very large, the entire ship is divided into blocks of hundreds of units for production and design. Hundreds of blocks constituting a large ship are assembled through various workshops, and in this process, they are brought in and out of the warehouse where the process must be performed at least several dozen times a day and at most hundreds of times a day.

In addition, blocks with completed or unassigned work are temporarily placed in an outdoor stockyard, or for some reason, the work is carried out by moving to an alternative preferred work site where work is possible, such as the planned work site. In this way, the blocks move fluidly and go through process steps, unlike the pre-configured plan for various reasons.

It is necessary to systematically manage blocks in order to smoothly proceed with the block process step, which takes a lot of time according to the schedule. In particular, in order to ensure that the assembly blocks can be assembled within a set period in the pre-mounting process, which is the stage before dock mounting, within the period established by the schedule, information on the workshop where the blocks for assembling the assembly blocks are located must be secured.

Most shipyards rely on the double task of field workers entering the work recorded during the day into the PC system after work hours. In addition, although GPS for location tracking and a PDA for inputting block movement information were installed in the transporter and used to determine the block location, there is a problem in many cases where movement performance is omitted due to network speed and inconvenient input devices.

As such, in the case of a system in which an operator manually inputs location information, a problem in the input device or a problem in which incorrect information is input due to an operator's judgment error or erroneous input may occur. Due to the wrongly inputted block location information, it may take several tens of minutes to several hours to find the corresponding block later in proportion to the size of the stockyard.

This causes problems in proceeding with the next task, which eventually leads to setbacks in the schedule. In this way, it is important to know the location of the block in the shipyard to avoid any disruption to the schedule.

In order to track the position of a block, both block coordinates and block number must be accurately entered into the system. If there is an error in either block coordinate or block number in the input block location information, the block location cannot be determined. For example, even if the block coordinates of the secured block location information are correct, if there is an error in the block number, eventually the location of the required block cannot be tracked.

Research to track the location of blocks in shipyards has been continuously conducted. However, most studies have been conducted to track the coordinates of blocks, and studies to identify block numbers are scarce. When tracking the block location, even if the block coordinates are correct, if the block number is incorrect, a problem occurs that the block location cannot be tracked. So, it is as important to accurately identify the number of a block as it is to accurately obtain the coordinates of the block.

When identifying block numbers to obtain block location information, errors may occur due to operator's erroneous input or judgment error. Therefore, it is necessary to develop a block identification system that does not require the manual operation of the operator to compensate for the problems that may occur when entering the block number into the block location information system due to the operator's erroneous input or judgment error.

Korean Patent Publication No. 10-2006-0102078

The present invention is to solve the above problems, and by modeling and learning a number of images obtained by rotating a hull block object in dimensional CAD data with multiple views based on a convolutional neural network (CNN), from the camera To provide a method for learning a hull block identification model based on a convolutional neural network that can automatically identify and classify hull blocks from acquired data.

A method for learning a hull block identification model based on a convolutional neural network according to an embodiment of the present invention is to acquire a plurality of images by continuously rotating a hull block object in three-dimensional CAD data at a predetermined angle for each coordinate plane axis. Step, applying data augmentation to the plurality of images to extend the number of images of the plurality of images, and based on a convolutional neural network (CNN), a plurality of identification images prepared based on the plurality of images It may be characterized in that it comprises the step of learning the hull block identification model by repeating the identification process.

In one embodiment, the step of acquiring the plurality of images includes loading the three-dimensional CAD data from a database and then using the hull block object in the three-dimensional CAD data from 0 degrees to 90 degrees along the X axis based on a coordinate plane system. Acquiring 360 or more images by continuously rotating at 10-degree intervals within an angle range, and continuously rotating at 10-degree intervals within an angle range of 0 to 350 degrees along the Z-axis. can

In an embodiment, the acquiring of the plurality of images includes extracting an object region and a background contour region for an object in the 360 or more images, and based on the size of the extracted object region and the background contour region, the object After generating a rectangular region including a region and a circle region tangent to the vertices of the rectangular region, the image size of each of the 360 or more images is corrected along the edge of the square region inscribed in the circle region. It may be characterized in that it further comprises the step of normalizing the image size.

In one embodiment, the step of extending the number of images of the plurality of images rotates each image at an angle of 30 degrees, 60 degrees and 90 degrees, or inverts the images in the vertical and left directions to extend the number of images to at least 6 or more. It may be characterized in that it comprises a step of making.

In one embodiment, the step of learning the hull block identification model repeats the process of classifying some images among the plurality of images, and first identifying the partial images classified based on the plurality of images based on a convolutional neural network. It may be characterized in that it comprises the step of repeating the process of identifying a plurality of pre-prepared identification images after the performing and the identification process for the partial images are repeatedly performed.

According to an aspect of the present invention, the hull block identification model can be modeled and learned so that the hull block identification, classification, and numbering can be automatically recognized with high accuracy from the data acquired from the camera. have

1 is a flowchart sequentially illustrating a method for learning a hull block identification model based on a convolutional neural network according to an embodiment of the present invention.
2 is a diagram illustrating a state in which a hull block object in 3D CAD data generated based on an actual hull block design drawing is output.
3 is a view showing different types of hull block types requiring classification.
Figure 4 is a view showing a state of rotating the hull block shown in Figure 3 in the X-axis, Z-axis.
5 is a diagram illustrating a plurality of images obtained through the process of FIG. 4 .
6 is a diagram illustrating a process of extracting an object area and a background contour area for a hull object from an image.
FIG. 7 is a diagram illustrating a process of standardizing an image size based on the sizes of an object region and a background contour region extracted through FIG. 6 .
8 is a diagram illustrating a process of expanding a plurality of images.
9 is a diagram illustrating a block model produced by outputting 3D CAD data to a 3D printer.
10 is a view showing a state in which only a portion of the hull blocks stacked in an actual shipyard stockyard is captured and captured as an image.
11 is a view showing a state in which the size of the image of FIG. 10 is corrected and standardized.
12 is a diagram illustrating loss values and accuracy according to training obtained images on three different types of convolutional neural network (CNN) models.

Hereinafter, preferred examples are presented to help the understanding of the present invention. However, the following examples are only provided for easier understanding of the present invention, and the content of the present invention is not limited by the examples.

1 is a flowchart sequentially illustrating a method for learning a hull block identification model based on a convolutional neural network according to an embodiment of the present invention, and FIG. 2 is a hull block object in 3D CAD data generated based on an actual hull block design diagram. is a diagram showing the output state, FIG. 3 is a diagram showing different types of hull block types that require classification, and FIG. 4 is a state in which the hull block shown in FIG. 3 is rotated on the X-axis and Z-axis. FIG. 5 is a diagram illustrating a plurality of images obtained through the process of FIG. 4 .

Referring to FIG. 1 , the method for learning a hull block identification model based on a convolutional neural network according to an embodiment of the present invention is first by continuously rotating a hull block object in three-dimensional CAD data at a predetermined angle for each axis of the coordinate plane system. It starts with a step (S101) of acquiring a plurality of images.

In step S101, after loading 3D CAD data from the database, the hull block object in the 3D CAD data is continuously rotated at intervals of 10 degrees within the range of 0 degrees to 90 degrees along the X axis based on the coordinate plane system. , and also continuously rotating at intervals of 10 degrees within an angle range of 0 degrees to 350 degrees along the Z axis to obtain 360 or more images.

More specifically, the three-dimensional CAD data for the ship block of the present invention may be visually confirmed through software running on a PC as shown in FIGS. 2 and 3 .

Referring to FIGS. 2 and 3 , the block classes to be classified are three types of blocks B1, B2, and B3 that have similar shapes as in FIG. 3 but have different length and width ratios and different characteristics of internal lattice structures.

In the present invention, in one embodiment, using the VBA (Visual Basic Application) module provided by Inventor, when the X-axis rotates by 10 degrees from 0 to 90 degrees, the Z-axis rotates by 10 degrees from 0 to 350 degrees. You can automatically acquire 360 images of 500*300 size.

Referring to FIG. 5(a), when the Z-axis is 0 degrees with respect to the coordinate plane, when the X-axis rotates from 0 to 90 degrees by 10 degrees, 10 images are acquired every 10 degrees, and Similarly, when the X axis is 0 degrees, when the Z axis rotates every 10 degrees from 0 degrees to 350 degrees, 36 images are acquired, and 360 images can be acquired.

Returning to FIG. 1 , if a plurality of images are acquired, data augmentation is applied to the plurality of images to extend the number of images ( S102 ).

More specifically, in step S102, the object area and background contour area for the ship block object in the 360 or more acquired images are extracted, and based on the sizes of the extracted object area and background contour area, a rectangular area including the object area and After generating a circle region tangent to the vertex of the rectangular region, the image size of each of 360 images or more is corrected along the corners of the square region inscribed in the circle region, thereby standardizing the image sizes of 360 or more images. This will be described with reference to FIGS. 6 to 8 .

6 is a diagram illustrating a process of extracting an object region and a background contour region for a hull object from an image, and FIG. 7 is a method for standardizing the image size based on the size of the object region and background contour region extracted through FIG. It is a diagram illustrating a process, and FIG. 8 is a diagram illustrating a process of expanding a plurality of images.

6 to 8, in the present invention, the outline of the object area and background for the hull block object of the image obtained using the Contour algorithm provided by the OpenCV library of Python. extract Also, using the maximum X value, maximum Y value, and minimum X value and minimum Y value of the extracted contour, a rectangular area suitable for the object area is formed, and a circle area touching the four vertices of the rectangular area is formed. Then, the image is cut along the edge of the square region inscribed with the formed circle region. As a result, as shown in FIG. 6 , the cropped image is corrected to a size of 224*224 and the size is standardized.

Referring to FIG. 8 , in the present invention, data augmentation is applied to a total of 1,080 images obtained by 360 sheets for each hull block class. If data expansion is applied to one image, the image can be rotated 30 degrees, 60 degrees, or 90 degrees, or reversed up and down, left and right to increase to 6 images.

1, in the present invention, based on a convolutional neural network (CNN) in a state in which the number of images of a plurality of images is expanded, the process of identifying a plurality of identification images prepared based on a plurality of images is repeatedly performed. The identification model is learned (S103).

More specifically, in step S103, a process of classifying some images among a plurality of images, first identifying some images classified based on a plurality of images based on a convolutional neural network, is repeated, and also an identification process for some images. After repeating, the process of identifying a plurality of previously prepared identification images is repeated.

Here, a plurality of pre-prepared identification images means data for evaluating (verifying) identification accuracy after learning a plurality of previously acquired images based on a convolutional neural network (CNN).

The identification image is as follows.

9 is a view showing a block model produced by outputting three-dimensional CAD data to a 3D printer, and FIG. 11 is a view showing a state in which the size of the image of FIG. 10 is corrected and standardized.

9 to 11 , the identification image is obtained from a block model produced by outputting 3D CAD data to a 3D printer. It is obtained by shooting the produced block model from multiple viewpoints using a camera. The original size of the obtained image is 3024*3024, but it is applied by modifying it to 224*224, the size that can be input into the convolutional neural network (CNN) model. At this time, 200 identification images are provided for each hull black class, and a total of 600 identification images can be prepared.

Meanwhile, the three convolutional neural network (CNN) models applied in the present invention use three types of convolutional neural network (CNN) models: VGGNet, ResNet, and DenseNet. same.

VGGNet is a simple structure that consists of a classification layer consisting of a full connection layer after a feature extraction layer in which the convolution layer and the pooling layer are repeated. ResNet is a structure in which each layer is connected by residual connection consists of When the CNN model learns an image input to the convolution layer by applying residual learning, it learns only from the difference value between the existing parameter of the convolution layer and the parameter newly created from the input image. In addition, it is an operation that simply adds the feature maps between the convolutional layers through residual connection, which reduces the amount of computation and also prevents the washout problem in which the features of the feature map disappear.

DenseNet uses a connection method in which feature maps are continuously concatenated. The connection method of DenseNet efficiently delivers advanced information to the latter layer of the deep CNN model, preventing the washout problem in which the features of the feature map of the input image disappear as the structure deepens, and the convolutional feature map connection method is the CNN model. It makes learning easier because it maximizes the efficiency of error backpropagation. In addition, because DenseNet uses a small number of convolution filters compared to depth, the number of parameters is small, so the time consumed for learning is reduced.

12 is a diagram illustrating loss values and accuracy according to training obtained images on three different types of convolutional neural network (CNN) models.

12, the loss values of the training and verification data of the three CNN models (VGG-19, Resnet-152V2 and Densenet-201) converge to 0 as the learning step progresses. And the accuracy of the training and validation data of the three CNN models converges to 1 as the learning stage progresses. Therefore, when the three CNN models are trained using the basic image set, it can be confirmed that the learning has been well accomplished with the loss and accuracy values of the learning and validation data appearing in the learning stage converging to an appropriate value.

Although the above has been described with reference to the preferred embodiment of the present invention, those skilled in the art will be able to variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can.

Claims (5)

acquiring a plurality of images by continuously rotating the hull block object in the three-dimensional CAD data at a predetermined angle for each axis of the coordinate plane;
extending the number of pages of the plurality of images by applying data augmentation to the plurality of images; and
Based on a convolutional neural network (CNN), learning the hull block identification model by repeating the process of identifying a plurality of identification images prepared based on the plurality of images; Based on the hull block identification model training method.
According to claim 1,
The step of acquiring the plurality of images includes:
After loading the three-dimensional CAD data from the database, the hull block object in the three-dimensional CAD data is continuously rotated at intervals of 10 degrees within the range of 0 degrees to 90 degrees along the X axis based on the coordinate plane system, In addition, obtaining 360 or more images by continuously rotating at 10 degree intervals within the range of 0 degrees to 350 degrees along the Z axis; Method for learning a hull block identification model based on a convolutional neural network, comprising: a.
3. The method of claim 2,
The step of acquiring the plurality of images includes:
extracting an object area and a background contour area for the ship block object in the 360 or more images; and
A rectangular region including the object region and a circle region tangent to a vertex of the rectangular region are generated based on the extracted object region and the size of the background contour region, and then, along the edge of the square region inscribed in the circle region Standardizing the size of the 360 or more images by correcting the image size of each of the 360 or more images;
According to claim 1,
The step of extending the number of images of the plurality of images,
Rotating each image at an angle of 30 degrees, 60 degrees and 90 degrees, or inverting it in the vertical and left directions to extend the number of sheets to at least 6 or more; How to train a hull block identification model.
According to claim 1,
Learning the hull block identification model comprises:
classifying some of the plurality of images and repeating the process of first identifying the partial images classified based on the plurality of images based on a convolutional neural network; and
After repeating the identification process for some of the images, repeating the process of identifying a plurality of previously provided identification images is a method for learning a hull block identification model based on a convolutional neural network, comprising:

Images