KR20240044026A - Method and system for estimating shape of ships - Google Patents

Abstract

A ship shape estimation method and system are disclosed. The ship shape estimation method according to an embodiment of the present invention includes the steps of constructing a ship shape estimation model by learning a plurality of datasets prepared for the ship, and automatically calculating the ship shape in conjunction with the generation of a shape estimation command for the target ship. A step of retrieving an AIS signal related to the target ship from an automatic identification system, preprocessing the AIS signal to configure a first input signal that can be input to the ship shape estimation model, and the first input signal It includes obtaining an image estimating the shape of the target ship output from the ship shape estimation model according to the input of .

Description

Ship shape estimation method and ship shape estimation system {METHOD AND SYSTEM FOR ESTIMATING SHAPE OF SHIPS}

The present invention relates to a method of shaping a ship by fusing a ship's AIS (Automatic Identification System) signal and a satellite image signal, and more specifically, to obtain an estimated image of the shape of a ship based on a learning model using AIS signals and satellite image signals. It is related to improving the convenience of ship monitoring through

Ships at sea continuously transmit AIS (Automatic Identification System) ship signals to inform of the ship's status, and these AIS ship signals are received through an antenna at a ground receiving station close to the ship at sea, or from a satellite without distance limit. possible.

Meanwhile, in the case of AIS ship signals transmitted from small ships, the transmission power is lower than that of large ships such as container ships or oil tankers, so it is difficult to receive them through satellite. Additionally, when the small ship is far away from the ground receiving station, the AIS signal is transmitted from the ground receiving station as well. Since ship signals cannot be received, it is difficult to determine the ship's status.

Therefore, small vessels located far from ground stations can only be monitored by imaging from high-resolution Earth observation satellites.

However, ship images captured by high-resolution Earth observation satellites only provide a limited-resolution top view of the ship's shape seen from directly below and location information including a certain amount of geometric error, provided by the AIS ship signal. Because various information such as the type of ship, ship name, size, location, direction, and speed are not included, there are limitations in accurately estimating the shape of the actual ship.

Therefore, there is a need for a technology that can image even small ships that are far from a ground station by utilizing AIS ship signals transmitted by ships together with satellite image signals.

The purpose of the embodiment of the present invention is to propose a method of generating an image estimating the shape of a specific ship using a learning-based artificial neural network when one or more information among AIS signals and satellite images is given for a specific ship. do.

An embodiment of the present invention is a ship shape estimation model that can generate an estimated image of the ship shape in each of the cases where only satellite image signals are collected, when only AIS signals are collected, and when both signals are collected for a specific ship. The purpose is to build.

An embodiment of the present invention fuses the ship's AIS signal and the satellite image signal to accurately estimate the ship shape of not only large ships, but also small ships of various shapes and sizes with relatively low AIS signal transmission power. The purpose.

The ship shape estimation method according to an embodiment of the present invention includes the steps of constructing a ship shape estimation model by learning a plurality of datasets prepared for the ship, and automatically calculating the ship shape in conjunction with the generation of a shape estimation command for the target ship. A step of retrieving an AIS signal related to the target ship from an automatic identification system, preprocessing the AIS signal to configure a first input signal that can be input to the ship shape estimation model, and the first input signal It may include obtaining an image that estimates the shape of the target ship output from the ship shape estimation model according to the input of .

In addition, the ship shape estimation system according to an embodiment of the present invention is linked to a learning processing unit that configures a ship shape estimation model by learning a plurality of datasets prepared for ships and the generation of a shape estimation command for the target ship. A preprocessing unit that searches the AIS signal related to the target ship from the automatic ship identification system, preprocesses the AIS signal to form a first input signal that can be input to the ship shape estimation model, and inputs the first input signal. Accordingly, it may include an acquisition processing unit that acquires an image that estimates the shape of the target ship output from the ship shape estimation model.

According to the present invention, the convenience of ship monitoring can be improved by constructing a ship shape estimation model that can estimate the actual ship shape with only limited information using AIS signals, satellite image signals, or both.

According to the present invention, more efficient data utilization is possible by narrowing down the measured data and actual shape, such as real-time ship monitoring in manned aircraft, reading training materials, accident situation identification, and reference materials when constructing learning materials.

Figure 1 is a block diagram showing the configuration of a ship shape estimation system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating the creation process of a ship shape estimation model included in the ship shape estimation system shown in FIG. 1.
Figure 3 is a diagram showing the configuration of a ship shape estimation system according to another embodiment of the present invention.
Figure 4 is a diagram illustrating a ship image obtained through a ship shape estimation model according to an input signal in the ship shape estimation system according to embodiments of the present invention.
Figure 5 is a diagram illustrating a ship patch image (second input signal) formed by pre-processing a satellite image signal in the ship shape estimation system according to embodiments of the present invention.
Figure 6 is a flowchart showing the sequence of a ship shape estimation method according to embodiments of the present invention.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, various changes may be made to the embodiments, so the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

Figure 1 is a block diagram showing the configuration of a ship shape estimation system according to an embodiment of the present invention.

Referring to FIG. 1, the ship shape estimation system 100 according to an embodiment of the present invention uses a ship shape generation model (hereinafter, ship shape estimation model) 200 to generate AIS information (dynamic/static) from the ship. ) 410 and a satellite image (scene unit) 420 from a satellite, an image estimating the shape of the target ship can be obtained.

Here, AIS information (dynamic/static) 410 is divided into dynamic information and static information. Dynamic information includes the frequently updated location, direction, and speed of the ship, while static information includes unchanging information such as ship type (ship type), ship name, and ship size, and these include the unique identification number (unique identification number) assigned to the ship. MMSI), and the nationality of the vessel can be identified from the top 3 digits of the MMSI, which consists of 7 digits.

Satellite image (scene unit) 420 is a top view image captured by a satellite, has a relatively limited resolution, and includes an approximate location calculated by angle and altitude, but the location of a marine vessel is The error is relatively large compared to the ground, and unlike AIS information (dynamic/static) 410, various information that affects the shape of the ship is not included.

The image estimating the shape of the target ship is based on whether the input signal used when estimating the shape of the target ship is either AIS information (dynamic/static) 410 or satellite image (scene unit) 420. Depending on whether there are both information (dynamic/static) 410 and satellite images (scene unit) 420, ship signal-based ship shape 430, ship patch-based ship shape 440, and fusion input-based ship shape ( 450).

Specifically, when AIS information (dynamic/static) 410 from a ship is input as the input signal, the ship shape estimation model 200 generates a ship shape based on the ship signal feature quantity generator 201 and the ship signal. The unit 204 may output a ship signal-based ship shape 430 having a shape expected from the nationality, size, ship type, and weight of the target ship identified from the AIS information (dynamic/static) 410.

In addition, when a satellite image (scene unit) 420 from a satellite is input as the input signal, the ship shape estimation model 200 includes a ship patch feature generation unit 202 and a ship patch-based ship shape generation unit 205. ), a ship patch-based ship shape 440 having the color expected from the RGB/NIR value of the target ship identified from the satellite image (scene unit) 420 can be output.

In addition, when both AIS information (dynamic/static) 410 from a ship and satellite image (scene unit) 420 from a satellite are input as the input signal, the ship shape estimation model 200 is a feature synthesis unit. (203), and a fusion input-based ship shape ( 450) can be output.

Therefore, the ship shape estimation system 100 can be used even when only AIS information 410 is collected for a specific ship, when only satellite images 420 are collected, and even when both AIS information 410 and satellite images 420 are collected. , using the ship shape estimation model 200, it is possible to obtain an image that estimates the shape of the ship even if there is a slight difference in quality (resolution) with only limited information.

Through this, according to the present invention, it is easy to estimate the shape of small vessels of various shapes and sizes that are far away from the ground receiving station, so it is possible to quickly determine whether a fishing vessel operating in our waters is a fishing vessel of another nationality and respond accordingly. can help with monitoring.

The ship shape estimation system 100 includes a ship signal input generator 101 that preprocesses the input signal to input AIS information 410 or satellite image 420 into the ship shape estimation model 200, It may be configured to include a ship patch input generation unit 102 and a ship selection and input information matching unit 103.

The ship signal input generator 101 extracts data (dynamic/static) affecting the shape of the ship from the AIS information 410 and generates a first input signal in a form that can be input to the ship shape estimation model 200. Compose.

Here, among the AIS information 410, the ship's nationality, ship type (ship type), ship size, ship weight, etc. are extracted as static information that affects the shape of the ship, and the ship's location (latitude and longitude) is extracted from the AIS information 410. Coordinates), direction, speed, etc. can be extracted as dynamic information used to generate the movement trajectory of the ship. This is because determining whether the ship's purpose is for trade (cargo ship) or fishing (fishing ship) based on the ship's movement trajectory helps shape the ship.

Therefore, the ship signal input generator 101 converts the static information extracted from the AIS information 410, along with the movement trajectory generated from the dynamic information, into a one-dimensional tensor that can be input to the ship shape estimation model 200. By arranging them, the first input signal can be formed.

The ship patch input generator 102 uses the ship patch image obtained by extracting the ship area detected from the scene-unit satellite image 420 to a certain size to provide a second input in a form that can be input to the ship shape estimation model 200. Construct the signal.

Since the ship patch image extracted from the satellite image 420 consists of a two-dimensional signal of 3 channels RGB or 4 channels with NIR added, using the second input signal, RGB/NIR values representing the color of the ship can be obtained. It can be helpful in estimating.

For example, referring to FIG. 5, the ship patch input generator 102 detects the area of the ship by applying an object detection model to the satellite images (scene units) 420 and 510, and displays the satellite images (scene units) 420 , 510), the posture of the ship can be estimated by applying the posture estimation model to the ship's position, size, and direction, and the ship patch image 520 can be generated by extracting the ship area to a certain size based on the estimated posture. .

As shown in FIG. 5, a portion of a ship portion detected and extracted from Topview satellite images (scene units) 420, 510 that capture a ship on the ground can be generated as a ship patch image 520. The plurality of ship patch images 520 generated in this way can be used as a second input signal for the ship shape estimation model 200.

The ship selection and input information matching unit 103 is configured to receive a first input signal of a specific ship configured by the ship signal input generator 101 and a second input signal of a specific ship configured by the ship patch input generator 102. It functions to match and correlate with the unique identification number of the specific vessel.

That is, the vessel selection and input information matching unit 103 selects AIS information 410 collected for a specific vessel from among a plurality of AIS information 410 collected for several ships and a plurality of scene-level satellite images 420. It functions to match the and satellite images 420 so that they can be used together when estimating the shape of the relevant ship.

If only AIS information 410 or satellite images 420 are collected for a specific vessel, matching of the first and second input signals by the vessel selection and input information matching unit 103 is omitted. It can be.

For example, when any one AIS information 410 among a plurality of AIS information 410 is selected by the manager, the vessel selection and input information matching unit 103 photographs the vessel that transmitted the corresponding AIS information 410. Satellite images 420 can be automatically searched based on location and stored in association with the unique identification number (MMSI) assigned to the vessel.

As another example, if the satellite image 420 is first selected by the manager, a ship patch image may be generated for each of the multiple ships in the satellite image 420. The ship selection and input information matching unit 103 automatically searches the AIS information 420 transmitted from the ship in each ship patch image using latitude and longitude coordinates, and correlates it with the ship patch image using the same unique identification number (MMSI). It can be built and aligned.

This is because ships generally continuously transmit AIS information at regular time intervals, so there may be multiple AIS information corresponding to one ship patch image, and if multiple ships are gathered together, AIS information with similar latitude and longitude coordinates may also be included. As the number of AIS information increases, the AIS information transmitted from a specific ship in one ship patch image is searched in advance among multiple AIS information and correlated with the same unique identification number to match the AIS information to create one ship patch image and the corresponding AIS information. , so that it can be used together to estimate the shape of the ship.

The first and second input signals formed by the ship signal input generation unit 101 and the ship patch input generation unit 102 are input to the ship shape estimation model 200, and are input to the ship signal feature generation unit 201. ) and the ship patch input generator 102, and have shape/color estimated based on the feature quantities by the ship signal-based ship shape generator 204 and the ship patch-based ship shape generator 205. It can be used to generate ship images 430, 440, and 450.

First, the ship signal feature generation unit 201 processes the one-dimensional first input signal (e.g., 61*1*1 ship signal) input to the ship shape estimation model 200 into a deconvolution layer. ) is converted into a two-dimensional characteristic quantity of the n layer (where n is a natural number of 3 or more).

The n-layer two-dimensional feature quantity refers to a multi-layer low-resolution feature quantity obtained by overlapping n layers of two-dimensional feature quantities corresponding to low resolution to high resolution. For example, the low-resolution two-dimensional feature quantity of "(3*3)" “2048*(3*3)”, which has 2048 layers (n=2048), can be taken as an example of the two-dimensional characteristic quantity of the n layer.

The resolution of the first input signal in the ship signal feature generation unit 201 increases as the deconvolution layer for feature transformation is repeated, and the characteristics of the second input signal in the ship patch feature generation unit 202 The resolution can be expanded repeatedly until the quantity (ship patch characteristic quantity) becomes the same as the sheet resolution (e.g., 2048*3*3).

The ship patch feature generation unit 202 generates a second input signal (e.g., a ship patch expressed in RGB of 3*96*96) input to the ship shape estimation model 200, and uses a convolutional neural network (CNN, Through 'ResNet-50', a base network of Convolutional Neural Network, it is converted into a 2-dimensional feature quantity (e.g., 2048*3*3) of the n layer.

The ship patch is usually expressed as a 2-dimensional signal of 3 channels RGB or 4 channels with NIR added, from which the ship patch feature generation unit 202 generates a 2-dimensional feature amount of the n layer as many as a random number of channels. In this process, the resolution of each sheet decreases and the number of channels increases. This process can be performed through convolution and pooling of a CNN-based network.

When both the ship signal and the satellite image are collected for a specific ship and the feature quantity of the first input signal and the feature quantity of the second input signal are obtained, the feature quantity synthesis unit 203 generates each feature tensor (3D information ) are synthesized to create one feature tensor (fused feature quantity).

That is, the feature synthesis unit 203 appropriately synthesizes at least a part of the feature amount of the first input signal, which is more helpful in understanding the shape, and at least a part of the feature amount of the second input signal, which is more helpful in understanding the color. By generating high-resolution fusion feature quantities, it is possible to shape a ship in which both the shape and color of the ship based on the fusion feature quantity exist simultaneously.

The ship signal-based ship shape generator 204 may generate a ship signal-based ship shape 430 based on the feature amount of the first input signal generated by the ship signal feature amount generator 201. For example, the ship signal-based ship shape 430 may be created as a ship shape image expressed in black and white as shown in FIG. 4.

The ship patch-based ship shape generator 205 may generate a ship patch-based ship shape 440 based on the feature amount of the second input signal generated by the ship patch feature amount generator 202. For example, the ship patch-based ship shape 440 may be created as a ship shape image expressed in color as shown in FIG. 4.

The fusion ship shape generator 206 may generate a fusion input-based ship shape 450 based on the fusion feature quantity generated by the feature quantity synthesis unit 203.

In this case, the fusion ship shape generator 206 can use different first and second input signals to expand the resolution of the feature quantities of each signal, thereby generating a three-channel RGB ship photo as the final shape. For example, the fusion input-based ship shape 450 can be created as a ship shape image that has both the color of the ship signal-based ship shape 430 and the color of the ship patch-based ship shape 440, as shown in FIG. .

FIG. 2 is a diagram illustrating the creation process of a ship shape estimation model included in the ship shape estimation system shown in FIG. 1.

Referring to FIG. 2, the ship shape estimation system 100 is a ship shape capable of outputting images 430, 440, and 450 that estimate the shape of the ship according to input signals 410 and 420 regarding the target ship. The estimation model 200 can be created by learning a plurality of datasets consisting of AIS ship signals from ships, ship patch images from satellite images, and ship images.

In order to secure a dataset to be used for learning, the ship shape estimation system 100 includes a ship signal input generator 101 and a ship patch input generator 102 that preprocess each input signal in FIG. 1, and each input A ship signal feature quantity generation unit 201, a ship patch feature quantity generation unit 202, and a feature quantity synthesis unit 203 that convert signals into feature quantities can be used.

The ship signal input generator 101 preprocesses the AIS information collected from the ship to construct a one-dimensional ship signal, and the ship patch input generator 102 preprocesses the satellite image collected from the satellite to create a two-dimensional ship signal. A ship patch image expressed in RGB/NIR (e.g., 3*96*96 RGB image) can be configured.

For example, the ship signal input generator 101 determines the nationality of the ship, the ship type (ship type), the overall length, the overall width, the weight, and a certain time interval (e.g., ±1 week) for a certain period of time (e.g., ±1 week) from the collected AIS information. , 12 hours) of latitude/longitude coordinates (e.g., a total of 28 position coordinates) are extracted to extract static/dynamic data to construct a one-dimensional 61*1*1 ship signal, and the ship signal feature quantity generator (201) ) can apply a one-dimensional 61*1*1 ship signal to a deconvolution layer and convert it into a two-dimensional feature quantity.

At this time, the ship signal feature generation unit 201 repeats the process until the resolution expanded by feature quantification of the ship signal becomes the same as the single-sheet resolution (e.g., 2048 * 3 * 3) of the feature quantity of the ship patch image, which will be described later. The resolution can be expanded by applying a deconvolution layer. This is to match the output size for fusion between the feature quantities of the ship signal and the feature quantities of the ship patch image in the feature quantity synthesis unit 203, which will be described later.

The ship patch input generator 102 applies the collected satellite images to an object detection model such as 'FasterRCNN' + 'ResNeSt101' and a posture estimation model such as 'Simple Pose model' to determine the positions and sizes of multiple detected ships. Based on the posture such as direction, a ship patch image (3*96*96) expressed in 3 RGB channels of a certain size (e.g., 96*96 pixels) is constructed, and the ship patch feature generation unit 202 The ship patch image (3*96*96) can be applied to the CNN base network 'ResNet-50' and converted to the n-layer 2D feature quantity (e.g., 2048*3*3).

The feature quantity synthesis unit 203 includes the ship signal feature quantity (2048*3*3) obtained from the ship signal feature quantity generation unit 201 and the feature quantity of the ship patch image obtained from the ship patch feature quantity generation unit 202 ( By applying 1*1 convolution to 2048*3*3), full connection at each pixel position is implemented, and the feature quantity of the ship signal and the feature quantity of the ship patch image are the same size (2048*). 3*3) fusion feature quantities can be generated.

When a dataset consisting of a ship signal feature, ship patch image feature, and fusion feature of size 2048*3*3 is secured through this process, and an image of an actual ship, the ship shape estimation system 100 uses the data set. The ship shape estimation model 200 can be constructed by learning and processing.

At this time, the ship shape estimation system 100 uses the image of the same actual ship as the correct value, but individually learns using different input values among the ship signal feature, ship patch image feature, and fusion feature, The first to third high resolution blocks can be generated.

The first high-resolution block can be generated by learning using the ship signal feature as input and the image of the actual ship as the correct value (GT, Ground Truth), and the second high-resolution block is the ship patch image. It can be generated by learning with a feature quantity as an input value and an image of an actual ship as an answer value, and the third high-resolution block uses a fusion feature quantity as an input value and an image of an actual ship as an answer value. It can be created through learning. In this way, the first to third high-resolution blocks are trained with the goal of producing the same result using different input values, so they have different weight values.

Thereafter, the ship shape estimation system 100 updates the weight values set in the first to third high-resolution blocks in batches by back-propagating the loss function (Loss) calculated for the same ship image, and creates a ship shape estimation model (200 ) can be optimized.

That is, the ship shape estimation system 100 uses the same ship image as GT and applies the test dataset to the first to third high-resolution blocks of the ship shape estimation model 200 learned using three input values. , Obtain the L2 loss function from three outputs (ship signal-based shape of size 3*48*48, satellite image-based shape, and fusion information-based shape) according to three input values, and optimize it in the direction of minimizing it. . Additionally, generative adversarial neural network (GAN) techniques can be applied to improve the performance of ship shape image generation.

When the ship shape estimation model 200 having the first to third high-resolution blocks is created and optimized using the three input values in this way, the ship shape estimation system 100 inputs the target ship in FIG. Regardless of whether the signal is an AIS signal, a satellite image signal, or both signals, the shape of the target ship can be estimated using the ship shape estimation model 200.

Accordingly, according to the present invention, the shape image of the target ship can be estimated and obtained based on the ship shape estimation model 200 with only limited information that can be obtained about the target ship in a poor marine environment.

Depending on the embodiment, each of the first to third high-resolution blocks in the ship shape estimation model 200 may be configured to include multiple levels of deconvolution layers. Accordingly, three feature quantities are input to the first to third high-resolution blocks, and in the process of generating a 3*48*48 ship shape image, each deconvolution layer in the first to third high-resolution blocks is passed. Since the resolution is doubled, a higher resolution ship shape image can be obtained by the ship shape estimation model 200 proposed in the present invention.

Figure 3 is a diagram showing the configuration of a ship shape estimation system according to another embodiment of the present invention.

Referring to FIG. 3, the ship shape estimation system 300 according to an embodiment of the present invention includes a learning processing unit 310, a pre-processing unit 320, an acquisition processing unit 330, and a database 340. You can.

The learning processing unit 310 constructs a ship shape estimation model by learning a plurality of datasets prepared about ships.

Prior to this, the learning processing unit 310 collects a plurality of data sets about ships to be used for learning from the automatic ship identification system 301 and the satellite 302, respectively, and stores them in the database 340, and stores a sufficient number of data sets on the ships. Once this is secured, learning can proceed.

Here, the Automatic Identification System (301) refers to a plurality of time-series AIS signals transmitted at regular time intervals by the ship's wireless transmitter and receiver and transmitted via a repeater, and associates them with the ship's unique identification number. Refers to a recording system.

The learning processing unit 310 can collect time-series AIS signals transmitted from various ships through the automatic vessel identification system 301, and can also collect satellite image signals from the satellite 302 of the ship for which the time-series AIS signal was collected. By collecting and pre-processing, a ship patch image can be prepared, and the actual shape image of the ship can be collected through the Internet, social networking system (SNS), video playback app, etc., or input from the administrator terminal, and this can be used as learning data. They can be stored in the database 340 as a set.

At this time, the learning processing unit 310 performs learning on a dataset that includes all the time-series AIS signals from the ship, the ship patch image from the satellite 302, and the actual shape image of the ship, but each dataset is different from the other. Learning can be done individually using different input values.

Specifically, the learning processor 310 creates a first high-resolution image in the ship shape estimation model by learning a dataset consisting of a time-series AIS signal (first input value) from the ship and a shape image (correct answer value) of the ship. Blocks can be created.

In addition, the learning processing unit 310 learns the dataset consisting of a ship patch image (second input value) extracted from a satellite image signal capturing the ship and a shape image of the ship (correct answer value), A second high-resolution block within the ship shape estimation model can be created.

In addition, the learning processing unit 310 learns the dataset consisting of the time-series AIS signal (first input value), the ship patch image (second input value), and the shape image of the ship (correct answer value), A third high-resolution block within the shape estimation model can be created.

The first to third high-resolution blocks in the ship shape estimation model generated by individual learning using different input values have different weights, and then the learning processor 310 controls the first high-resolution block, the Optimization of the ship shape estimation model can be performed by backpropagating the loss function (Loss) calculated in the learning process for each of the second high-resolution block and the third high-resolution block and collectively updating the set weights. there is.

In this way, the ship shape estimation model including the first to third high-resolution blocks generated and optimized through individual learning using different input values may be stored in the database 340.

Thereafter, as a shape estimation command for the target vessel is generated by the manager terminal, the pre-processing unit 320 and the acquisition processing unit 330, which will be described later, use AIS signals, satellite image signals, or By using both of them as input signals, the shape of the target ship can be estimated using the ship shape estimation model stored in the database 340.

The preprocessor 320 searches for the AIS signal related to the target ship from the automatic ship identification system 301 in conjunction with the generation of a shape estimation command for the target ship, preprocesses the AIS signal, and models the ship shape estimation model. Configures the first input signal that can be input to .

To describe the process of configuring the first input signal in detail, the preprocessor 320 selects a unique identification number (e.g., 7) assigned to the target vessel from among the plurality of time-series AIS signals recorded in the automatic vessel identification system 301. digits), you can search for AIS signals related to the target vessel recorded over a set period of time (e.g., 12-hour intervals for ±1 week).

The pre-processing unit 320 may extract static information affecting the shape of at least one of the nationality, type, size, and weight of the target ship from the AIS signal retrieved from the automatic ship identification system 301.

In addition, the preprocessor 320 extracts dynamic information that changes with time, such as the location, direction, and speed of the target vessel, and considers the dynamic information to generate a movement trace of the target vessel during the specified period. . Here, the movement trajectory of the target vessel can be used to shape the vessel according to its purpose because it can determine whether the target vessel is a cargo ship for trade or a fishing vessel for fishing.

The preprocessor 320 may configure the first input signal expressed as a one-dimensional tensor (eg, 61*1*1) by arranging the static information and movement trajectory of the target ship in one dimension.

The acquisition processing unit 330 acquires an image estimating the shape of the target ship output from the ship shape estimation model according to the input of the first input signal.

At this time, the preprocessor 320 converts the first input signal input to the ship shape estimation model into the n-layer two-dimensional feature quantity (the above) through a deconvolution layer connected to the input terminal of the ship shape estimation model. The resolution can be expanded by converting to (n is a natural number of 3 or more).

The n-layer two-dimensional feature quantity refers to a multi-layer low-resolution feature quantity obtained by overlapping n layers of two-dimensional feature quantities corresponding to low resolution to high resolution. For example, the low-resolution two-dimensional feature quantity of "(3*3)" “2048*(3*3)”, which has 2048 layers (n=2048), can be taken as an example of the two-dimensional characteristic quantity of the n layer.

The acquisition processing unit 330 inputs the n-layer two-dimensional feature quantity for the first input signal obtained through the preprocessor 320 into the first high-resolution block in the ship shape estimation model, and calculates the AIS signal-based A shape image of the target vessel in which the shape exists can be obtained.

As such, according to the present invention, when an 'AIS signal' is acquired in relation to a target ship, a ship shape image having a shape based on the AIS signal can be estimated and obtained by the first high-resolution block in the ship shape estimation model. You can.

Depending on the embodiment, when a satellite image signal capturing the target ship is received from the satellite 302, the preprocessor 320 preprocesses the satellite image signal to generate a second input signal that can be input to the ship shape estimation model. It can be configured.

Specifically, the preprocessor 320 divides the satellite image signal into scenes, applies a selected object detection model, detects the vessel area within the satellite image signal, and applies the selected posture estimation model, Estimating attitude information including the current position, size, and direction of movement of the vessel area in the satellite image signal, and extracting the vessel area to a certain size from the satellite image signal based on the estimated attitude information, RGB The second input signal can be configured to be expressed as a 2-dimensional signal of 3 channels or 4 channels with NIR added.

In addition, the preprocessor 320 receives the second input signal through 'ResNet-50', a base network of a convolutional neural network (CNN) connected to the input terminal of the ship shape estimation model, in the same way as the first input signal. It can be converted to a 2-dimensional feature quantity of the n layer (e.g., 2048*3*3).

At this time, when the AIS signal related to the target vessel is not searched from the automatic vessel identification system 301 and only the satellite image signal photographing the target vessel is received, the acquisition processing unit 330 determines n for the second input signal. Input the 2D feature quantity of the layer (e.g., 2048*3*3) into the second high-resolution block in the ship shape estimation model to obtain a shape image of the target ship in which the color based on the satellite image signal exists. can do.

As such, according to the present invention, when only a 'satellite image signal' is acquired in relation to the target ship, the ship has a color based on the ship patch image extracted from the satellite image signal by the second high-resolution block in the ship shape estimation model. It can be obtained by estimating the shape image.

In addition, when both the 'AIS signal' and 'satellite image signal' related to the target ship are acquired, the color and shape based on the AIS signal and the ship patch image are used by the third high-resolution block in the ship shape estimation model. It can be obtained by estimating a ship shape image having .

Specifically, when a satellite image signal capturing the target vessel is received from a satellite and the AIS signal is queried for the target vessel, the preprocessor 320 determines the two-dimensional features of the n layer for the first input signal. Combining at least a part of the quantity and at least a part of the two-dimensional feature quantity of the n-layer for the second input signal to create a two-dimensional fusion feature quantity of the n-layer (e.g., 2048*3*3) for the target ship and the acquisition processing unit 330 inputs the fused feature quantity into a third high-resolution block in the ship shape estimation model, and the object for which the shape based on the AIS signal and the color based on the satellite image signal exist together. An image of the shape of the ship can be obtained.

In this way, according to the present invention, by constructing a ship shape estimation model that can estimate the actual ship shape with only limited information using AIS signals, satellite image signals, or both, any input signal related to the target ship Even if you use , it is possible to estimate the shape of the ship through the ship shape estimation model, thereby improving the convenience of ship monitoring.

Figure 4 is a diagram illustrating a ship image obtained through a ship shape estimation model according to an input signal in the ship shape estimation system according to embodiments of the present invention.

Referring to FIG. 4, the ship shape estimation system of the present invention is related to the shape estimation of the target ship extracted from the AIS signal 410 when the AIS signal 410 transmitted by the target ship is input as an input signal. A one-dimensional first input signal that can be input to the ship shape estimation model by arranging the movement traces generated from static information such as nationality, size, ship type, and weight, and dynamic information such as position, direction, and speed of the target ship, in one dimension. By configuring the first input signal to a two-dimensional feature quantity of the n layer (e.g., 2048 * 3 * 3) and expanding the resolution, 2 of the n layer of the first input signal is obtained through the ship shape estimation model. A ship signal-based ship shape 430 having an expected shape based on dimensional features can be generated at high resolution.

In addition, the ship shape estimation system of the present invention, when a satellite image 420 of a target ship is input as the input signal, extracts the ship area detected from the satellite image 420 to a certain size, and generates 3-channel RGB The ship patch image (4 channels with NIR added) is configured as a two-dimensional second input signal that can be input to the ship shape estimation model, and the second input signal is converted into a two-dimensional feature quantity of the n layer (e.g., 2048 * 3 * 3). ), a ship patch image (satellite image)-based ship shape 440 with color expected based on the two-dimensional feature quantity of the n layer of the second input signal can be generated at high resolution through the ship shape estimation model. .

In addition, the ship shape estimation system of the present invention, when both the AIS signal 410 and the satellite image 420 related to the target ship are input as the input signals, the two-dimensional feature quantity of the n layer of the first input signal and By appropriately synthesizing the n-layer two-dimensional feature quantities of the second input signal and generating the n-layer two-dimensional fusion feature quantity, a fusion input-based ship that has both color and shape based on the fusion feature quantity through the ship shape estimation model. The shape 450 can be created with high resolution.

In this way, the ship shape estimation system of the present invention is such that when only AIS information 410 is collected for a specific ship, when only satellite image 420 is collected, both AIS information 410 and satellite image 420 are collected. Even in this case, by using the ship shape estimation model, an image that estimates the shape of the ship can be obtained with only limited information, even if there is a slight difference in quality (resolution).

Figure 5 is a diagram illustrating a ship patch image (second input signal) formed by pre-processing a satellite image signal in the ship shape estimation system according to embodiments of the present invention.

Referring to FIG. 5, the ship shape estimation system of the present invention can detect a part of a ship from a Topview satellite image 510 that captures a ship on the ground and generate a part extracted as a ship patch image 520.

Specifically, the ship shape estimation system receives satellite images 510 of marine ships on the ground from satellites, divides them into scenes, and then applies an object detection model (e.g. 'FasterRCNN' + 'ResNeSt101') to determine the area of the ship. , and apply a posture estimation model (e.g., 'Simple Pose model') to estimate the posture, such as the position, size, and direction of the detected vessel, and based on the estimated posture, size the vessel area to a certain size (e.g., 96 *96 pixels), and can generate a ship patch image (520) of 3*96*96 size expressed in 3 RGB channels (or 4 channels with NIR added).

When multiple ship patch images 520 are acquired in this way, the ship shape estimation system automatically identifies the AIS signal corresponding to each ship patch image 520, that is, the AIS signal transmitted from the ship in the ship patch image 520. By searching from the system and associating the searched AIS signal and ship patch image 520 with the unique identification number assigned to the ship, matching is performed between the ship patch image 520 and the AIS signal.

Accordingly, when the dataset (ship patch image, AIS signal, and ship image) to be used for learning the ship shape estimation model is secured, the ship shape estimation system sets the ship image as GT and determines the different values among the ship patch image and AIS signal. By performing learning using the input values, a ship shape estimation model 200 having the first to third high-resolution blocks can be constructed as shown in FIG. 2.

Figure 6 is a flowchart showing the sequence of a ship shape estimation method according to embodiments of the present invention.

The ship shape estimation method according to this embodiment can be performed by the ship shape estimation systems 100 and 300 described above.

Referring to FIG. 6, in step 610, the ship shape estimation system 100, 300 constructs a ship shape estimation model by learning a plurality of datasets prepared for ships.

In step 620, the ship shape estimation system 100, 300 searches for an AIS signal related to the target ship from the automatic ship identification system in conjunction with the generation of a shape estimation command for the target ship.

In step 630, the ship shape estimation system (100, 300) preprocesses the AIS signal to configure a first input signal that can be input to the ship shape estimation model.

In step 640, the ship shape estimation system 100, 300 pre-processes a satellite image signal of a target ship received from a satellite and configures a second input signal that can be input to the ship shape estimation model.

In step 650, the ship shape estimation system (100, 300) inputs the first input signal, the second input signal, or both to the ship shape estimation model to determine the shape of the target ship output from the ship shape estimation model. Obtain the estimated image.

In this way, according to the present invention, by constructing a ship shape estimation model that can estimate the actual ship shape with only limited information using AIS signals, satellite image signals, or both, any input signal related to the target ship Even if you use , it is possible to estimate the shape of the ship through the ship shape estimation model, thereby improving the convenience of ship monitoring.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied permanently or temporarily. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the following claims.

100, 300: Ship shape estimation system
101: Ship signal input generation unit
102: Ship patch input generation unit
103: Ship selection and input information matching unit
200: Ship shape estimation model
201: Ship signal characteristic quantity generation unit
202: Ship patch feature generation unit
203: Characteristic quantity synthesis part
204: Ship signal-based ship shape generation unit
205: Ship patch-based ship shape generation unit
206: Fusion ship shape creation unit
410: AIS information (dynamic/static)
420: Satellite image (scene unit)
430: Ship shape based on ship signals
440: Ship shape based on ship patch
450: Ship shape based on fusion input
310: Learning processing unit
320: Preprocessing unit
330: Acquisition processing unit
340: database
301: Automatic vessel identification system
302: satellite

Claims (17)

Constructing a ship shape estimation model by learning a plurality of datasets prepared about the ship;
In conjunction with the generation of a shape estimation command for the target vessel,
Retrieving AIS signals related to the target vessel from an Automatic Identification System;
Preprocessing the AIS signal to configure a first input signal that can be input to the ship shape estimation model; and
Obtaining an image estimating the shape of the target ship output from the ship shape estimation model according to the input of the first input signal.
A ship shape estimation method including. According to paragraph 1,
The ship automatic identification system records a plurality of time-series AIS signals transmitted at regular time intervals by the ship's wireless transmitter and receiver and transmitted via a repeater in association with the ship's unique identification number,
The step of checking the AIS signal is,
Among the plurality of time-series AIS signals, using the unique identification number assigned to the target vessel, searching for the AIS signal recorded in the automatic vessel identification system for a specified period of time
A ship shape estimation method including. According to paragraph 2,
The step of configuring the first input signal is,
Extracting at least one static information of the nationality, type, size, and weight of the target vessel and at least one dynamic information of the location, direction, and speed of the target vessel from the AIS signal;
Taking the dynamic information into consideration, generating a movement trajectory of the target vessel during the predetermined period of time; and
Arranging the static information and movement trajectory of the target ship to configure the first input signal expressed as a one-dimensional tensor.
A ship shape estimation method including. According to paragraph 3,
The step of constructing the ship shape estimation model is,
generating a first high-resolution block in the ship shape estimation model by learning a dataset consisting of a time-series AIS signal from the ship and a shape image of the ship;
Generating a second high-resolution block in the ship shape estimation model by learning a dataset consisting of a ship patch image extracted from a satellite image signal capturing the ship and a shape image of the ship; and
Generating a third high-resolution block in the ship shape estimation model by learning a dataset consisting of the time series AIS signal, the ship patch image, and the shape image of the ship.
A ship shape estimation method including. According to paragraph 4,
The ship shape estimation method is,
Through a deconvolution layer connected to the input terminal of the ship shape estimation model, the first input signal input to the ship shape estimation model is converted into a two-dimensional feature quantity of n layer (where n is a natural number of 3 or more). Steps to convert to
It further includes,
The step of acquiring an image estimating the shape of the target ship is,
Inputting the n-layer 2D feature for the first input signal into a first high-resolution block in the ship shape estimation model to obtain a shape image based on the AIS signal for the target ship.
A ship shape estimation method including. According to clause 5,
The ship shape estimation method is,
When a satellite image signal capturing the target vessel is received from a satellite,
Preprocessing the satellite image signal to configure a second input signal that can be input to the ship shape estimation model; and
Converting the second input signal into a two-dimensional feature of the n layer through a convolutional neural network (CNN) connected to the input terminal of the ship shape estimation model.
It further includes,
If the AIS signal is not searched for the target vessel,
The step of acquiring an image estimating the shape of the target ship is,
Inputting the n-layer two-dimensional feature quantity for the second input signal into a second high-resolution block in the ship shape estimation model to obtain a shape image based on the satellite image signal for the target ship.
A ship shape estimation method further comprising: According to clause 6,
When the AIS signal is searched for the target vessel,
The ship shape estimation method is,
Combining at least a part of the n-layer two-dimensional feature for the first input signal and at least a part of the n-layer two-dimensional feature for the second input signal to generate a fused feature for the target ship. steps to do
It further includes,
The step of acquiring an image estimating the shape of the target ship is,
Inputting the fused feature quantity into a third high-resolution block in the ship shape estimation model to obtain a shape image based on the AIS signal and the satellite image signal for the target ship.
A ship shape estimation method further comprising: According to clause 6,
The step of configuring the second input signal is,
Dividing the satellite image signal into scenes and applying a selected object detection model to detect a vessel area within the satellite image signal;
applying a selected attitude estimation model to estimate attitude information including the current position, size, and direction of movement of the vessel area within the satellite image signal; and
Based on the estimated attitude information, extracting the vessel area of a certain size from the satellite image signal and configuring the second input signal expressed as a two-dimensional signal of 3 channels RGB or 4 channels with NIR added.
A ship shape estimation method including. A learning processing unit that configures a ship shape estimation model by learning a plurality of datasets prepared about ships;
In conjunction with the generation of a shape estimation command for the target vessel,
A pre-processing unit that searches for an AIS signal related to the target ship from the automatic ship identification system, preprocesses the AIS signal, and configures a first input signal that can be input to the ship shape estimation model; and
An acquisition processing unit that acquires an image estimating the shape of the target ship output from the ship shape estimation model according to the input of the first input signal.
A ship shape estimation system including. According to clause 9,
The automatic ship identification system records a plurality of time-series AIS signals transmitted at regular time intervals by the ship's wireless transmitter and receiver and transmitted via a repeater in association with the ship's unique identification number,
The preprocessor,
Among the plurality of time-series AIS signals, use the unique identification number assigned to the target vessel to query the AIS signal recorded in the automatic vessel identification system for a set period of time.
Ship shape estimation system. According to clause 10,
The preprocessor,
Extracting at least one static information of the nationality, type, size, and weight of the target ship and at least one dynamic information of the location, direction, and speed of the target ship from the AIS signal,
Taking the dynamic information into consideration, generating a movement trajectory of the target vessel during the specified period,
Arranging the static information and movement trajectory of the target ship to configure the first input signal expressed as a one-dimensional tensor
Ship shape estimation system. According to clause 11,
The learning processing unit,
Generate a first high-resolution block in the ship shape estimation model by learning a dataset consisting of a time-series AIS signal from the ship and a shape image of the ship,
Generate a second high-resolution block in the ship shape estimation model by learning a dataset consisting of a ship patch image extracted from a satellite image signal capturing the ship and a shape image of the ship,
By learning a dataset consisting of the time series AIS signal, the ship patch image, and the shape image of the ship, a third high-resolution block in the ship shape estimation model is generated.
Ship shape estimation system. According to clause 12,
The preprocessor,
Converting the first input signal input to the ship shape estimation model into an n-layer two-dimensional feature quantity (where n is a natural number of 3 or more) through a deconvolution layer connected to the input terminal of the ship shape estimation model,
The acquisition processing unit,
Inputting the n-layer two-dimensional feature quantity for the first input signal into the first high-resolution block in the ship shape estimation model to obtain a shape image based on the AIS signal for the target ship.
Ship shape estimation system. According to clause 13,
The preprocessor,
When a satellite image signal capturing the target vessel is received from a satellite,
Preprocessing the satellite image signal to configure a second input signal that can be input to the ship shape estimation model,
Converting the second input signal into a two-dimensional feature of the n layer through a convolutional neural network connected to the input terminal of the ship shape estimation model,
If the AIS signal is not searched for the target vessel,
The acquisition processing unit,
Inputting the n-layer two-dimensional feature quantity for the second input signal into a second high-resolution block in the ship shape estimation model to obtain a shape image based on the satellite image signal for the target ship.
Ship shape estimation system. According to clause 14,
When the AIS signal is searched for the target vessel,
The preprocessor,
Combining at least a part of the two-dimensional feature quantity of the n-layer for the first input signal and at least a part of the two-dimensional feature quantity of the n-layer for the second input signal to generate a fused feature quantity for the target ship do,
The acquisition processing unit,
Inputting the fusion feature quantity into a third high-resolution block in the ship shape estimation model to obtain a shape image based on the AIS signal and the satellite image signal for the target ship.
Ship shape estimation system. According to clause 14,
The preprocessor,
Dividing the satellite image signal into scenes and applying a selected object detection model to detect the vessel area within the satellite image signal,
Applying the selected attitude estimation model, estimate attitude information including the current position, size, and direction of movement of the vessel area in the satellite image signal,
Based on the estimated attitude information, the vessel area is extracted to a certain size from the satellite image signal, and the second input signal is expressed as a two-dimensional signal of 3 channels RGB or 4 channels with NIR added.
Ship shape estimation system. A computer-readable recording medium recording a program for executing the method of claim 1.

Images