KR20210044197A - Autonomous navigation method using image segmentation - Google Patents

Abstract

The present invention relates to an autonomous navigation method using image segmentation. According to one aspect of the present invention, the autonomous navigation method using a maritime image and an artificial neural network comprises the steps of: obtaining labeling data including a plurality of labeling values; outputting output data; training the artificial neural network by using an error function in consideration of a difference between the labeling value and the output value; acquiring a maritime image; acquiring information on types of obstacles and distance information included in the maritime image; acquiring direction information of an obstacle included in the maritime image; acquiring a position on an obstacle map of the obstacle included in the maritime image; generating an obstacle map; generating a following path to be followed by the vessel; and generating a control signal so that the vessel follows the following path by using the following path. According to the present invention, surrounding environment detection is performed with an artificial neural network, which performs image segmentation to perform autonomous navigation.

Description

Autonomous navigation method using image segmentation {AUTONOMOUS NAVIGATION METHOD USING IMAGE SEGMENTATION}

The present invention relates to an autonomous navigation method using image segmentation, and more particularly, to an autonomous navigation method using an artificial neural network that performs image segmentation.

Many accidents occur in the operation of ships, and the main cause of the accident is known as negligence of man's operation. It is expected that maritime accidents can be greatly reduced when self-operated or unmanned ships that are not operated by humans and operated by themselves are commercialized. To this end, global companies are working on self-driving ship development projects. Rolls-Royce is working with Google to develop autonomous ships, and Norway's Yara and Kongsberg and Japan's Nippon Yusen are also participating.

In order to complete the autonomous ship, it is necessary to be able to detect obstacles without relying on the naked eye. Currently, various types of obstacle sensors are used, but their effectiveness is low and there are still limitations. For example, in the case of ECDIS, there are limitations due to the inaccuracy of GPS, the update period of the AIS, and the AIS unregistered mobile object, and in the case of radar, there are limitations due to the presence of a non-navigation area and noise. As a result, it is still necessary to check with the naked eye for accurate detection of obstacles, which makes it difficult to complete the autonomous ship.

An object of the present invention is to provide an autonomous navigation method capable of sensing the surrounding environment using an artificial neural network that performs image segmentation.

Another object of the present invention is to provide an autonomous navigation method using an artificial neural network that performs route planning and route tracking.

Another object of the present invention is to provide an autonomous navigation method using an end-to-end learned artificial neural network.

The problem to be solved by the present invention is not limited to the above-described problems, and the problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the present specification and the accompanying drawings. .

According to an aspect of the present invention, in a method of autonomous navigation of a ship using a maritime image and an artificial neural network, a learning image including a plurality of pixel values and a learning obstacle type information and distance information included in the learning image are determined to be reflected. Obtaining labeling data including a plurality of labeling values, the training image and the labeling data corresponding to each other; Receiving, by the artificial neural network, the training image and outputting output data, wherein the output data includes a plurality of output values, and the labeling value and the output value correspond to each other; Training the artificial neural network using an error function in consideration of a difference between the labeling value and the output value; Acquiring the sea image from a camera installed on the ship; Acquiring type information and distance information of an obstacle included in the maritime image using the artificial neural network, the obstacle included in the maritime image includes a plurality of pixel values; Acquiring direction information of an obstacle included in the sea image based on a position of the pixel value of the obstacle included in the sea image on the sea image; Acquiring a position of an obstacle included in the sea image on an obstacle map using distance information and direction information of the obstacle included in the sea image; Generating the obstacle map by using the position on the obstacle map; Generating a following path followed by the ship using the obstacle map and ship state information, the ship state information including position information and attitude information of the ship; And generating a control signal so that the ship follows the following path using the following path, the control signal including a propeller control signal and a bow direction control signal of the ship. Can be.

According to another aspect of the present invention, in a method of autonomous navigation of a ship using a maritime image and an artificial neural network, a learning image including a plurality of pixel values and a learning obstacle type information and distance information included in the learning image are determined to be reflected. Obtaining labeling data including a plurality of labeling values, the training image and the labeling data corresponding to each other; Receiving, by the artificial neural network, the training image and outputting output data, wherein the output data includes a plurality of output values, and the labeling value and the output value correspond to each other; Training a first artificial neural network using an error function in consideration of a difference between the labeling value and the output value; Learning a second artificial neural network that outputs a control signal using the learning obstacle type information and distance information; Acquiring the sea image from a camera installed on the ship; Acquiring type information and distance information of an obstacle included in the maritime image using the first artificial neural network, the obstacle included in the maritime image includes a plurality of pixel values; Acquiring direction information of an obstacle included in the sea image based on a position of the pixel value of the obstacle included in the sea image on the sea image; And generating a control signal by inputting type information, distance information, direction information, and ship state information of the obstacle included in the maritime image to the second artificial neural network.- The ship state information includes position information and attitude information of the ship. And the control signal includes a propeller control signal and a bow direction control signal of the ship.

According to another aspect of the present invention, in the autonomous navigation method of a ship using end-to-end learning, the step of learning a first artificial neural network using learning data and labeling data-The learning data includes images, location information, and Including attitude information and the labeling data includes a propeller control signal and a bow direction control signal; Acquiring a sea image from a camera installed on the ship; Acquiring ship state information including position information and attitude information of the ship; And generating a control signal using the first artificial neural network, wherein the control signal includes a propeller control signal and a bow direction control signal of the ship, wherein the first artificial neural network is configured for an obstacle. An autonomous navigation method including at least one of object information—the object information includes distance information of an obstacle—, information on an obstacle map, and information on a path followed by a ship may be provided.

The solution means of the subject of the present invention is not limited to the above-described solution means, and solutions not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings. I will be able to.

According to the present invention, it is possible to perform autonomous navigation by detecting the surrounding environment with an artificial neural network that performs image segmentation.

According to the present invention, it is possible to control a vessel through a control signal output from an artificial neural network that performs route planning and route tracking.

According to the present invention, autonomous navigation can be performed using an end-to-end learned artificial neural network.

The effects of the present invention are not limited to the above-described effects, and effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings.

1 is a block diagram of an autonomous navigation method according to an embodiment.
2 is a block diagram illustrating a learning step of an artificial neural network according to an embodiment.
3 is a block diagram of an inference step of an artificial neural network according to an embodiment.
4 is a diagram of a ship sensor system according to an embodiment.
5, 6, and 7 are diagrams illustrating examples of image segmentation according to an exemplary embodiment.
8 is a diagram for data expansion according to an embodiment.
9 is a diagram illustrating a method of learning an artificial neural network according to an embodiment.
10 and 11 are diagrams for obtaining location information using image pixels according to an exemplary embodiment.
12 is a block diagram of obtaining final location information according to an embodiment.
13, 14, and 15 are diagrams illustrating examples of obtaining final location information according to an embodiment.
16 and 17 are block diagrams illustrating examples of a path planning step according to an embodiment.
18 is a block diagram illustrating an obstacle map update step according to an embodiment.
19 is a diagram of an obstacle map according to an embodiment.
20 is a block diagram illustrating a step of converting location information according to an embodiment.
21 is a diagram of an update area according to an embodiment.
22 and 23 are diagrams for correction of location information according to an exemplary embodiment.
24 is a diagram for setting a weight in consideration of movement of an obstacle according to an embodiment.
25 is a diagram for setting weights in consideration of a navigation rule according to an embodiment.
26 is a diagram for setting a weight in consideration of a buffer area according to an embodiment.
27 and 28 are block diagrams illustrating examples of an obstacle map update step according to an embodiment.
29 is a block diagram illustrating a path generation step according to an embodiment.
30 is a diagram illustrating a path generation step according to an embodiment.
31 is a block diagram illustrating a path following step according to an embodiment.
32 is a diagram for visualization according to an embodiment.
33, 34, and 35 are block diagrams illustrating examples of autonomous navigation using image segmentation according to an embodiment.
36 is a block diagram of a learning method of an artificial neural network outputting a control signal according to an embodiment.
37 is a block diagram illustrating an output of an artificial neural network according to an embodiment.

The embodiments described in the present specification are intended to clearly explain the spirit of the present invention to those of ordinary skill in the technical field to which the present invention pertains, so the present invention is not limited by the embodiments described herein, and the present invention The scope of should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as general terms currently widely used in consideration of functions in the present invention, but this varies according to the intention, custom, or the emergence of new technologies of those of ordinary skill in the technical field to which the present invention belongs. I can. However, if a specific term is defined and used in an arbitrary meaning unlike this, the meaning of the term will be separately described. Therefore, the terms used in the present specification should be interpreted based on the actual meaning of the term and the entire contents of the present specification, not a simple name of the term.

The drawings attached to the present specification are for easy explanation of the present invention, and the shapes shown in the drawings may be exaggerated and displayed as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a well-known configuration or function related to the present invention may obscure the subject matter of the present invention, a detailed description thereof will be omitted as necessary.

According to an aspect of the present invention, in a method of autonomous navigation of a ship using a maritime image and an artificial neural network, a learning image including a plurality of pixel values and a learning obstacle type information and distance information included in the learning image are determined to be reflected. Obtaining labeling data including a plurality of labeling values, the training image and the labeling data corresponding to each other; Receiving, by the artificial neural network, the training image and outputting output data, wherein the output data includes a plurality of output values, and the labeling value and the output value correspond to each other; Training the artificial neural network using an error function in consideration of a difference between the labeling value and the output value; Acquiring the sea image from a camera installed on the ship; Acquiring type information and distance information of an obstacle included in the maritime image using the artificial neural network, the obstacle included in the maritime image includes a plurality of pixel values; Acquiring direction information of an obstacle included in the sea image based on a position of the pixel value of the obstacle included in the sea image on the sea image; Acquiring a position of an obstacle included in the sea image on an obstacle map using distance information and direction information of the obstacle included in the sea image; Generating the obstacle map by using the position on the obstacle map; Generating a following path followed by the ship using the obstacle map and ship state information, the ship state information including position information and attitude information of the ship; And generating a control signal so that the ship follows the following path using the following path, the control signal including a propeller control signal and a bow direction control signal of the ship. Can be.

Here, the labeling value and the output value may be selected from a plurality of identification values determined by a combination of the type information and distance information of the obstacle.

Here, the distance information of the obstacle for determining the identification value may include a plurality of categories each having a distance range.

Here, the control signal may be generated based on at least one of a control signal of a previous frame, current, and wind.

Here, the autonomous navigation method further includes the step of learning a second artificial neural network that receives a tracking path and outputs a control signal through reinforcement learning, wherein the control signal is based on the control signal of the previous frame. 2 Can be generated from artificial neural networks.

Here, the step of generating the obstacle map may further use information on the movement of the obstacle included in the marine image.

Here, the step of obtaining the marine image may include obtaining an image from the camera; And generating a marine image by pre-processing the image, wherein the pre-processing may be characterized in that the fog included in the image is removed.

According to another aspect of the present invention, in a method of autonomous navigation of a ship using a maritime image and an artificial neural network, a learning image including a plurality of pixel values and a learning obstacle type information and distance information included in the learning image are determined to be reflected. Obtaining labeling data including a plurality of labeling values, the training image and the labeling data corresponding to each other; Receiving, by the artificial neural network, the training image and outputting output data, wherein the output data includes a plurality of output values, and the labeling value and the output value correspond to each other; Training a first artificial neural network using an error function in consideration of a difference between the labeling value and the output value; Learning a second artificial neural network that outputs a control signal using the learning obstacle type information and distance information; Acquiring the sea image from a camera installed on the ship; Acquiring type information and distance information of an obstacle included in the maritime image using the first artificial neural network, the obstacle included in the maritime image includes a plurality of pixel values; Acquiring direction information of an obstacle included in the sea image based on a position of the pixel value of the obstacle included in the sea image on the sea image; And generating a control signal by inputting type information, distance information, direction information, and ship state information of the obstacle included in the maritime image to the second artificial neural network.- The ship state information includes position information and attitude information of the ship. And the control signal includes a propeller control signal and a bow direction control signal of the ship.

According to another aspect of the present invention, in the autonomous navigation method of a ship using end-to-end learning, the step of learning a first artificial neural network using learning data and labeling data-The learning data includes images, location information, and Including attitude information and the labeling data includes a propeller control signal and a bow direction control signal; Acquiring a sea image from a camera installed on the ship; Acquiring ship state information including position information and attitude information of the ship; And generating a control signal using the first artificial neural network, wherein the control signal includes a propeller control signal and a bow direction control signal of the ship, wherein the first artificial neural network is configured for an obstacle. An autonomous navigation method including at least one of object information—the object information includes distance information of an obstacle—, information on an obstacle map, and information on a path followed by a ship may be provided.

Here, using a second artificial neural network, obtaining at least one of the object information, information on the obstacle map, and information on a path followed by the ship from the first artificial neural network.

Hereinafter, an autonomous navigation method using image segmentation according to an exemplary embodiment will be described.

For autonomous driving of a moving object according to an exemplary embodiment, the moving object may perform steps of sensing the surrounding environment, planning a route, and following a route. The moving object may determine an obstacle and/or a drivable area in the vicinity through sensing the surrounding environment, generate a route based on this, and then follow the generated route to drive by itself.

1 is a block diagram of an autonomous driving method according to an embodiment. Referring to FIG. 1, the moving object acquires object information through the surrounding environment detection step (S1000), acquires a following path through the path planning step (S3000) based on the acquired object information, and obtains a following path based on the acquired following path. As a result, a control signal may be generated through the path following step (S5000) to allow autonomous driving.

For autonomous driving of the moving object, the above-described steps may be continuously performed. Hereinafter, the detection of the surrounding environment (S1000), the path planning step (S3000), and the path following step (S5000) are performed once and expressed as one frame.

In addition, the following mainly describes the autonomous operation of the ship during autonomous driving of the mobile body, but it is not limited to the ship, and it is revealed in advance that it can be applied to various mobile bodies such as automobiles and drones.

Autonomous navigation can be performed using an artificial neural network. An artificial neural network is a type of algorithm modeled after the structure of a human neural network, and can include one or more nodes or one or more layers including neurons, and each node can be connected through a synapse. Data (input data) input to the artificial neural network may be output (output data) through a node through a synapse, and information may be obtained through this.

Types of artificial neural networks include a convolution neural network (CNN) that extracts features using a filter, and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back to the input. In addition, there are various types such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), and relational networks (RN).

Before using the artificial neural network, a step of learning is necessary. Alternatively, it can be learned by using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and a step of using the artificial neural network as a reasoning step.

There are various methods of learning artificial neural networks such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning. Here, reinforcement learning may be expressed as a Markov decision process. Alternatively, reinforcement learning may refer to a method in which the agent behaves so that the reward is maximized in an environment.

2 is a block diagram illustrating supervised learning in an example of a learning step of an artificial neural network according to an embodiment. Referring to FIG. 2, an artificial neural network that has not been trained receives training data or training data, outputs output data, compares the output data and labeling data, and backpropagates the error. Neural networks can be trained. The training data may include an image, and the labeling data may include ground truth. Alternatively, the labeling data may be data generated through a user or a program.

3 is a block diagram of an inference step of an artificial neural network according to an embodiment. Referring to FIG. 3, the learned artificial neural network may receive input data and output output data. Information that can be inferred in the inference stage may vary according to the information of the learning data in the learning stage. In addition, the accuracy of the output data may vary depending on the learning degree of the artificial neural network.

In the case of autonomous navigation using an artificial neural network, referring to FIG. 1, the surrounding environment detection step (S1000) is performed with an artificial neural network, the route planning step (S3000) is performed with an artificial neural network, or a path following step (S5000) is performed. This can be done with an artificial neural network. Alternatively, two or more of the above-described steps may be performed with an artificial neural network, or one step may be performed with a plurality of artificial neural networks. When using a plurality of artificial neural networks, the artificial neural networks may be connected in parallel or in series with each other. For example, when artificial neural networks are connected in series, an output of one artificial neural network may be an input of another artificial neural network.

Hereinafter, the surrounding environment sensing step, the path planning step, and the path following step will be described in detail. In order to implement the autonomous navigation method, it is not necessary to perform all the steps to be described later, but only some of them may be performed, and certain steps may be performed repeatedly.

In order to operate a ship, obstacles and/or an operable area must be identified through the detection of the surrounding environment, and this is the same in the case of autonomous navigation. Here, the obstacle refers to all objects that may become obstacles during navigation, such as terrain, buildings, ships, buoys, and people. In addition, sensing the surrounding environment is a comprehensive meaning that includes not only detecting obstacles, but also acquiring information about the situation around the ship. Detecting an obstacle includes acquiring information about the presence, type, and location of the obstacle.

4 is a diagram of a ship sensor system according to an embodiment. Referring to FIG. 4, it is common to use an obstacle detection sensor such as a radar, a lidar, and an ultrasonic detector, and an automatic identification system (AIS) when a ship is operated.

In addition to this, obstacles can be detected using a camera. Referring to FIG. 4, examples of the camera include a monocular camera, a binocular camera, an infrared camera, and a TOF camera, but are not limited thereto.

Image segmentation can be used as one of the methods of detecting obstacles with images acquired from a camera. Hereinafter, it will be described in detail with respect to performing the step of detecting the surrounding environment using image segmentation.

Image segmentation may mean dividing an image area for each attribute. Alternatively, it may be a process of allocating an attribute value for each pixel of the image. For example, the attribute may be the type of object. In this case, image segmentation may mean dividing an object included in the image for each pixel. Alternatively, it may mean indicating which object a specific pixel corresponds to.

Image segmentation can be performed using an artificial neural network. One artificial neural network may be used, and each artificial neural network may perform image segmentation using a plurality of artificial neural networks, and the results may be combined to sense the surrounding environment. The network structure of the artificial neural network for image segmentation can be applied to various structures such as the ENet structure.

Hereinafter, a case in which the sensing of the surrounding environment (S1000) of FIG. 1 is performed through image segmentation using an artificial neural network will be described in detail.

An artificial neural network that performs image segmentation to detect the surrounding environment may receive an image and output object information. 2 and 3, the training data and the input data may be in the form of an image, and the image may include a plurality of pixels. Output data and labeling data may be object information. Additionally, by visualizing the output data and labeling data, information can be visually delivered to the user.

There is no limit to the type of image for performing image segmentation. The image may be an image taken with a camera. Referring to FIG. 4, images acquired from various cameras, such as a monocular camera, a binocular camera, an infrared camera, and a TOF camera, may be used. In addition, it is not limited to only two-dimensional images, but three-dimensional images and the like are also possible.

When image segmentation is performed once, only one image can be input. Alternatively, a plurality of images may be input.

After pre-processing the image captured by the camera, it can be input into the artificial neural network. Here, pre-processing refers to all kinds of processing performed on the captured image, and image normalization, image resize, cropping, noise removal, and fogging included in the image, It may include removing fine dust, removing salt, removing water droplets, and combinations thereof. Looking at the fog removal as an example of the pre-processing, the fog removal may mean converting an image of a foggy area into an image of a clear area through pre-processing. Looking at normalization as another example of preprocessing, normalization may mean obtaining an average of RGB values of all pixels of an RGB image and subtracting this from the RGB image. Looking at the removal of water droplets as another example of pre-processing, water droplet removal refers to removing water droplets through pre-treatment from images captured such as water droplets condensed on the front of the camera, or removing rainwater from the image through pre-processing on a rainy day. can do. Through such image pre-processing, the performance/accuracy of the artificial neural network can be improved.

In the above, we have looked at the method of preprocessing the image and then inputting it into the artificial neural network. Alternatively, artificial neural networks including pre-processing can be learned and used. For example, even if an image of a foggy area is taken as an input of the artificial neural network, the artificial neural network can be trained to obtain the same output as when an image of a clear area is taken as an input.

The output data/labeling data of the artificial neural network may correspond to the input image/learning image. Also, the output data and labeling data may correspond to each other.

The output data and labeling data may include a plurality of pieces of information. Alternatively, the output data and labeling data may be determined by reflecting a plurality of pieces of information.

The output data and labeling data may each include a plurality of output values and a plurality of labeling values. The output value and the labeling value may include a plurality of pieces of information. Alternatively, the output value and the labeling value may be determined by reflecting a plurality of pieces of information.

The output value and the labeling value may correspond to pixels of an image input to the artificial neural network. In addition, the output value and the labeling value may correspond to each other.

The number of output values and labeling values may be the same as the number of pixels in the input image. For example, when an image of 256 x 256 pixels is input, the number of output values and labeling values may also be 256 x 256 = 65536. Alternatively, the number of output values and labeling values may be an integer multiple of the number of input image pixels. Hereinafter, a case where the number of pixels of the input image and the number of output values and labeling values are the same is mainly described, but the present invention is not limited thereto, and the number of pixels, the output value, and the number of labeling values may be different.

The output data and labeling data may be object information. Here, the object information refers to information on the property of an object included in the image, and there is no limitation if it is information on the object.

The object may include not only obstacles such as terrain, buildings, ships, buoys, people, etc., but also an area that the ship can navigate, such as the ocean, and an area that may not be related to the operation of the ship, such as sky. The information is not limited in the presence or absence of an object, its type, location, movement direction, movement speed, and route marker information. Here, the route sign information may include a side sign, a bearing sign, an isolated obstacle sign, a safe zone sign, a special sign, and other signs. In addition, the location of the object may include a distance and a direction to the object, and may be a relative location or an absolute location. The object information may include only a part of the object and the information, or may include all of the object. For example, information on an obstacle among objects may be referred to as obstacle information.

Hereinafter, location information among object information will be described in detail.

When location information is learned in the learning stage of the artificial neural network, location information can be inferred. Here, the location information means information related to the location of the object. For example, the location information may be a distance and/or a direction. Alternatively, it may be a coordinate in which the object exists.

The distance information of the object may be a distance from the camera that captured the image to the object. Alternatively, it may be a distance from an arbitrary point to an object. The direction information of the object may be an angle between the object and the camera that captured the image. Alternatively, it may be an angle between an arbitrary point and an object.

The location information may refer to a relative position of an object or may refer to an absolute position.

Position information to an object included in the image may be obtained through image segmentation. Alternatively, location information for each pixel or pixel group of the image may be obtained. 5 is a diagram illustrating image segmentation according to an exemplary embodiment. Referring to FIG. 5, through image segmentation, output data 130 obtained by dividing an input image 110 according to a specific attribute may be obtained. The input image 110 includes an object including two ships, sea, and land, and by inputting the input image 110 to an artificial neural network, the object is divided according to the type and distance to obtain the output data 130 can do. For example, in the output data 130, two ships having different distance information may be expressed differently.

5 is an example of a method of expressing location information through image segmentation. In addition to this, location information can be expressed in various ways. The location information may include only distance information, or other object information such as direction information, existence of an object, and type of an object in addition to the distance information.

6 and 7 are diagrams of image segmentation according to an exemplary embodiment. Hereinafter, exemplary embodiments for expressing location information will be described with reference to FIGS. 6 and 7, but a method of expressing the location information is not limited thereto.

The location information may be expressed in a plurality of categories having a certain range. For example, distance information may be expressed in a short distance, a medium distance, a far distance, and the like, and the direction information may be expressed in a left direction, a front direction, a right direction, and the like. It will be possible to combine these and express them in the near-left or far-right distance.

The category can be expressed using numbers. For example, of the distance information, a short distance may be expressed as -1, a medium distance may be expressed as 0, and a long distance may be expressed as +1. Referring to FIG. 6, when an image of 256 x 256 pixels is used as the training image and the input image 110, output data and labeling data may be a matrix having a size of 256 x 256, and each element of the matrix corresponds to distance information. Accordingly, it may have any one of -1, 0, and +1.

The location information may be expressed as an actual distance value and a direction value. For example, distance information may be expressed in meters (m), and direction information may be expressed in degrees. Referring to FIG. 6, each element of a matrix of output data and labeling data representing distance information may have a distance value in units of meters, such as 10 m and 15 m.

Position information can be expressed by converting actual distance values and direction values into values in a certain range. The location information may be uniformly normalized, or may be non-uniformly normalized using a log function or an exponential function. For example, distance information may be expressed by normalizing to a value between 0 and 1. Referring to FIG. 6, each element of an output data representing distance information and a labeling data matrix may have a value between 0 and 1.

Position information can be expressed as an RGB value. In this case, the output data and labeling data become RGB images, so that the user can easily check the location information without a separate visualization step. Referring to FIG. 7, ships and terrain included in the labeling data and the output data 130 may correspond to different RGB values due to different distance information and may be represented in different colors. In comparison with FIG. 6, in terms of a matrix, the labeling data and the output data 130 may be a matrix having a size of 256 x 256 x 3.

Location information and other object information can be expressed together. In the following, object information other than location information is referred to as additional information.

Additional information can be expressed independently of location information. Alternatively, location information and additional information can be simultaneously reflected and expressed.

When the location information and the additional information are independently expressed, the location information and the additional information may be expressed as separate data sets. For example, when an image of 256 x 256 pixels is input to the artificial neural network, location information may be a matrix having a size of 256 x 256 and additional information may be n matrices having a size of 256 x 256. Here, n may be the number of additional information.

When the location information and the additional information are simultaneously reflected and expressed, the location information and the additional information may be expressed as one data set. For example, the method of expressing the above-described position information in a plurality of categories having a predetermined range may be expressed by considering the type of obstacle together. Table 1 is a representation method in which distance information and an obstacle type are considered together according to an embodiment. Referring to Table 1, a class may be set in consideration of distance information and an obstacle type together, and an identification value may be allocated for each class. For example, the identification value 1 may be assigned in consideration of a short distance, which is distance information, and a terrain, which is obstacle type information, together. Referring to Tables 1 and 6, each element of the output data and labeling data matrix may have any one of 1 to 10 identification values. Table 1 is an example of a case in which distance information and obstacle type information are considered together, and other additional information such as direction information, obstacle movement direction, speed, and route indicator may also be considered. In addition, not all identification values must include a plurality of pieces of information, and not all of the identification values must include the same type of information. For example, a specific identification value includes only distance information, another identification value includes only obstacle type information, and another identification value includes both distance information and obstacle type information.

Identification value class One Close range + terrain 2 Medium range + terrain 3 Ranged + Terrain 4 Close range + fixed obstacle 5 Medium range + fixed obstacle 6 Ranged + Fixed Obstacles 7 Close range + dynamic obstacles 8 Medium range + dynamic obstacles 9 Ranged + dynamic obstacles 10 Etc

The learning process is important for the accuracy of artificial neural networks. Artificial neural networks can be learned in a variety of ways without any restrictions on methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

In order to improve the accuracy of the artificial neural network, it may be necessary to secure various learning images. It is possible to respond to various input images through learning of the artificial neural network using various learning images. For example, a misty training image may be needed to obtain information from a misty image through image segmentation. Alternatively, it may be necessary to use an image of a rainy day as a learning image in order to obtain an accurate image segmentation result with an image of a rainy day.

In the case of maritime images, it may be difficult to secure a learning image due to problems such as having to take a picture while boarding a ship. In particular, it may be more difficult because various maritime situations must be considered in order to improve the accuracy of the artificial neural network. Here, as a maritime situation, there may be sea fog, fog, sea clutter, snow, rain, and the like.

In addition to the method of acquiring a learning image by going out on a ship, it is possible to acquire a learning image through data augmentation. Here, data expansion refers to all methods other than a method of directly capturing and obtaining a training image. As an example, fog may be added to an image of a sunny day using a program or the like to secure an image of a foggy day as a learning image. As another example, the color of the sea may be arbitrarily changed to generate a learning image that considers different colors of the sea according to regions. 8 is a diagram for data expansion according to an embodiment. Referring to FIG. 8, an image of a misty day, an image of a rainy day, and an image of a misty and rainy day can be obtained from the maritime image on a clear day.

There are cases in which labeling data corresponding to the training image is required according to the learning method of the artificial neural network. For example, for supervised learning, labeling data may be required. Labeling data can be generated from the training image.

The labeling data may include various types of information according to the purpose of training the artificial neural network. For example, the labeling data may include location information corresponding to each pixel and/or pixel group of the image, or may include location information corresponding to each area of the image. Alternatively, it may include location information of an object included in the image. Since the method of expressing the labeling data may vary according to the method of expressing the above-described location information, it may be necessary to generate the labeling data in consideration of this.

The acquisition of labeling data including distance information will be described in more detail. Distance information may be obtained using a depth camera. Here, the depth camera may be a stereo type, a structured pattern type, a time-of-flight method (TOF type), or the like, and two or more of them may be mixed and used. One labeling data may be generated by obtaining distance information for each pixel of the image from the depth camera. In addition to this, various methods for obtaining distance information may be used.

Labeling data may be generated to further include additional information in addition to distance information. In the case of generating labeling data independently including distance information and additional information, the above-described method can be used as it is. When the distance information and the additional information are considered together and expressed as a single value, as shown in Table 1, the modified data in which the distance information acquired through a depth camera and the additional information are considered together may be the labeling data.

Labeling data can be created manually by the user. Alternatively, the labeling data may be generated through an artificial neural network or other algorithm that generates it. For example, the output of the first artificial neural network that has already been learned may be used as labeling data of the second artificial neural network to be trained. 9 is a diagram illustrating a method of training a second artificial neural network using an output of a first artificial neural network as labeling data according to an exemplary embodiment. Referring to FIG. 9, there may be a learned first artificial neural network that receives a visible ray image and performs image segmentation, and an untrained second artificial neural network that receives an infrared image and performs image segmentation. When a visible ray image and an infrared image are acquired for the same scene and inputted to the first artificial neural network and the second artificial neural network, respectively, the output of the first artificial neural network is an accurate segmentation result, and the output of the second artificial neural network is an inaccurate result. I can. The second artificial neural network can be trained by comparing the output of the first artificial neural network with the output of the second artificial neural network by using the output of the first artificial neural network as labeling data of the second artificial neural network and backpropagating the error to the second artificial neural network.

The automatically generated information can be corrected later and used as labeling data. For example, after correcting distance information for each pixel acquired from a depth camera, labeling data may be generated based on the corrected data. Here, the correction of information may mean more accurate values. For example, correction of distance information may mean more accurate distance values.

Labeling data can be created by intentionally distorting information. For example, distance information obtained from a depth camera may be intentionally distorted and reflected in labeling data. Small obstacles such as buoys can be labeled larger than their actual size. Due to this, the artificial neural network can be learned to detect small obstacles well.

Another method for improving the accuracy of the artificial neural network may be to adopt an appropriate error backpropagation method. The error (loss) is a measure that can indicate the accuracy of the artificial neural network, and may mean the difference between the output data of the artificial neural network and the labeling data.

The total error of the output data may be calculated using the error per output value of the output data. Alternatively, the total error of the output data may be expressed as a function of the error per output value of the output data. The error can be defined in various ways, such as a mean squared error and a cross entropy error. For example, the total error may be a sum of errors per output value. Alternatively, the total error may be calculated by considering a constant value (weight) to the error per output value. For example, the total error may be calculated as a sum of a value obtained by multiplying the error per output value by a weight.

The total error can be calculated by reflecting the information included in the output data. For example, the total error may be calculated by reflecting the type information of the object included in the output data. Equation 1 below is an example of the total error calculated by reflecting the object type information, as shown in Table 1.

Equation 1

이고, P_class는 객체를 구성하는 픽셀의 개수, c는 하이퍼 파라미터(hyper parameter), E_c는 식별값이다.here,

, P _class is the number of pixels constituting an object, c is a hyper parameter, and E _c is an identification value.

When calculating the total error, the error of all output values can be considered. Alternatively, when calculating the total error, only errors of some output values may be considered and errors of other output values may not be considered. For example, when the output value includes information on the type of object, only errors of some objects may be considered and errors of other objects may not be considered. Here, the object to be considered may be an object necessary for navigation, and an object that is not considered may be an object that is not required for navigation. In addition, an object required for navigation may be an obstacle, and an object not required for navigation may be a sky.

As another method for improving the accuracy of the artificial neural network, different artificial neural networks can be used depending on the conditions. For example, the artificial neural network used can be different depending on the color of the sea. Alternatively, the artificial neural network may be different depending on the region. The selection of the artificial neural network can be made manually or automatically by the user. For example, after detecting an area using GPS, the artificial neural network can be automatically changed according to the area. Alternatively, the artificial neural network can be automatically changed by detecting the color of the sea with a sensor or the like.

Cameras can be installed on the ship to acquire learning images/input images. The vessel may be an operating vessel or an autonomously operating vessel. Alternatively, the vessel may be a vessel other than an operating vessel or an autonomously operating vessel. In addition, the camera may be installed in a port, land, etc., in addition to a ship, without limitation.

When the camera is installed on the ship, there is no restriction on the location where the camera is installed. The camera can be installed in any position such as the front, side, and rear of the ship. In addition, the camera may be installed toward the front of the ship, and may be installed toward the side or rear.

The detection range may vary depending on the camera. For example, the distance and/or angle (field of view, FOV) that can be detected by the camera may vary according to the lens and/or focal length of the camera. The detection range of the camera may vary depending on the size and purpose of the ship. For example, in the case of a large ship, a camera having a longer sensing distance than a small ship may be installed.

A single camera can be installed on the ship. Alternatively, a plurality of cameras may be installed. As a plurality of cameras are used, the detection range may increase compared to the case of using a single camera. For example, the FOV may be wider or the detection area may be increased. Alternatively, as a plurality of cameras are used, the accuracy of sensing the surrounding environment may be improved.

A plurality of cameras may be installed at different locations. For example, the first camera may be installed at the front of the ship and the second camera may be installed at the rear of the ship. Alternatively, a plurality of cameras may be installed at different heights. For example, a second camera may be installed above the first camera.

The detection areas of the plurality of cameras may be different from each other. For this reason, it can be more effective in detecting the surrounding environment than when a single camera is installed. For example, the first camera may detect an area near the ship, and the second camera may detect an area far from the ship. As a specific example, a first camera that detects a short distance (for example, 0 to 150 m) with a wide-angle lens and a second camera that detects a distance (for example, 150 to 300 m) with a zoom lens are used together. If you do, both near and far (for example, 0 to 300m) can be detected. As another example, the first camera may detect the area in front of the ship, the second camera may detect the area lateral to the ship, and the third camera may detect the area behind the ship. As another example, the first camera may detect the left front area of the ship, and the second camera may detect the right front area of the ship. The detection ranges of the plurality of cameras may partially overlap.

In the case of using a plurality of cameras, an image generated by each of the plurality of cameras may be matched into one image and then input to an artificial neural network to perform image segmentation. Alternatively, image segmentation may be performed after some of the images generated by each of the plurality of cameras are matched.

Alternatively, image segmentation may be performed by individually inputting images generated by each of the plurality of cameras into the artificial neural network. For example, image segmentation that expresses distance information in short, medium, and long distances, for example, is a first image generated by a first camera that detects a distance of 0 to 150m and a second image that is generated by a second camera that detects a distance of 150 to 300m. When performing image segmentation with the first image, a short distance (eg, 0 to 50 m), a medium distance (eg, 50 to 100 m) and a far distance (eg, 100 to 150 m) results are obtained from the first image, and the second An image segmentation result that obtains short distance (e.g., 150-200m), medium-distance (e.g., 200-250m), and long-distance (e.g., 250-300m) results from an image and calculates a total distance range of 6 steps. Can be obtained.

The number of cameras installed may vary depending on the size and purpose of the ship. For example, in the case of a large ship, a larger number of cameras may be installed than a small ship.

When installing a camera on a ship, initial setting may be required for various reasons, such as the type of ship and the location of the camera. Here, the initial setting may mean setting so that the result of image segmentation using pixels or artificial neural networks appearing in the camera image/image is normal.

As a method of initial setting, after giving the user an appropriate installation guide, the module can determine the camera's position and orientation by comparing the result value and the error of the AIS/Radar.

Alternatively, after the camera is installed, a setting value of the camera may be calculated in a direction that minimizes an error according to the result of the test run, and thereafter, it may be used to detect the surrounding environment. In particular, it is possible to find a setting value in a direction that minimizes an error based on a correlation between the topographic features of the electronic chart and the result value of image segmentation. If the result value found after that is, for example, if the camera is installed too downward and cannot see a sufficient distance, a warning can be given and re-installation can be induced.

A processing method when an image captured by a camera is used as an input image for image segmentation will be described.

The camera can take images at regular intervals. The time interval may be fixed or may be dynamically changed.

The image acquisition period of the camera and the operation period of image segmentation may be different. For example, image segmentation may be performed on only some of the images acquired through the camera. When the camera acquires an image of 60 frames per second, image segmentation may be performed based on an image of 10 frames. As a method of selecting some images, a method of selecting an arbitrary image may be used. Alternatively, you can select a clear image from among the captured images. Alternatively, images of a plurality of frames may be synthesized and converted into one image, and then image segmentation may be performed. For example, segmentation can be performed with an image obtained by combining a high-light image and a low-light image.

One image may be used to perform one image segmentation, but a plurality of images may be used. Here, the plurality of images may be images captured by the same camera or different cameras.

When image segmentation is performed using images captured by different cameras, the detection ranges of each camera may be different. For example, a first camera may capture an image at a short distance and a second camera may capture an image at a distance.

In addition to image segmentation, the surrounding environment can be detected in other ways. For example, the surrounding environment may be sensed using an obstacle detection sensor such as a radar, a lidar, or an ultrasonic detector, or information obtained by the obstacle detection sensor may be processed through an artificial neural network to detect the surrounding environment. Hereinafter, the location information obtained as a result of image segmentation is referred to as first location information, and the location information obtained by other methods is referred to as second location information.

In the case of an image captured by a camera, position information per pixel of the image may be obtained using setting information including the type and position (height, angle, etc.) of the camera. The acquired location information may be a location of a camera or a location relative to an arbitrary reference, or may be an absolute location.

10 is a diagram for obtaining location information using image pixels according to an exemplary embodiment. Referring to FIG. 10, a left-right direction of an image may correspond to an angle of a pixel, and an up-down direction may correspond to a distance to a pixel. In the case of Pixel A, it can be seen that the distance is about 30° to the left, and the distance is about 15m, and in the case of Pixel B, it can be seen as about 10° to the right and the distance is about 20m.

Depending on the camera's posture, an error in the above-described location information may occur. For example, the initial installation posture of the camera may be changed later, resulting in an error. The location information may be corrected by reflecting the installation posture of the camera.

An error in location information may occur depending on the movement (position) of the object on which the camera is installed. For example, when a camera is installed on a ship, position information per pixel of an image may vary according to a roll, pitch, etc. of the ship. In order to obtain accurate location information, it is possible to consider the attitude of an object on which a camera is installed, such as the attitude of a ship. For example, information about the ship's attitude, such as heading or yaw, pitch, roll, etc., is acquired from the inertial measurement unit (IMU) mounted on the ship, and the position per pixel of the image is reflected. Information can be obtained. 11 is a diagram for obtaining location information reflecting the attitude of a ship according to an embodiment. In FIG. 11, only the pitch of the ship was considered. When comparing the case where the ship's bow is looking upward and downward, the corresponding distance may be different even at the same position on the image. Referring to FIG. 11, both the pixel A and the pixel B may be different from the center of the image, but the distance between the pixel A and the pixel B is 40 m and the distance between the pixel B and the pixel B are 10 m. Therefore, distance information may be inaccurate if the pitch of the ship is not considered through an inertial measurement device or the like.

The surrounding environment can be detected by combining methods other than image segmentation and image segmentation. Alternatively, information acquired through image segmentation may be corrected through a method other than image segmentation. Here, the meaning of correction is not limited to the meaning of obtaining absolutely accurate information, but includes the meaning that the information changes.

Hereinafter, the fusion of the first location information and the second location information will be described in detail.

12 is a block diagram of obtaining final location information according to an embodiment. Referring to FIG. 12, after acquiring first location information through image segmentation S1500 and acquiring second location information by other methods, final location information may be obtained by taking this into account.

It is not necessary to fuse the location information for all the output values of the image segmentation output data. Some output values may obtain final location information using first location information and second location information, and other output values may obtain final location information using only one of the first location information and second location information. For example, it is possible to determine whether to fusion of location information according to the type of an object obtained through image segmentation. Location information may be fused with respect to obstacles, and location information may not be fused with a navigable area such as the sea.

The types of the first location information and the second location information may be different. For example, the first location information may contain distance information, and the second location information may contain only direction information. In this case, final location information may be obtained by adding the first location information and the second location information.

The first location information and the second location information may include the same type of information. In this case, the first location information and the second location information may or may not match.

The case where the location information matches may be a case where the first location information and the second location information are the same or similar, or within an error range. For example, the first location information may be expressed as a distance area, and the second location information may be included in the distance area of the first location information. The final location information may be one of first location information and second location information. Alternatively, the final location information may be a value obtained by correcting the first location information with the second location information.

13 is a diagram for obtaining final location information when location information is matched according to an embodiment. The first location information is expressed as a distance area, and if the output value is -1, it means a distance of less than 10m, if it is 0, it means a distance of 10m or more and less than 20m, and if it is 1, it means a distance of more than 20m, and the second location information is in units of m. It is expressed as the distance value of. The final location information is expressed as a distance area by reflecting the first location information and the second location information. For example, the first location information may determine a maximum value of the final location information, and the second location information may determine a minimum value of the final location information. Looking at the central output value, the first location information is 0, the second location information is 14m, the maximum value is 20m is determined by the first location information, the minimum value is 14m is determined by the second location information, and the final location information is It may be in the range of 14 to 20 m.

As another example, there may be a case where both the first location information and the second location information are expressed as a distance value, and the first location information and the second location information match. In this case, the final location information may be any one of first location information and second location information. Alternatively, the final location information may be an average of the first location information and the second location information. In addition, various methods such as mixing the first location information and the second location information at a specific ratio may be used.

When the location information does not match, the first location information and the second location information may be different or outside the error range. In this case, the final location information may be any one of first location information and second location information. For example, the final location information may be a smaller value of the first location information and the second location information. Alternatively, for an output value in which the location information is mismatched, the location information of the corresponding output value may be set as an unknown value. Alternatively, if there is an output value in which the location information is inconsistent, the location information may be set as an unknown value for all output values of the corresponding output data.

14 is a diagram for obtaining final location information when location information is mismatched according to an embodiment. 14 shows first location information and second location information in the same manner as in FIG. 13. Referring to FIG. 14, the highlighted output value on the upper left is mismatched, and the first location information is prioritized and selected as the final location information.

When performing image segmentation, the second location information may be used as input data along with the image. 15 is a diagram for obtaining final location information according to an embodiment. Referring to FIG. 15, final location information may be obtained using an image and second location information as an input of the image segmentation S1500. When the image segmentation S1500 is performed using an artificial neural network, the second location information may be input at the same stage as the image or at another stage. For example, when the artificial neural network includes a plurality of sequential layers, an image may be input from a first layer, and second location information may be input from another layer. As a result, operations for all layers may be performed for an image, and operations for one or more layers may be excluded for the second location information.

In the above, the case of using one first location information and one second location information to obtain one final location information has been described, but is not limited thereto, and a plurality of first location information is used to obtain one final location information. And/or a plurality of second location information may be used. In addition, the description is made on the assumption that the second location information is expressed as an actual distance value, but the present disclosure is not limited thereto.

The path planning step may refer to a step of receiving object information and outputting a path (hereinafter referred to as a'following path') that the ship should follow. 16 is a block diagram of a path planning step according to an embodiment. Referring to FIG. 16, in addition to object information, it is possible to output a following route through a route planning step (S3000) by receiving input of a starting point, an arrival point of a vessel, vessel condition information, weather environment information, and navigation rules.

The starting point and the arrival point may be the initial departure position and the final arrival position of the ship, but are not limited thereto. For example, the starting point and the ending point may be any location on the sea. Alternatively, the starting point may be a current position of the ship, and the arrival point may be a final arrival position. Alternatively, the starting point may be the current position of the ship, and the arrival point may be an arbitrary position on the obstacle map.

The ship status information means information about the ship, such as the ship's position, speed, and attitude information. Here, the attitude information means information about the attitude of the ship, such as the bow direction, pitch, and roll.

The meteorological environment information refers to information related to the meteorological environment such as tide, wind, and waves.

Operation rules refer to promises or laws and regulations that must be observed when operating a ship. For example, the operating rules may be international regulations for preventing collisions at sea (COLREG).

The following path is a path that the ship follows, and may be a waypoint indicating a position through which the ship passes. Alternatively, it may be a path through which the ship passes.

It is possible to create a following route by reflecting the information of the route indication. Here, the route indication may be obtained through sensing the surrounding environment such as image segmentation.

The above-described input information is an example, and the information received in the route planning step may include all of the above-described information, only a part of the information, or other information.

The path planning step can be implemented through a single algorithm or artificial neural network. For example, when object information is input, an artificial neural network that outputs a tracking path can be trained and used. Alternatively, a plurality of algorithms or artificial neural networks may be used.

The path planning step may include an obstacle map update step and a path generation step. 17 is a block diagram of a path planning step according to an embodiment. The obstacle map update step S3500 may receive object information and output an obstacle map, and the path generation step S3700 may receive an obstacle map and output a following path.

18 is a block diagram illustrating an obstacle map update step according to an embodiment. Referring to FIG. 18, an obstacle map may be updated in consideration of weather environment information such as object information, tide, wind, etc. and navigation rules.

The object information is not limited to that obtained through image segmentation, and object information obtained through other sensors such as radar and lidar may also be input to the obstacle map update step. It is also possible for some or all of these to be combined.

The obstacle map refers to a means of expressing object information. For example, the obstacle map may be a grid map. The grid map may divide a space into unit areas and display object information for each unit area. As another example, the obstacle map may be a vector map. Obstacle maps are not limited to two-dimensional, but three-dimensional obstacle maps are also possible. Meanwhile, the obstacle map may be a global map expressing all areas related to the operation of the ship from the starting point to the arrival point or a local map expressing a certain area around the ship.

The obstacle map may include a plurality of unit areas. The plurality of unit areas may be expressed in various ways according to classification criteria. For example, the obstacle map may include a navigable area, a navigable area, and an unconfirmed area in which navigation is impossible. Specifically, the operable area may be an area where there is no obstacle or an area corresponding to an area where there is a low probability of an obstacle, and the non-operable area is an area where an obstacle exists or an area corresponding to an area where there is a high probability of an obstacle. Can be Alternatively, the operable area may be an area corresponding to an area suitable for navigation, and the non-operable area may be an area corresponding to an area unsuitable for navigation. The unconfirmed area may be an area excluding an operable area and a non-operable area.

As another example, the obstacle map may include an area corresponding to an area belonging to a detection range such as a camera or an obstacle detection sensor and an area that does not correspond to it. Here, the detection range of the camera may be expressed as a viewing angle. An area corresponding to an area belonging to the detection range may include a detection area and an undetected area other than the detection area. The unit area can be divided into a detection area and a non-detection area according to whether object information can be acquired by a camera or an obstacle detection sensor. Alternatively, the unit area may be divided into a detection area and a non-detection area according to the accuracy of object information acquired by a camera or an obstacle detection sensor. For example, an area occluded by an obstacle may correspond to an undetected area.

Alternatively, the obstacle map may include an update area and other areas according to whether or not to update.

A weight may be assigned to each unit area of the obstacle map. The weight may reflect flight suitability. For example, in the case of a unit area corresponding to an area suitable for navigation, such as the sea, a high weight value is assigned, and in the case of a unit area corresponding to an area unsuitable for navigation due to the presence of an obstacle such as a ship, a low value weight Can be assigned. Conversely, a lower weight may be assigned as the flight suitability increases.

Hereinafter, mainly describing the representation of obstacles on the 2D grid map, but is not limited thereto.

Also, the term update can be interpreted as a concept including generation. For example, when an obstacle map of a current frame is updated and there is no obstacle map of a previous frame, updating the obstacle map may be interpreted as generating an obstacle map.

Object information may be expressed on the obstacle map. For example, the existence of an object can be displayed. In addition, various types of information such as object type, location, movement direction, speed, and route marker information can be expressed. It is also possible to mark only obstacles on the obstacle map. It can be expressed as the presence or absence of an obstacle, or it can be expressed as the probability of the existence of an obstacle. Alternatively, it may be expressed as a weight by using a number of a certain range.

When the obstacle is expressed as a weight, the weight can be classified into a plurality of numerical ranges. Each numerical range may correspond to an operable area, an unoperable area, and an unconfirmed area. 19 is a diagram of an obstacle map according to an embodiment. FIG. 19 is an example of a two-dimensional grid map. Referring to FIG. 19(a), the presence or absence of an obstacle in each grid/unit area/operation suitability is expressed by using an integer between 0 and 255 as a weight. The smaller the value, the higher the probability of the existence of an obstacle or unsuitable for navigation, and the unit area without a value is an area with a weight of 255. The non-operable area may correspond to a weight of 0~a, the unconfirmed area a+1~b, and the operable area b+1~255. Here, a and b are integers satisfying 0≦a<b≦254. For example, a non-operable area may correspond to a weight of 0, an operable area may correspond to a weight of 255, and an unconfirmed area may correspond to a weight of 1 to 254. Alternatively, the non-operable area may correspond to 0 to 50, the unconfirmed area may correspond to 51 to 200, and the operable area may correspond to 201 to 255. Referring to FIG. 19(b), an obstacle map expressed by weights can be visualized. For example, an obstacle map can be expressed by adjusting the brightness of an achromatic color. In Fig. 19(b), the smaller the weight, the lower the brightness. A detailed method of allocating weights to the unit area will be described later.

The object information for updating the obstacle map may be information based on an attribute of a means for obtaining the object information. For example, the location information of an obstacle acquired through image segmentation may be relative location information of a camera that has captured an image input to the image segmentation. The relative location information may be converted into location information on the obstacle map to be reflected on the obstacle map. Both the relative location information and the location information on the obstacle map may be used when updating the obstacle map, and only one of them may be used. Hereinafter, the coordinates on the obstacle map are referred to as absolute coordinates.

Absolute coordinates can be obtained using GPS, IMU, and ship/camera direction information. 20 is a block diagram illustrating a step of converting location information according to an embodiment. Referring to FIG. 20, the position information of the obstacle can be converted into absolute coordinates through the position of the ship/the position of the camera calculated by GPS installed on the ship, the position information of the ship through the IMU, and the camera position information through the camera installation information. have.

When updating the obstacle map, the entire obstacle map can be updated. Alternatively, it is possible to update only a partial area of the obstacle map.

The update area may be defined based on the update distance and the update angle. FIG. 21 is a diagram of an update area according to an embodiment, and shows a global map 310 and an area map 330 including a starting point A, a destination point B, and a current location C of a ship. The update area 331 may be defined by the update distance r and the update angle θ of the area map 330.

The update area may be determined according to the position of the ship, the direction of the bow, and the like. Alternatively, the update area may be determined according to an obstacle detection sensor such as a camera, a radar, or a lidar. For example, the update area may be determined by the angle of view and/or the maximum viewing distance of the camera. The update angle may be determined by the angle of view of the camera, and the update distance may be determined by the maximum viewing distance. Alternatively, the update area may be a predetermined area around the ship. It may be desirable that the update area is larger than the minimum area for evasive maneuvering after obstacle discovery.

Hereinafter, a case of expressing whether or not there is an obstacle on the obstacle map/whether suitable for operation is expressed as a weight will be described in detail. If the weight is low, the probability of the existence of an obstacle is high or it is described as being unsuitable for navigation, but is not limited thereto.

The weight can be different in consideration of the type information of the object. For example, when detecting a ship and a buoy as an obstacle, the weight of the ship may be set lower than the weight of the buoy. As another example, when detecting the fixed body and the moving body, the weight of the fixed body may be set lower than the weight of the moving body.

The weight may be set based on position information of the corrected object in consideration of the type information of the object. Alternatively, a weight of an undetected area may be set based on the object type information.

22 is a block diagram illustrating correction of location information according to an exemplary embodiment. Referring to FIG. 22, type information and location information of an object included in an image may be acquired through image segmentation, and an obstacle map may be updated by reflecting the type information to the acquired location information.

For example, a weight may be set by calculating the size of the object based on the type information of the object, and calculating a unit area corresponding to the object on an obstacle map using this. Here, the unit area may vary according to the direction of the object.

As another example, distance information obtained through image segmentation may include only the shortest distance to an object. When the object type information is reflected, not only the shortest distance but also the longest distance can be obtained, and a weight can be set based on this.

23 is a diagram for setting weights according to an embodiment. The distance information obtained through image segmentation by receiving the image of FIG. 23(a) may include only the shortest distance to the obstacle. In this case, referring to FIG. 23B, a region 351 that is not detected in the image may occur in which the presence or absence of an obstacle is unknown. On the obstacle map, the undetected area 351 may be treated as an area in which an obstacle exists or as an area in which an obstacle does not exist. Alternatively, some may be treated as an area in which an obstacle exists, and some as an area in which an obstacle does not exist.

The unit area on the obstacle map corresponding to the area 351 in which the presence or absence of the obstacle is not known may be reduced by using the type information of the obstacle. For example, as shown in FIG. 23(c), the size of the obstacle is calculated from information on the type of the obstacle, and an area on the obstacle map corresponding thereto may be processed as a non-navigation area. Alternatively, a predetermined unit area may be added to the area on the obstacle map corresponding to the size of the obstacle to be processed as a non-navigation area. Here, the weight of the predetermined unit region may be different from the weight of the region on the obstacle map corresponding to the size of the obstacle. The size of the obstacle may be an exact size of the obstacle or may be determined based on the type of the obstacle. Alternatively, the size of the obstacle may be a predetermined value according to the type of the obstacle.

Referring to FIG. 23D, even when the size of the obstacle is obtained from information on the type of the obstacle, there may be an area where the existence of the obstacle is unknown. The area may be an occluded area that is obscured by an obstacle and is not visible. The location information on the occluded area may be inaccurate information. The weight of the unit area on the obstacle map corresponding to the occluded area may be assigned as a value located between the weight of the unit area where the obstacle exists and the weight of the unit area where the obstacle does not exist.

The obstacle map may be updated by calculating weights from the corrected location information. Alternatively, the weight may be calculated based on location information before correction, and the obstacle map may be updated, and then the weight may be corrected in consideration of the type information.

The weight can be set in consideration of the moving direction and speed of the object. For example, the weight of the direction in which the object is to move can be set to be low. Alternatively, as the moving speed of the object increases, the weight may be set lower. FIG. 24 is a diagram for setting a weight in consideration of movement of an obstacle according to an embodiment, and a weight is expressed according to the object information expression method of FIG. 19 in consideration of a movement direction and speed. Referring to FIG. 24(a), when a ship, which is an obstacle, is stopped, the obstacle map may be divided into a non-navigation area 353 corresponding to the ship and a navigable area 355 corresponding to the sea. When the ship moves to the left, referring to FIG. 24(b), the weight of the movement predicted area 357 located on the left side of the ship on the obstacle map is determined by the weight of the non-navigation area 353 and the navigable area ( It can be set to a value located between the weights of 355). Referring to FIG. 24C, as the moving speed of the ship increases, the weight of the movement predicted area 357 may decrease. Alternatively, as the moving speed of the ship increases, the area of the expected movement area 357 may increase.

The weight can be set differently by judging the risk of the object. For example, an object with a higher risk may set a lower weight. The risk level may be calculated in consideration of object information such as the type of the object, the direction of movement, and the speed.

Weights can be set in consideration of weather environment information such as tide or wind. For example, a unit area that is difficult to go in a direction that goes against tide or wind may have a low weight, and a unit area that is easy to go due to a direction going through the tide or wind may have a high weight. Data on tide, wind, etc. can be acquired through a separate sensor. Alternatively, current/wind information according to region or time can be stored and used in advance. Alternatively, information about this can be received from outside in real time.

In the case of the terrain, it may be expressed on an obstacle map through the detection of the surrounding environment, but may be expressed on an obstacle map without sensing the surrounding environment by referring to a chart. Chart information and surrounding environment detection results can also be used together. The terrain may be set as a non-navigation area on the obstacle map.

Weight can be set in consideration of flight rules such as COLREG. According to the operating rules, the weight of the area in which the ship must be operated can be increased or the weight of the area in which the ship should not be operated can be decreased. For example, when a ship comes to the side, it may be necessary to detour to the right of the moving direction by COLREG. In this case, it is possible to increase the weight of the right unit area or decrease the weight of the left unit area so that the ship satisfies COLREG and operates. 25 is a diagram for setting weights in consideration of a navigation rule according to an embodiment. When COLREG is not considered, only a unit area corresponding to an oncoming ship may be displayed as a non-navigation area 353 on the obstacle map. In order to satisfy COLREG, the weight of the navigation rule reflection area 359, which is the left unit area of the obstacle map, may be reduced. Here, the weight of the navigation rule reflection area 359 may be set to a value located between the weight of the navigational area 355 and the weight of the non-navigation area 353. A route detouring to the right side of an oncoming ship may be generated by the navigation rule reflection area 359.

A buffer area having an arbitrary weight may be placed around the non-navigation area and/or the navigable area. The weight of the buffer area may be set to a value located between the weight of the navigable area and the weight of the non-navable area. Alternatively, the weight of the buffer area may be determined in consideration of a distance from the non-navigation area and/or the navigable area. For example, the weight of the area close to the navigable area may be greater than the weight of the area close to the non-navable area. 26 is a diagram for setting a weight in consideration of a buffer area according to an embodiment. An area map 330 is expressed according to the object information expression method of FIG. 19, and a non-navigation area 353, a navigable area 355 and a buffer area 358 are shown in the update area 331.

The obstacle map can be updated for the update area using the object information. You can update the presence or absence of obstacles on the obstacle map. For example, if an obstacle is found, the obstacle can be marked. Alternatively, the weights may be updated on the obstacle map. For example, a weight can be calculated using object information and an obstacle map can be updated using this.

Hereinafter, an obstacle map update according to the object information expression method of FIG. 19 will be described, but the obstacle map update method is not limited thereto, and may be applied to a case where a range of weights is different and an obstacle map is expressed with a probability of existence of an obstacle. have.

The obstacle map can be updated based on only the object information acquired in the current frame, ignoring the object information of the previous frame. For example, an obstacle map may be generated only with information obtained through the above-described weight setting.

Alternatively, the obstacle map may be updated in consideration of both the object information of the previous frame and the current frame. For example, the obstacle map may be updated in consideration of the weight of the current frame acquired through the above-described weight setting, the weight of the previous frame, or the obstacle map.

27 is a block diagram illustrating an obstacle map update step according to an embodiment. Referring to FIG. 27, the obstacle map of the current step may be generated in consideration of the obstacle map of the previous step (S3500). For example, a final obstacle map in the current frame may be generated by adding the obstacle map of the previous frame and the obstacle map of the current frame at a predetermined ratio. Here, a final obstacle map may be generated using a plurality of previous frames.

The area detected by the resolution image of the current frame ignores the previous frame, and the area not detected may update the obstacle map in consideration of the previous frame. For example, the weight of the unit area determined as an obstacle in the current frame is set to 0, the weight of the navigable area is set to 255, and the weight of the undetected area considers the weight of the previous frame and the weight of the current frame as a certain ratio. Can be set. Here, the weight of the undetected area may converge to a specific value over time. In addition, the specific value may be different according to the type of object. For example, in the case of a fixed obstacle, it may converge with a lower weight than that of a dynamic obstacle. Alternatively, in the case of an undetected area, the weight may not be updated.

A unit area corresponding to an area that does not belong to the viewing angle of the resolution image of the current frame may be processed in the same manner as the above-described undetected area.

Weights may not be updated for a specific area. For example, when a stationary obstacle that does not move in a region is detected, the weight of the region may be fixed to a specific value such as 0 and may not change even after time. Here, the fixed obstacle may be land, island, reef, or the like.

28 is a block diagram illustrating an obstacle map update step according to an embodiment. Referring to FIG. 28, the position information of the obstacle can be converted into absolute coordinates through the position of the ship/the position of the camera calculated by the GPS installed on the ship, the position of the ship through the IMU, and the camera posture information through the camera installation information. Yes (S3100). The weight is set using the converted object information, weather environment information, and navigation rules (S3300), and the final obstacle map of the current frame is output using the obstacle map of the previous frame and the update area set on the obstacle map (S3600). Can be (S3500).

The path creation step is a step of creating a path through which the ship can navigate. A route can be created in consideration of the movement distance, travel time, and operation cost of the ship. In addition, a route can be created in consideration of external environments such as tide and wind, and navigation rules such as COLREG can be considered.

Different routes can be created depending on the case by considering the shortest distance, minimum time, minimum cost, and risk. For example, because people are more likely to be present on the coast than the original one, it is possible to create a route in a high-safety mode for safe operation. In the case of original or large ships, a route can be created in an energy-efficient mode to reduce fuel consumption. It is also possible to create a path in an optimized manner by considering a plurality of variables in an appropriate ratio. These modes can be selected manually or automatically. For example, by using vessel location information such as GPS, a route may be automatically changed to a mode having high safety on the coast and a mode having high energy efficiency on the far shore to generate a route. As another example, when a large number of obstacles are detected, the mode is automatically changed to a high safety mode to generate a path.

29 is a block diagram illustrating a path generation step according to an embodiment. Referring to FIG. 29, a following route may be generated by inputting an obstacle map, a starting point, an arrival point, ship condition information, weather environment information, and a navigation rule (S3700). The input information may include all of the above-described information, or only part of the information, and information other than the above-described information may also be included.

The following path can be created using an artificial neural network. For example, an artificial neural network that outputs a following path by inputting an obstacle map or the like may be trained and used in the path generation step.

A tracking path can be created using an algorithm other than an artificial neural network. Any path generation algorithm such as Dijkstra algorithm, A* algorithm, D* algorithm, and theta* algorithm can be used without any restrictions.

The path can be created in consideration of the bow direction and the maximum rudder angle of the ship. 30 is a diagram illustrating a path generation step according to an embodiment. Referring to FIG. 30, the ship 371 is looking at a specific bow direction 373. When the bow direction 373 and the maximum steering angle of the ship 371 are not considered, a path such as a solid line is generated, and when this is considered, a path such as a dotted line may be generated. The path may be expressed as an intermediate point 375. A path such as the solid ship may be a path that an actual ship cannot follow or is difficult to follow. On the other hand, a path such as the dotted line may be a path that an actual ship can easily follow or can follow. Using such a ship-specific algorithm, it is possible to create a useful path for path tracking.

The path following step is a step of generating a control signal so that the ship follows the planned path. 31 is a block diagram illustrating a path following step according to an embodiment. Referring to FIG. 31, in the path following step (S5000), a following path, ship condition information, weather environment information, and navigation rules may be input and a control signal of a ship may be output. The input information may include all of the above-described information, or only part of the information, and information other than the above-described information may also be included.

The control signal includes a speed control signal and a bow direction control signal. The speed control signal may be a signal for adjusting the rotational speed of the propeller. Alternatively, the speed control signal may be a signal for adjusting the number of revolutions per unit time of the propeller. The bow direction control signal may be a signal for controlling a rudder. Alternatively, the bow direction control signal may be a signal for controlling a key (wheel, helm).

In addition, the control signal of the previous frame can be used in the path following step of the current frame.

The path following step can be performed using an artificial neural network. In this case, there are no restrictions on the method of learning the artificial neural network, such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

In the case of learning an artificial neural network through reinforcement learning, the artificial neural network can be trained with rewards such as flight time, flight distance, safety level, and energy efficiency. For example, if the flight time is short, the reward can be increased. Alternatively, the reward can be increased if the operating distance is short. Alternatively, as the number of collisions with an obstacle decreases, the reward may be increased. Alternatively, the less fuel used, the larger the reward. In addition to this, various rewards can be set, and rewards can be given by considering multiple variables rather than considering only one variable. For example, you can choose a reward setting method that considers both flight time and energy efficiency.

A path following step may be performed using other path tracking algorithms. For example, model predictive control (MPC), pure pursuit, Stanley's method, which is a technology that generates an optimal control signal by receiving a position to be followed for an arbitrary T time from the current point of time. method), vector pursuit, etc. algorithms can be used.

Even during autonomous navigation, humans may need information to monitor the vessel. This information can be communicated to a person through visualization. The results used or generated in the above-described surrounding environment detection, path planning, and path following steps may be processed and visualized.

Visualization data can be generated from a single image. Alternatively, visualization data may be generated from a plurality of images. Here, the plurality of images may be images acquired from different cameras as well as images acquired from the same camera.

When visualizing using a plurality of images, regions represented by each image may be different from each other. In this case, a plurality of images can be matched and visualized. For example, around-view monitoring may be possible by matching a plurality of images representing an area around a ship. Here, the around view monitoring may mean providing the environment around the ship as a bird's eye view.

When visualizing the result of image segmentation, the result of displaying the object detected by the image segmentation may be output through the display panel. For example, seas, ships, terrain, route signs, etc. included in the image can be expressed in different colors.

In addition, obstacle characteristics including distance, speed, risk, size, collision probability, and the like of the obstacle may be output. Obstacle characteristics may be output using color, and the color may vary depending on the distance, speed, risk, size, and collision probability of the obstacle.

Obstacle characteristics detected from the obstacle sensor can be output together. For example, a result may be output for an area observed by image segmentation, and a result detected by an obstacle sensor may be output for an area not observed.

Obstacle maps, following routes and/or route history can be visualized. For example, a ship-centered obstacle map, an existing path, and an obstacle avoidance path may be output in a bird's eye view format.

32 is a diagram for visualization according to an embodiment. Referring to FIG. 32, the display panel may output a relative position and characteristics of an obstacle based on a current position of a ship. For example, the black area 610 may indicate a first obstacle that is closer than a predetermined distance, and the gray area 630 may indicate a second obstacle that is farther than the predetermined distance. Alternatively, a black area 610 may indicate a high-risk area, a gray area 630 may indicate a normal-risk area, and a white area 650 may indicate a low-risk area.

33 is a block diagram of an autonomous navigation method using image segmentation according to an embodiment. Referring to FIG. 33, a photographed image is obtained by photographing an image with a camera (S1100), a pre-processed image is obtained through a pre-processing process of the photographed image (S1300), and first object information is obtained through segmentation of the pre-processed image. (S1500), the obstacle map is updated using the first object information and the second object information acquired through the obstacle sensor (S3500), and a following path is created using the obstacle map, starting point, arrival point, and ship status information. Then (S3700), a control signal of the current frame may be generated by using the following path, the ship state information, and the control signal of the previous frame (S5000). Here, the image segmentation S1500 may be performed using an artificial neural network, and the remaining steps may be performed using the above-described algorithm, not the artificial neural network. Alternatively, image segmentation is performed using an artificial neural network (S1500), path planning steps (S3500, S3700) use a non-artificial neural network algorithm, and path following step (S5000) may use an artificial neural network.

After the specific step is repeated a plurality of times within one frame, the next step may proceed. For example, after performing image segmentation (S1500) several times, path planning (S3500, S3700) and path following (S5000) may be performed. Alternatively, an obstacle map update (S3500) may be performed after several image segmentation (S1500).

Also, it is not necessary to execute all of one frame and execute the next frame. For example, while the path generation step (S3700) of one frame is executed, the image segmentation (S1500) of the next frame may be performed. Alternatively, a path following step of one frame (S5000), a path generation step of the next frame (S3700), and an image segmentation step of the next frame (S1500) may be performed together.

The embodiment of FIG. 33 separately performs each step of sensing the surrounding environment, planning a route, and following a route. Alternatively, at least any two of the steps of sensing the surrounding environment, planning a path, and following a path may be performed through a single artificial neural network. Hereinafter, a case in which path planning and path tracking are performed through a single artificial neural network and a case in which all of the surrounding environment detection, path planning, and path tracking are performed through a single artificial neural network will be described, but is not limited thereto.

34 is a block diagram of autonomous navigation using image segmentation according to an embodiment. Referring to FIG. 34, a photographed image is obtained by photographing an image with a camera (S1100), a pre-processed image is obtained through a pre-processing process of the photographed image (S1300), and first object information is obtained through segmentation of the pre-processed image. (S1500), the control signal of the current frame may be generated by using the second object information obtained through the first object information, the obstacle sensor, etc., the starting point, the arrival point, the ship status information, and the control signal of the previous frame (S7000) ).

According to an example of the autonomous navigation method of FIG. 33, a first artificial neural network for performing image segmentation, a second artificial neural network for path planning, and a third artificial neural network for path following are separately trained and then inferred. Can be used for Unlike this, one artificial neural network that performs all of image segmentation, path planning, and path tracking can be trained and used at a time. Learning of such artificial neural networks is called end-to-end learning.

35 is a block diagram of autonomous navigation using image segmentation according to an embodiment. Referring to FIG. 35, detection of the surrounding environment, path planning, and path tracking may be performed using a single artificial neural network. For example, images, object information from obstacle detection sensors, ship status information (ship's position and speed from GPS, ship's bow direction from IMU, etc.), starting point, arrival point, intermediate point, tide, wind, navigation rules It can be used for autonomous navigation by learning an artificial neural network that receives information such as control signals and outputs ship control signals. The above-described input information is an example, all of them may be used, only some of them may be used, and other information may be used.

36 is a block diagram of a learning method of an artificial neural network outputting a control signal according to an embodiment. This is a learning method that can be useful when it is difficult to secure a control signal as learning data. In the inference stage, a global artificial neural network is used. A local artificial neural network is used for learning of the global artificial neural network. Local artificial neural networks can cope with linearity, and global artificial neural networks can cope with non-linearity. The local artificial neural network can receive a path and an image and output a control signal. The path prediction artificial neural network may receive a path, an image, and a control signal and output a next path. The local artificial neural network can be trained through error learning based on the next path output from the control signal and path prediction output from the local artificial neural network. The global artificial neural network receives paths and images, outputs a control signal, and trains the global artificial neural network through a classifier using a control signal output from a local artificial neural network and a control signal output from the global artificial neural network. There may be a plurality of local artificial neural networks and path prediction artificial neural networks. The global artificial neural network can respond to all inputs and outputs with a single network by learning errors with several local artificial neural networks.

37 is a block diagram illustrating an output of the artificial neural network according to an embodiment, and is a diagram illustrating only the output of the artificial neural network of FIG. 35. Compared with FIG. 35, object information, an obstacle map, and a following path may be obtained from an artificial neural network for visualization. Hereinafter, this is referred to as intermediate output information. The intermediate output information may be visualized through the above-described visualization method.

A separate artificial neural network may be used to obtain intermediate output information. In the artificial neural network of FIG. 37 (hereinafter referred to as “autonomous artificial neural network”), intermediate output information may be obtained through an artificial neural network for visualization (hereinafter referred to as “visualized artificial neural network”). For example, the visualization artificial neural network may receive information output from the autonomously operating artificial neural network and output object information, an obstacle map, and a tracking path. Multiple intermediate output information can be obtained with one artificial neural network. Alternatively, a separate artificial neural network may be used for each intermediate output information.

The visualization artificial neural network can be learned separately from the autonomous navigation artificial neural network. Alternatively, it can be learned with an autonomous navigation artificial neural network.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The present invention described above is capable of various substitutions, modifications, and changes within the scope of the technical spirit of the present invention to those of ordinary skill in the technical field to which the present invention pertains. It is not limited by the drawings. Further, the embodiments described in this document are not limitedly applicable, but all or part of each of the embodiments may be selectively combined so that various modifications may be made. Furthermore, steps constituting each embodiment may be used individually or in combination with steps constituting other embodiments.

110: input image 130: output data
310: global map 330: regional map
331: update area 351: undetected area
353: non-operable area 355: operational area
357: expected movement area 358: buffer area
359: Area for reflecting the rules of operation 371: Vessel
373: forward direction 375: midpoint

Claims (1)

In the autonomous navigation method of a ship using maritime images and artificial neural networks,
Acquiring labeling data including a learning image including a plurality of pixel values and a plurality of labeling values determined by reflecting type information and distance information of a learning obstacle included in the learning image-the learning image and the labeling data Correspond to each other―;
Receiving, by the artificial neural network, the training image and outputting output data, wherein the output data includes a plurality of output values, and the labeling value and the output value correspond to each other;
Training the artificial neural network using an error function in consideration of a difference between the labeling value and the output value;
Acquiring the sea image from a camera installed on the ship;
Acquiring type information and distance information of an obstacle included in the maritime image using the artificial neural network, the obstacle included in the maritime image includes a plurality of pixel values;
Acquiring direction information of an obstacle included in the sea image based on a position of the pixel value of the obstacle included in the sea image on the sea image;
Acquiring a position of an obstacle included in the maritime image on an obstacle map using distance information and direction information of the obstacle included in the maritime image;
Generating the obstacle map by using the position on the obstacle map;
Generating a following path followed by the ship using the obstacle map and ship state information, the ship state information including position information and attitude information of the ship; And
Generating a control signal so that the ship follows the following path using the following path, wherein the control signal includes a propeller control signal and a bow direction control signal of the ship.
How to operate autonomously.

Images