KR20230065968A - Device and method for monitoring a berthing - Google Patents

Abstract

The present invention relates to a berthing monitoring method performed by a computing means, which includes the steps of acquiring a harbor image including the sea and a plurality of ships using a camera installed in a harbor and capturing an image according to an embodiment, an input image and Using an artificial neural network trained using a learning set in which class values indicating the sea, ship, and feature are labeled on pixels corresponding to objects including the sea, ship, and feature included in the input image, respectively Generating a segmentation image of the objects from a harbor image; Tracking the plurality of ships based on first pixels to which a class value indicating a ship of the segmentation image is assigned, and representing each of the plurality of ships. Obtaining tracking information including location information of points, determining a target ship to be berthing-guided among the tracked ships based on the tracking information, and determining the distance between the bow of the target ship and the quay based on the segmentation image. Disclosed is a berthing monitoring method comprising the step of obtaining berthing guide information including a distance between the stern and the quay, which is the distance between the stern and the quay wall of the target ship and the bow distance.

Description

Eyepiece monitoring device and method {DEVICE AND METHOD FOR MONITORING A BERTHING}

The present invention relates to an eyepiece monitoring device and method, and more particularly to an eyepiece monitoring device and method.

Recently, many accidents have occurred in the operation of ships and berthing and unloading in ports. There are many accidents mainly caused by the fact that the situation around the ship or in the port cannot be accurately confirmed through video when berthing.

Conventionally, berthing is supported using various types of sensors such as ECDIS and radar. However, in the case of ECDIS, there are limitations due to GPS inaccuracy, AIS update cycle, and AIS unregistered moving objects. There are limitations due to presence and noise.

Therefore, it is necessary to develop a technology for monitoring the situation around the ship or in the port when berthing, through images.

One problem to be solved by the present specification is to provide a berthing monitoring device and method for monitoring the vicinity of a ship and a port.

Another problem to be solved by the present specification is to provide a berth monitoring device and method for determining a vessel to be monitored and calculating and outputting information on the monitoring target.

Another problem to be solved by the present specification is to provide an eyepiece monitoring device and method using image segmentation.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. .

According to one aspect of the present specification, in the berth monitoring method performed by a computing means, obtaining a harbor image including the sea and a plurality of ships using a camera installed in the harbor to take an image, an input image and Using an artificial neural network trained using a learning set in which class values indicating the sea, ship, and feature are labeled on pixels corresponding to objects including the sea, ship, and feature included in the input image, respectively Generating a segmentation image of the objects from a harbor image; Tracking the plurality of ships based on first pixels to which a class value indicating a ship of the segmentation image is assigned, and representing each of the plurality of ships. Obtaining tracking information including location information of points, determining a target ship to be guided to berth among the tracked ships based on the tracking information, and determining the distance between the bow of the target ship and the quay based on the segmentation image. A berthing monitoring method may be provided that includes obtaining berth guide information including a distance between the bow distance and the stern distance, which is the distance between the stern and the quay wall of the target ship.

In addition, according to another aspect of the present specification, a camera installed in a harbor to capture an image, obtaining a harbor image including the sea and ships captured by the camera, and input images and seas and ships included in the input images and Segmentation of the objects from the harbor image using an artificial neural network trained using a learning set in which class values indicating sea, ship, and feature are labeled in pixels corresponding to objects including features, respectively An image is generated, and tracking information including location information of representative points representing each of the plurality of ships is tracked based on first pixels to which a class value indicating a ship of the segmentation image is assigned. Obtaining, determining a target ship to be berthing-guided among the ships to be tracked based on the tracking information, and based on the segmentation image, a bow distance that is a distance between a bow and a quay of the target ship and a stern and a quay of the target ship. A control module for obtaining berth guide information including a stern distance, which is the distance between the berth, and a berthing monitoring device including a communication module for transmitting berthing guide information of the target ship to a remotely located terminal may be provided.

The solutions to the problems of this specification are not limited to the above-mentioned solutions, and solutions not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. You will be able to.

According to one embodiment of the present specification, by calculating berthing guide information for monitoring the surroundings of the ship and the port, it is possible to perform monitoring of the ship and the port.

According to one embodiment of the present specification, it is possible to efficiently perform berthing monitoring by determining a vessel to be monitored, and calculating and outputting information on the determined monitoring target.

According to an embodiment of the present specification, a vessel may be recognized using image segmentation, and a monitoring target may be determined based on this.

Effects of the present invention are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

1 is a diagram related to image-based monitoring according to an embodiment.
2 is a diagram of an eyepiece monitoring device according to an embodiment.
3 and 4 are diagrams of an embodiment of an eyepiece monitoring device according to an embodiment.
5 is a view related to a viewing angle and a depth of field according to an exemplary embodiment.
6 and 7 are diagrams of an installation position of a sensor module according to an embodiment.
8 is a diagram related to image analysis according to an embodiment.
9 to 11 are diagrams of an object recognition step according to an embodiment.
12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
14 and 15 are diagrams related to estimating position/movement information of an object according to an embodiment.
16 is a flowchart of target vessel determination according to an embodiment.
17 is a diagram related to an example of ship tracking according to an embodiment.
18 is a diagram related to an example of determining a target vessel based on location information according to an embodiment.
19 is a diagram related to an example of determining a target vessel based on movement information according to an embodiment.
20 is a diagram related to an example of determining a target vessel based on voyage information according to an embodiment.
21 is a diagram related to an example of determining a target vessel based on user input information according to an embodiment.
22 is a diagram related to acquisition of berthing guide information of a target ship according to an embodiment.
23 is a flowchart of monitoring information output according to an embodiment.
24 is a diagram related to an example of monitoring information output according to an embodiment.
25 to 27 are diagrams related to other examples of outputting monitoring information according to an embodiment.
28 is a diagram related to image-based monitoring based on a plurality of images according to an embodiment.
29 and 30 are diagrams related to other examples of outputting monitoring information according to an embodiment.
31 and 32 are diagrams related to another example of outputting monitoring information according to an embodiment.
33 and 34 are diagrams related to viewpoint conversion according to an exemplary embodiment.
35 is a flowchart of eyepiece monitoring according to an embodiment.

The embodiments described in this specification are intended to clearly explain the spirit of the present invention to those skilled in the art to which the present invention belongs, and the present invention is not limited by the embodiments described in this specification, and the present invention The scope of should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as general terms that are currently widely used as much as possible in consideration of the functions in the present invention, but they may vary depending on the intention, custom, or the emergence of new technologies of those skilled in the art in the technical field to which the present invention belongs. can However, in the case where a specific term is defined and used in an arbitrary meaning, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not the simple name of the term.

The drawings accompanying this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

If it is determined that a detailed description of a known configuration or function related to the present invention in this specification may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

The term 'port image' used herein may be understood as an image related to a harbor, and for example, a harbor image includes an image captured by a camera installed in a harbor, an image including at least a part of the harbor, and the like can do.

According to one aspect of the present specification, in the berthing monitoring method performed by a computing means, obtaining a harbor image including the sea and a plurality of ships using a camera installed in the harbor to take an image; An artificial neural network trained using a learning set in which class values indicating the sea, ship, and feature are labeled on pixels corresponding to the input image and objects including the sea, ship, and feature included in the input image, respectively generating a segmentation image for the objects from the harbor image using the image; obtaining tracking information including location information of representative points indicating each of the plurality of ships by tracking the plurality of ships based on first pixels assigned with class values indicating ships of the segmentation image; Determining a target ship to be guided to berth among the ships to be tracked based on the tracking information; And acquiring berth guide information including a bow distance, which is the distance between the bow and the quay wall of the target ship, and a stern distance, which is the distance between the stern and the quay wall, of the target ship, based on the segmentation image; A method of monitoring the eyepiece including a may be provided.

Here, the representative point may be one point on the segmentation area of each of the plurality of ships of the segmentation image.

Here, the representative point may be a point located at the center of the segmentation area of each of the plurality of ships.

Here, the acquiring of the tracking information may include obtaining a distance between the representative point and the quay wall, and the target ship may be determined based on the distance between the representative point and the quay wall.

Here, the target ship may be determined as a ship of which the distance between the representative point and the quay wall is less than a predetermined value among the plurality of ships.

Here, the target ship may be determined as a ship in which the representative point is located on a predetermined area among the plurality of ships.

Here, the acquiring of the tracking information includes acquiring movement information of the representative point based on location information of the representative point, and the target ship may be determined based on the movement information of the representative point.

Here, the target ship may be determined based on the size of the approach speed of the representative point to the quay among the plurality of ships.

Here, the obtaining of the tracking information includes acquiring at least one of VTS information and AIS information of the plurality of ships, and the target ship may be determined based on at least one of the VTS information and AIS information. .

Here, the target ship may be determined based on a destination according to at least one of the VTS information and the AIS information among the plurality of ships.

Here, the acquiring of the tracking information includes acquiring user input information on the plurality of ships from a user terminal, and the target ship may be determined based on the user input information on the plurality of ships.

Here, the target ship may be determined as a ship selected from among the plurality of ships according to the user input information.

Here, the target ship may be determined as one ship that meets the condition later when there are a plurality of ships that meet the condition used for determining the target ship.

Here, the artificial neural network assigns the sea, the tugboat, the ship excluding the tugboat, and the feature to pixels corresponding to objects including the input image and the sea, tugboat, ship excluding the tugboat, and feature included in the input image, respectively. It can be learned using a learning set labeled with the indicated class values.

Here, the step of obtaining the berthing guide information is the step of extracting a pair of points corresponding to both ends of the bottom of the target ship in contact with the sea level, and calculating the bow distance and the stern distance based on the pair of points. Acquisition steps may be included.

Here, the step of obtaining the berthing guide information is a bow speed, which is a speed at which the bow of the target ship approaches the quay, and a speed at which the stern of the target ship approaches the quay, based on the bow distance and the stern distance. It may include acquiring the stern speed.

Here, the obtaining of the eyepiece guide information may include obtaining a distance between the target vessel and the other vessel based on the segmentation image.

Here, the obtaining of the berthing guide information may include obtaining a relative speed between the target vessel and the other vessel based on the distance between the target vessel and the other vessel.

Here, the step of obtaining the berthing guide information may include acquiring information related to a collision risk of the ship with another ship or the quay.

Here, the berthing monitoring method includes outputting the berthing guide information together with the harbor image; may further include.

Here, the harbor image may include a panoramic image in which a plurality of harbor images are matched.

Also, a recording medium on which a program for executing the above-described method is recorded may be provided.

In addition, according to one aspect of the present specification, the camera is installed in the port to take an image; Acquiring a harbor image including the sea and ships captured by the camera, and assigning the sea, ships, and landmarks to pixels corresponding to objects including the sea, ships, and landmarks included in the input image and the input image, respectively A first pixel to which a class value indicating a ship of the segmentation image is assigned by generating a segmentation image for the objects from the harbor image using an artificial neural network trained using a learning set labeled with class values indicating Obtain tracking information including positional information of representative points representing each of the plurality of ships by tracking the plurality of ships based on , and determine a target ship to be berthing guided among the ships to be tracked based on the tracking information. And, based on the segmentation image, a control module for obtaining berth guide information including a bow distance, which is a distance between a bow and a pier of the target ship, and a stern distance, which is a distance between the stern and the pier of the target ship; And a communication module for transmitting the berthing guide information of the target ship to a terminal located remotely; An eyepiece monitoring device including a may be provided.

Hereinafter, an eyepiece monitoring device and method according to an embodiment will be described.

In this specification, monitoring refers to grasping or recognizing a surrounding situation, and not only detecting a sensing object such as a certain area or a specific object using various sensors and providing the sensing result to the user, but also through calculation based on the sensing result. It should be interpreted broadly, including providing additional information.

Image-based monitoring may mean grasping or recognizing a surrounding situation based on an image. For example, monitoring may mean obtaining information for recognizing other vessels or obstacles by acquiring images around the vessel during operation of the vessel, or calculating berthing guide information when the vessel is berthing or unloading.

Berthing guide information includes the surrounding environment necessary for berthing, such as recognizing other vessels or obstacles, grasping the port situation, whether the berth is accessible, distance and speed from the quay, distance and speed from other vessels, and determining whether obstacles exist on the navigation route. can mean information about In the present specification, monitoring is mainly described when berthing is performed in a ship and a port, but is not limited thereto and may be applied to driving of a vehicle, operation of an aircraft, and the like.

1 is a diagram related to image-based monitoring according to an embodiment. Referring to FIG. 1 , image-based monitoring may include an image acquisition step (S10) and an image analysis step (S20).

The image acquisition step ( S10 ) may refer to a step in which the device 10 acquires an image. Here, the type of image may be various such as an RGB image, an IR image, a depth image, a lidar image, and a radar image, and is not limited thereto. In addition, not only 2D images but also 3D images are possible.

The image analysis step ( S20 ) may mean a step of obtaining an analysis result based on the image. For example, the image analysis step ( S20 ) may include calculating eyepiece guide information through an image. Alternatively, the image analysis step ( S20 ) may refer to a step of analyzing the characteristics of an object included in the image. Alternatively, the image analysis step (S20) may include determining a situation represented by the image. or,

Details of the image acquisition step ( S10 ) and the image analysis step ( S20 ) will be described later. Hereinafter, information acquired through the image acquisition step (S10) or the image analysis step (S20) is referred to as monitoring information.

2 is a diagram of an eyepiece monitoring device according to an embodiment. Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100 , a control module 200 and a communication module 300 .

The sensor module 100 may sense information about a ship or its surroundings and a port. The sensor module 100 may include an automatic identification system (AIS), an image generation unit, a position measurement unit, a posture measurement unit, and a casing.

The image generating unit may generate an image. The image generating unit may include a camera, lidar, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

The position measurement unit may measure the position of a component included in a sensor module, such as a sensor module or an image generating unit. For example, the location measurement unit may be a Global Positioning System (GPS). In particular, RTK-GPS (Real-Time Kinematic GPS) may be used to improve the accuracy of position measurement.

The location measurement unit may obtain location information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module. For example, when the sensor module is installed on a moving object such as a ship, the position measurement unit may obtain position information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the position measurement unit may acquire position information at every long time interval. A time interval for acquiring location information of the location measurement unit may be changed.

The posture measurement unit may measure the posture of a component included in a sensor module, such as a sensor module or an image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measurement unit may obtain posture information at every predetermined time interval. Here, the time interval may vary depending on the installation location of the sensor module. For example, when the sensor module is installed on a moving object such as a ship, the attitude measurement unit may acquire attitude information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the attitude measurement unit may acquire attitude information at every long time interval. A time interval for acquiring attitude information of the attitude measuring unit may be changed.

The casing may protect sensor modules such as an image generating unit, a position measurement unit, and a posture measurement unit.

At least one of an image generating unit, a position measurement unit, and a posture measurement unit may be present inside the casing. The casing can prevent equipment, such as an image generating unit, from being corroded by salt water. Alternatively, the casing may protect the equipment existing therein by preventing or mitigating impact applied thereto.

A cavity may be formed inside the casing to contain an image generating unit or the like therein. For example, the casing may have a hollow rectangular parallelepiped shape, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like may be disposed.

When the image generating unit is disposed inside the casing, an opening may be formed in one region of the casing or a transparent material such as glass may be formed in one region of the casing to secure the field of view of the image generating unit. The image generation unit may capture an image of the ship's periphery and the harbor through the opening or the transparent area.

The casing may be made of a strong material to protect the image generating unit from external impact. Alternatively, the casing may be made of a material such as an alloy for seawater to prevent corrosion due to salt.

The casing may include equipment for removing foreign substances from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove foreign substances. As another example, foreign substances may be removed by applying water or washer fluid through a liquid spray included in the casing, or foreign substances may be physically removed using a wiper after application.

The foreign matter removal equipment may be operated manually, but may also be operated automatically. For example, the foreign material removal equipment may be operated at predetermined time intervals. Alternatively, the foreign material removal equipment may be operated using a sensor that detects whether or not a foreign material is attached to the image generating unit. Alternatively, after determining whether a foreign material is captured in the image using the image captured by the image generating unit, the foreign material removing device may be operated when it is determined that the foreign material exists. Here, whether a foreign substance is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as two or more identical cameras.

The control module 200 may perform image analysis. In addition, an operation of receiving various data through the sensor module 100, an operation of outputting various outputs through an output device, an operation of storing various data in a memory or acquiring various data from a memory, and the like are of the control module 200. It can be done by control. Hereinafter, various operations or steps disclosed as embodiments of this specification may be interpreted as being performed by the control module 200 or under the control of the control module 200 unless otherwise stated.

Examples of the control module 200 include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), a state machine, and an on-demand Semiconductors (Application Specific Integrated Circuits, ASICs), radio-frequency integrated circuits (Radio-Frequency Integrated Circuits, RFICs), and combinations thereof may be used.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or unidirectional communication. For example, the device 10 may transfer information to an external output device through the communication module 300 and output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.

The sensor module 100, the control module 200, and the communication module 300 may include a control unit. The control unit may process and calculate various types of information within the module, and may control other components constituting the module. The controller may be physically provided in the form of an electronic circuit that processes electrical signals. A module may physically include only a single control unit, but may also include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be provided as processors that are mounted on physically separated servers and terminals and collaborate through communication. Examples of the control unit include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, and an application specific semiconductor (Application Specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100, the control module 200, and the communication module 300 may include a communication unit. The modules may transmit and receive information through a communication unit. For example, the sensor module 100 may transmit information acquired from the outside through the communication unit, and the control module 200 may receive the information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100, the control module 200, and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing program processing, or such processing result data. For example, the memory may store data necessary for learning and/or reasoning, learning is in progress or a learned artificial neural network, and the like. Memory is a non-volatile semiconductor memory, hard disk, flash memory, RAM (Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or any other tangible non-volatile recording medium. etc. can be implemented.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100, and each sensor module 100 may further include two cameras.

3 and 4 are diagrams of an embodiment of an eyepiece monitoring device according to an embodiment.

Referring to FIG. 3 , the monitoring device 10 may include a sensor module 100 and a control module 200. The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may change the viewpoint of the image by performing a viewpoint conversion to be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 and perform image analysis such as location/movement information estimation and image matching, which will be described later, through the control unit 220. In addition, the control module 200 may transmit analysis results such as location/movement information and matched images to the cloud server through the communication unit 210 . The cloud server may transmit the analysis result received from the control module 200 to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

Referring to FIG. 4 , the monitoring device 10 may include a sensor module 100 . The sensor module 100 may generate an image through the camera 130 and transmit the image to a cloud server through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may change the viewpoint of the image by performing a viewpoint conversion to be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as location/movement information estimation and image matching, which will be described later. In addition, the cloud server may transmit an image analysis result to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

The device 10 shown in FIGS. 2 to 4 is only an example, and the configuration of the device 10 is not limited thereto.

As an example, the device 10 may include an output module 400 . The output module 400 may output a result of an operation performed by the control module 200 and the like. For example, the output module 400 may output analysis results. The output module 400 may be, for example, a display, a speaker, a signal output circuit, etc., but is not limited thereto. In this case, information may be output through the output module 400 in addition to outputting the information by transferring the information to an external output device such as a user terminal.

As another example, device 10 may not include a sensor module. In this case, the control module 200 may perform an image-based monitoring operation such as performing image analysis by receiving information from an external sensor device. For example, the control module 200 may perform image analysis by receiving information from an AIS, camera, lidar, radar, or the like already installed in a ship or port.

In addition, the steps performed by each component in FIGS. 2 to 4 are not necessarily performed by the corresponding component, but may be performed by other components. For example, in FIG. 3 above, it is described that the controller 120 of the sensor module 100 performs the viewpoint conversion, but the controller 220 of the control module 200 or the cloud server may perform the viewpoint conversion. .

Hereinafter, the monitoring device 10 and method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100 . For example, an image may be obtained through an image generation unit included in the sensor module 100 . Alternatively, as described above, an image may be obtained from an external sensor device. Images for vessel and harbor monitoring will generally include sea, vessel, buoy, obstacle, terrain, harbor, sky, building, and the like. Hereinafter, a method of analyzing and monitoring an image acquired through a visible ray camera will be mainly described, but is not limited thereto.

Depending on the image generating unit, a field of view and a depth of field may vary. 5 is a view related to a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 5 , the field of view (FOV) may mean how much range is included in an image horizontally or vertically, and is generally expressed as an angle (degree). The meaning of having a larger viewing angle may mean generating an image including an area with a wider width horizontally or vertically. The depth of field may mean a distance range recognized as being in focus, and the depth of field may mean that the distance range recognized as being in focus is wide. Referring to FIG. 5 , according to the depth of field (DOF), the image may include an area A1 recognized as being in focus and an area A2 other than that. Hereinafter, the area included in the image is referred to as the imaging area (A1 + A2), and the area recognized as being in focus is referred to as the effective area (A1), and image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on or a part of the imaging area, the area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and shallow depth of field is a wide-angle camera. Examples of a camera with a small field of view and a deep depth of field include a high-magnification camera and a zoom camera.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, crane, ship, etc. in a port, and there is no limitation in the number thereof. However, depending on the type and performance of the sensor module 100, the installation location or number may vary. For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more from the surface of the water for efficient monitoring, or a plurality may be installed to have different imaging areas. In addition, the position and attitude of the sensor module 100 may be manually or automatically adjusted during or after installation.

6 and 7 are views of an installation position of the sensor module 100 according to an embodiment. Referring to FIGS. 6 and 7 , the sensor module 100 may be installed in a fixed location such as a port or land, or installed in a moving object such as a ship. Here, when the sensor module 100 is installed on a ship, it can be installed on a ship to be monitored (hereinafter referred to as “target ship”) as shown in FIG. 7, and docking or unloading the target ship as shown in FIG. It can also be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition to this, the sensor module 100 may be installed in a drone or the like to monitor a target ship.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or in a separate location.

As described above, image analysis for image-based monitoring may include obtaining object characteristics. Examples of objects may include ships, harbors, buoys, seas, terrain, sky, buildings, people, animals, fire, and smoke. Examples of object properties may include the type of object, the position of the object, the distance to the object, the absolute and relative speed and speed of the object, and the like. For another example,

Image analysis for image-based monitoring may include recognizing/determining the surrounding situation. For example, image analysis may determine that a fire situation has occurred from an image of a fire in a port, or that an intruder has entered from a captured image of a person entering a port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each module 100 and 200 .

8 is a diagram related to image analysis according to an embodiment. Referring to FIG. 8 , image analysis may include an object recognition step (S210) and a location/movement information estimation step (S220).

Image analysis may include an object recognition step (S210). The object recognizing step ( S210 ) may include recognizing an object included in the image. For example, object recognition may be determining whether an object such as a ship, a tugboat, the sea, or a harbor is included in an image. Furthermore, object recognition may be determining where an object exists in an image.

9 to 11 are diagrams of an object recognition step according to an embodiment.

9 is an image taken by a camera, and an object can be recognized as shown in FIG. 10 or 11 through an object recognizing step.

Specifically, FIG. 10 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation. In this case, the object recognition step may mean the segmentation step. Through segmentation, a characteristic corresponding to a pixel on the image may be allocated or calculated from the image. One might say that a pixel is assigned or labeled with a property. Referring to FIGS. 9 and 10 , a segmented image as shown in FIG. 10 may be obtained by performing segmentation based on an image captured by the camera of FIG. 9 . In FIG. 10 , the first pixel area P1 is an area on the image of a pixel corresponding to a ship, the second pixel area P2 is the sea, the third pixel area P3 is the quay wall of the harbor, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area P5 is an area on the image of a pixel corresponding to the sky.

Although FIG. 10 illustrates calculating information about the type of object corresponding to each pixel on an image by performing segmentation, information that can be obtained through segmentation is not limited thereto. For example, characteristics such as location, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, different characteristics may be expressed independently or may be reflected and expressed at the same time.

Table 1 is a table related to labeling that simultaneously reflects information on the type and distance of an object according to an embodiment. Referring to Table 1, classes may be set in consideration of object type and distance information, and an identification value may be assigned to each class. For example, identification value 2 may be assigned by considering both topography, which is information about the type of object, and short distance, which is information about distance. Table 1 is an example of a case in which information on type and information on distance are considered together. In addition, other information such as direction information, moving direction of obstacles, speed, and navigational aids may also be considered. In addition, all identification values do not have to include a plurality of pieces of information, nor do they have to include the same type of information. For example, a specific identification value includes only type information (eg, identification value 1 does not include information about distance), and other identification values include information about type and distance. It can be expressed in various ways.

identification value class 0 sky and other One ocean 2 terrain + close range 3 Terrain + Medium 4 terrain + distance 5 fixed obstacles + short range 6 fixed obstacles + medium distance 7 fixed obstacles + far 8 Dynamic Obstacles + Short Range 9 Dynamic Obstacles + Medium Range 10 Dynamic Obstacles + Far

FIG. 11 shows where an object exists in an image with a bounding box, which is also referred to as detection. In this case, the object recognition step may mean the detection step. Compared to segmentation, detection can be seen as detecting a position in which an object is included in a box shape rather than calculating characteristics for each pixel of an image. Referring to FIGS. 9 and 11 , a detection image as shown in FIG. 11 may be obtained by performing detection based on an image captured by the camera of FIG. 9 . In FIG. 11, it can be seen that the ship is detected on the image and the location of the ship is expressed as a rectangular bounding box (BB). Although only one object is illustrated in FIG. 11 as being detected, two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks and combine the results to calculate a final result.

An artificial neural network is a type of algorithm modeled after the structure of a human neural network, and may include one or more layers including one or more nodes or neurons, and each node may be connected through a synapse. Data (input data) input to the artificial neural network may be output (output data) through nodes through synapses, through which information may be obtained.

Types of artificial neural networks include a convolution neural network (CNN) that extracts features using a filter and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back to an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), and relational networks (RN) can be applied, and there are limitations. It is not.

Before using an artificial neural network, a training step is required. Alternatively, it may be trained using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and the step of using it as an inference step.

The artificial neural network may be trained through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.

12 is an example of a learning step of an artificial neural network, in which an untrained artificial neural network receives learning data or training data, outputs output data, and compares the output data with labeling data. The artificial neural network can be trained through backpropagation of the error. Training data, output data, and labeling data may be images. Labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or program.

13 is an example of an artificial neural network inference step, in which the trained artificial neural network receives input data and outputs output data. Information that can be inferred in the inference step may vary depending on the information of the learning data in the learning step. In addition, the accuracy of output data may vary according to the degree of learning of the artificial neural network.

Image analysis may include a position/movement information estimation step (S220). The location/movement information estimating step (S220) may include estimating information about the location and/or movement of at least some of the objects recognized in the object recognizing step (S210). Here, the location information may include an absolute location such as the coordinates of an object, a relative location from a specific reference, distance (distance from an arbitrary point, distance range, etc.), direction, and the like, and movement information may include absolute speed, relative speed, It may include information about the object's movement, such as speed.

The location/movement information of the object may be used when the ship is berthing or unloading. For example, it is possible to assist or guide the safe berthing or unloading of a vessel by using the distance from a berth or quay, the approach speed based on them, the distance from other vessels, and the relative speed during berthing or unloading of a vessel.

The location/movement information of the object may be used during ship navigation. For example, it can assist or guide the safe navigation of a ship by detecting another ship or obstacles around the ship, warning of a collision using the distance to them and their moving speed, or creating/recommending a route. . Alternatively, autonomous navigation may be performed based on such information.

Position/movement information of an object may be calculated based on an image. For example, location/movement information of a ship may be calculated based on an image including a ship, sea, and land as objects. Hereinafter, an object for estimating location/movement information is referred to as a target object. For example, in the above example, a ship may be a target object. Also, the number of target objects may be plural. For example, when estimating the position or speed of each of a plurality of ships included in the image, the plurality of ships may be the target object.

Location/movement information of an object may be expressed in a plurality of categories having a certain range. For example, distance information may be expressed as short distance, middle distance, and long distance, and direction information may be expressed as a left direction, a front direction, and a right direction. It will also be possible to combine these and express them as left near distance, right far distance, and the like. Movement information may be expressed as high speed, low speed, and the like.

The location/movement information of the object may be expressed as an actual distance value, direction value, and speed value. For example, distance information may be expressed in units of meters (m), direction information may be expressed in units of degrees, and movement information may be expressed in units of cm/s.

Position/movement information of an object may be estimated based on areas or points. For example, the distance between the ship and the quay may be estimated by calculating the distance between one point of the ship and one point of the quay, or may be estimated by calculating the shortest distance between one point of the ship and the quay. As another example, the distance between ships may be estimated by calculating the distance between one point of the first ship and one point of the second ship. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

Position/movement information of an object may be estimated based on image pixels. As described above, when location/movement information is estimated based on a point, a point may correspond to a pixel on an image. Accordingly, the location/movement information of the object may be calculated based on the interval between image pixels.

Distance information between points may be calculated based on an interval between pixels. For example, a predetermined distance may be assigned to each pixel interval and a distance between points may be calculated in proportion to the interval between pixels. As another example, distances between pixels may be calculated based on coordinate values of pixels on an image, and distances between points may be calculated based on the distances between pixels.

Movement information between points may be calculated based on changes in distance information between points. In this case, movement information may be calculated based on a plurality of images or video frames. For example, movement information between points may be calculated based on a distance between points in a previous frame, a distance between points in a current frame, and a time interval between frames.

14 and 15 are diagrams related to estimating position/movement information of an object according to an embodiment.

Referring to FIG. 14, the position / movement information estimation step is the position / movement information (f1, f2) with the quay wall (OBJ2) or the position / movement with other ships (OBJ3, OBJ4) when the ship (OBJ1) berths or unloads It may include estimating information (f3, f4). As shown in FIG. 14 , position/movement information f1 and f2 between the ship OBJ1 and the quay wall OBJ2 may be calculated for two points of the ship OBJ1. In this case, the two points may correspond to a point where the ship OBJ1 comes into contact with the sea. Also, the distance between the ship OBJ1 and the quay wall OBJ2 may be the shortest distance between the two points and the quay wall OBJ2. The position/movement information f3 and f4 between the ship OBJ1 and the other ships OBJ3 and OBJ4 may be position/movement information between points corresponding to the bow or stern of the ships OBJ1, OBJ3 and OBJ4. . In this way, when the location/movement information is used to assist or guide the berthing or unloading of a ship, it may be referred to as berthing guide information or emancipation guide information.

Referring to FIG. 15 , the position/movement information estimating step may include estimating position/movement information (f5, f6) with another ship (OBJ6) or an obstacle (OBJ7) such as a buoy during the operation of the ship (OBJ5). there is.

Port operation or management can be performed based on the data calculated in the location/movement information estimation step. For example, when a ship collides with an insect repellent (fender), the replacement time of the fender may be predicted by calculating an amount of impact from movement information such as a ship's speed.

In the above, an embodiment of image analysis of a method of estimating location/movement information after performing an object recognition step has been described. Alternatively, object recognition and location/movement information estimation may be performed in one step. For example, an object may be recognized by performing segmentation or detection, and location/movement information of the object may be estimated.

When all ships on the image are monitored, the processing speed of the device 10 may decrease due to the large amount of data calculation required to obtain berthing guide information of the ships. In addition, by determining a target ship to be monitored, obtaining berthing guide information of only the target ship (eg, bow distance, stern distance, etc.), and outputting the acquired berthing guide information, the user can exclude other unnecessary information. We can provide you with the information you need for docking. To this end, the control module 200 may perform image analysis to determine a target ship when a plurality of ships are included in at least one image acquired using the image generating unit.

16 is a flowchart of target vessel determination according to an embodiment.

Referring to FIG. 16 , image-based monitoring may include a vessel tracking step (S213), a tracking information acquisition step (S216), and a target ship determination step (S219).

According to one embodiment, the control module 200 may track the ship on the image (S213).

Tracking refers to an act of tracking a temporally and spatially changing feature point (object, tracking target) on an image. For example, tracking may include tracking the position, shape, motion, trajectory, information, etc. of a feature point that changes in space and time on an image. Also, tracking may include keeping images of feature points on an image.

According to an embodiment, the control module 200 may perform ship tracking based on the segmented image. Specifically, the control module 200 may track a ship based on a region in which a class value of a pixel of the segmented image corresponds to a ship. For example, the control module 200 may track a class value of segmented images (sequential) of sequentially photographed images based on a pixel corresponding to a ship. Here, the control module 200 may track a representative point (pixel) of a ship within a ship area formed by a pixel having a class value corresponding to the ship between successive frames.

17 is a diagram related to an example of ship tracking according to an embodiment.

Referring to FIG. 17 , the control module 200 may track ships in an image by tracking representative points 141 , 142 , and 143 representing ships in continuously photographed images. For example, the control module 200 may track one point on an area corresponding to each ship of the segmentation image.

The representative points 141, 142, and 143 of the vessel may be any feature point, but since the segmented image has a single class value, the control module 200 exemplarily sets a point located in the center of the area corresponding to the vessel. (141) can be determined as a representative point.

Of course, even if the control module 200 determines the representative points 141, 142, and 143 of the ship as points at other locations, such as the uppermost point, the lowest point 142, the leftmost point, and the rightmost point 143 of the ship area. In addition, the control module 200 may track the outline of the region corresponding to the ship or the region itself instead of determining one pixel as the representative point of the ship and tracking it.

Various techniques can be used for tracking, such as Kalman filter, Extended Kalman filter, Unscented Kalman filter, Particle filter, Information filter ), a histogram filter, and the like, and at least one of these may be applied to tracking.

Of course, this is not necessarily the case, and the control module 200 can track a ship based on an image such as a captured image, such as tracking a ship using a background image and a difference image. You can track the ship.

Also, the tracking step ( S213 ) may include the control module 200 tracking a ship other than a tugboat among ships included in the image. For example, the control module 200 may track a ship other than a tugboat among ships based on the segmentation image. Specifically, the control module 200 assigns pixels corresponding to objects including an input image and a sea, a tugboat, a ship excluding tugboats, and a feature included in the input image, respectively. Image segmentation may be performed using an artificial neural network trained using a learning set labeled with class values indicating water, and ships other than tugboats may be tracked. Here, the generated segmentation image may include pixels labeled to correspond to ships other than the tugboat, and the control module 200 may track pixels corresponding to ships other than the tugboat.

According to one embodiment, the control module 200 may obtain tracking information of a ship tracked on the image (S216).

Tracking information may refer to information about a tracked ship, and tracking information may be used to determine a target ship. For non-limiting examples, the tracking information may include ship location information, ship movement information, ship VTS information, ship AIS information, user input information, and the like.

According to one embodiment, obtaining tracking information (S216) may include obtaining, by the control module 200, ship location information from an image. For example, the control module 200 may obtain positional information of a representative point of a ship tracked from continuously photographed images. Here, the location information of the representative point of the ship may be information about the distance between the representative point and the quay, the distance between the representative point and another object, the coordinates of the representative point on an image, and the like.

According to an embodiment, the step of obtaining tracking information (S216) may include obtaining, by the control module 200, movement information of the vessel from the image. For example, the control module 200 may obtain movement information of a representative point of a ship tracked from continuously photographed images. Here, the control module 200 may obtain movement information of the representative point based on location information of the representative point. Here, the movement information of the representative point of the ship may be information about an approach speed of the representative point to the quay, an approach speed of the representative point to another object, a change in speed of the representative point, and the like.

According to one embodiment, the step of obtaining tracking information (S216) may include acquiring navigation information such as VTS information and AIS information of the ship to be tracked by the control module 200. For example, the control module 200 may obtain at least one of VTS information and AIS information. Here, the control module 200 may obtain information about the destination of the ship according to the voyage information, the expected docking time of the ship, and the like.

According to an embodiment, obtaining tracking information (S216) may include obtaining user input information about the ship being tracked by the control module 200. For example, the control module 200 may obtain user input information used to determine a target ship from a user terminal.

However, acquisition of tracking information does not need to be limited to being performed by the above-described method and may be performed in various ways.

According to one embodiment, the control module 200 may determine a target ship based on the acquired tracking information (S219).

According to an embodiment, the control module 200 may determine a target ship based on the acquired location information of the ship. For example, the control module 200 may determine the target vessel based on the distance between the vessel and the quay.

18 is a diagram related to an example of determining a target vessel based on location information according to an embodiment.

Referring to FIG. 18 , the control module 200 may determine a target vessel to be monitored based on location information when there are a plurality of vessels on an image.

According to one embodiment, the control module 200 may determine the target ship based on the distance from the quay of the tracked ship. Here, the distance from the quay wall of the ship to be tracked may be the distance between the representative point of the ship to be tracked and the quay wall.

For example, the control module 200 may determine, as a target ship, a ship whose distance from the quay is less than a critical distance among the ships to be tracked. Referring to FIG. 18 , the control module 200 may determine, among the ships to be tracked, a ship whose distance from the quay is smaller than a predetermined critical distance d ₀ as a target ship. The control module 200 may determine that the ship 151 is not the target ship when the ship 151 determined as the target ship moves and is located at a place where the distance from the quay wall is greater than d ₀ . In addition, the control module 200 may determine the ship 152 as a target ship when the ship 152 is moved and the distance from the quay is closer than d ₀ .

The control module 200 may variously set the threshold distance used for determining the target vessel. For example, the control module 200 may set the threshold distance based on the recognized size of the vessel. Specifically, the control module 200 may set the threshold distance larger as the size of the ship increases. In addition, the control module 200 may determine as a target ship a ship with a distance smaller than a critical distance to a specific object (structure, etc.) other than a quay among ships on the image.

According to an embodiment, the control module 200 may determine the target ship based on the position of the tracked ship on the image. Here, the location on the image of the ship to be tracked may be the location on the image of the representative point of the ship to be tracked. For example, the control module 200 may determine a ship located on a predetermined specific area of the image among ships on the image as a target ship. Referring to FIG. 18 , the control module 200 may determine a ship 151 located within a predetermined square-shaped area 153 on an image among tracked ships as a target ship. The control module 200 may determine the vessel 152 as a target vessel when the vessel 152 moves and is located within the area 153 . In addition, the control module 200 may exclude the ship 151 from the target ship when the ship 151 determined as the target ship moves and is located outside the area 153 .

The control module 200 may set the area 153 in various ways. For example, the control module 200 may variously determine the size, shape, number, and the like of the area 153 . Specifically, the control module 200 may determine the area 153 as an area around a path where ships mainly enter from the berth where the image is captured, an area where ships mainly dock, and the like. Conversely, the control module 200 may also determine a ship located in an area other than the predetermined specific area 153 among the ships on the image as the target ship.

According to an embodiment, the control module 200 may determine a target ship based on the acquired movement information of the ship. For example, the control module 200 may determine the target vessel based on the speed at which the vessel approaches the quay.

19 is a diagram related to an example of determining a target vessel based on movement information according to an embodiment.

Referring to FIG. 19 , the control module 200 may determine a target ship based on movement information when there are a plurality of ships on an image.

According to one embodiment, the control module 200 may determine the target ship based on the speed of the tracked ship. For example, the control module 200 may determine, as a target ship, a ship having a greater speed (absolute value of speed) than a threshold speed among ships on the image. Referring to FIG. 16 , the control module 200 may determine a ship 161 approaching at a speed v _o that is larger than a critical speed value among the ships being tracked as a target ship. The control module 200 may determine that the ship 161 is not the target ship when the ship 161 determined as the target ship is stopped at the berth after berthing is completed. In addition, the control module 200 may determine the ship 162 as a target ship when the anchored ship 162 moves at a speed greater than the threshold speed (regardless of direction).

The control module 200 may variously set the magnitude of the threshold speed used for determining the target vessel. For example, the control module 200 may determine the critical speed used for determining the target vessel based on the size of the vessel. Specifically, the control module 200 may determine the critical speed to increase as the size of the ship increases.

In addition, the control module 200 may determine a target vessel based on conditions related to movement information, such as direction, magnitude, and speed change (acceleration) of a tracked vessel. For example, the control module 200 may determine, as a target ship, a ship having an acceleration magnitude (absolute value of acceleration) greater than a preset value among ships on the image.

According to one embodiment, the control module 200 may determine a target ship based on the acquired navigation information of the ship. Here, the voyage information may include information related to the navigation of the ship, and for example, the voyage information may include AIS information, VTS information, CITS information, and the like. Sailing information may include information such as a departure point, a destination point, a mooring point, and a voyage route of the ship in relation to the voyage of the ship.

For example, the control module 200 may determine the target ship based on the destination according to the navigation information of the tracked ship.

20 is a diagram related to an example of determining a target vessel based on voyage information according to an embodiment.

Referring to FIG. 20 , the control module 200 may determine a target ship based on navigation information when there are a plurality of ships on an image.

According to one embodiment, the control module 200 may determine the target ship based on the destination according to the navigation information of the tracked ship. For example, the control module 200 may determine, among the ships on the image, a ship whose destination according to the voyage information is a berth in which a camera capturing an image is installed as a target ship. Referring to FIG. 17 , among the ships to be tracked, a ship 171 whose destination according to navigation information is a berth in which a camera capturing a corresponding image is installed may be determined as a target ship. The control module 200 may determine that the ship 171 is not the target ship when the ship 171 determined as the target ship embeds or departs according to the navigation information. In addition, the control module 200 may determine the ship 172 as a target ship when the ship 172 is moored elsewhere and then docks at a berth where a camera for capturing a corresponding image according to new navigation information is installed. .

In addition, the control module 200 may determine the target vessel based on other conditions related to voyage information. For example, the control module 200 may determine a ship recognized on an image as a target ship at a scheduled time to dock at a berth according to navigation information.

According to one embodiment, the control module 200 may determine a target ship based on the acquired user input information. User input information may include information input and received from a user's device, and for example, user input information may include information input from a user related to selection of a vessel to be monitored, selection of a berth, and the like.

21 is a diagram related to an example of determining a target vessel based on user input information according to an embodiment.

Referring to FIG. 21 , the control module 200 may determine a target ship based on user input information when there are a plurality of ships on an image. For example, the control module 200 may determine a ship selected as a monitoring target according to user input information among tracked ships as a target ship. Referring to FIG. 18 , the control module 200 may determine a ship 181 selected from an indicator 193 displayed on a screen according to user input information among tracked ships as a target ship. The control module 200 may determine a ship 181 that is not selected according to user input information as not being a target ship after the ship 181 determined as a target ship is docked. In addition, the control module 200 may determine the ship 182 as a target ship when the non-selected ship 182 is selected by the user according to user input information.

Here, user input information used for determining the target vessel may be provided in various ways. For example, the user input information may be information for determining a target ship by selecting a ship from a list of ships entering and exiting a port. In addition, the user input information may be information for determining a ship at a position selected on an output harbor image (eg, an output image of a user terminal) as a target vessel.

In addition, the control module 200 may determine an image of a berth to which monitoring information is output from among images captured at a plurality of berths. For example, the control module 200 may output an image of a berth selected according to user input information among images captured from a plurality of berths and determine a ship included in the image as a target ship.

Also, the step of determining the target vessel based on the obtained tracking information (S216) may include the control module 200 determining one target vessel on the image.

According to an embodiment, the control module 200 may determine, as a target ship, one ship among a plurality of ships that meets a condition for being determined as a target ship on the image. For example, the control module 200 may determine one ship determined later as the target ship as the target ship, rather than when there are a plurality of ships meeting the conditions for being determined as the target ship on the image. For example, referring to FIG. 18 , the control module 200 determines that when the ship 151 and the ship 152 are sequentially located within the area 153, the ship 151 is initially a target ship and then the ship 152 ) is located within the area 153, the target vessel may be determined as the vessel 152. Conversely, the control module 200 may determine one vessel determined as the target vessel as the target vessel prior to a case in which there are a plurality of vessels meeting the condition to be determined as the target vessel on the image.

However, the determination of the target vessel is not necessarily limited to being performed by the above-described method and may be performed in various ways.

22 is a diagram related to acquisition of berthing guide information of a target ship according to an embodiment.

Referring to FIG. 22 , the control module 200 may obtain berthing guide information of the target ship 191 among the ships 191 and 192 being tracked. Here, the berthing guide information may include a bow distance, a stern distance, a bow speed, a stern speed, a distance between a target ship and another ship, a relative speed between a target ship and another ship, and the like. The bow distance may mean the distance the bow of the ship is separated from the quay wall, and the stern distance may mean the distance the stern of the ship is separated from the quay wall. The bow speed may mean the speed at which the bow of the ship approaches the quay, and the stern speed may mean the speed at which the stern of the ship approaches the quay.

According to one embodiment, the control module 200 may acquire the distance between the target ship 191 and the quay wall.

For example, the control module 200 may obtain a bow distance 193 that is a distance between a bow of the target ship 191 and a quay wall. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship and obtain a distance between the one point corresponding to the bow and the quay wall. For another example, the control module 200 may obtain a stern distance 194 that is a distance between the stern and the quay wall of the target ship 191 . Specifically, the control module 200 may determine one point corresponding to the stern of the target ship and obtain a distance between the one point corresponding to the stern and the quay wall.

According to one embodiment, the control module 200 may extract a pair of points corresponding to both ends of the bottom of the ship in contact with the sea level. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may obtain the distance between one of a pair of points corresponding to both ends of the bottom of the ship in contact with the sea level and the quay wall in order to obtain the bow distance. In addition, the control module 200 may obtain a distance between the quay wall and another one of a pair of points corresponding to both ends of the bottom of the ship in contact with the sea level in order to obtain the stern distance.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191 and obtains the bow and stern distances of the ship, instead of obtaining one point or two or more points of the target ship 191. It is also possible to obtain the distance between the ship and the quay by extracting points, or it is okay to obtain the distance between the ship and the quay based on the outline of the region or the region itself.

According to one embodiment, the control module 200 may acquire the speed at which the target ship 191 approaches the quay wall.

For example, the control module 200 may acquire the bow speed, which is the speed at which the bow of the target ship 191 approaches the quay wall. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship 191 and acquire the speed at which the one point corresponding to the bow approaches the quay wall. Here, the control module 200 may acquire the speed at which the bow of the target vessel 191 approaches the quay based on the distance 193 of the bow of the target vessel 191 . The control module 200 compares the bow distance 193 of the target ship 191 in the current frame with the bow distance of the target ship 191 in the next frame, and the speed at which the bow of the target ship 191 approaches the quay wall. can be obtained.

For another example, the control module 200 may obtain a stern speed, which is a speed at which the stern of the target ship 191 approaches the quay. Specifically, the control module 200 may determine one point corresponding to the stern of the target ship 191 and acquire the speed at which the one point corresponding to the stern approaches the quay wall. Here, the control module 200 may obtain a speed at which the stern of the target ship 191 approaches the quay based on the distance 194 to the stern of the target ship 191 . The control module 200 compares the stern distance 194 of the target ship 191 in the current frame with the stern distance of the target ship 191 in the next frame, and the speed at which the stern of the target ship 191 approaches the quay wall. can be obtained.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191 to obtain the bow speed and stern speed of the ship, instead of obtaining any one point or two or more points of the target ship 191. It is also possible to obtain the speed at which the ship approaches the quay wall by extracting the points, or the speed at which the ship approaches the quay wall may be obtained based on the outline of the area or the area itself.

According to one embodiment, the control module 200 may obtain a distance between the target vessel 191 and another vessel 192 .

For example, the control module 200 may acquire the distance between the bow of the target vessel 191 and another vessel 192 . Specifically, the control module 200 may determine one point corresponding to the bow of the target vessel 191 and obtain a distance between the one point corresponding to the bow and the other vessel 192 .

For another example, the control module 200 may acquire a distance between the stern of the target ship 191 and another ship 192 . Specifically, the control module 200 may determine one point corresponding to the stern of the target ship 191 and obtain a distance between the one point corresponding to the stern and the quay wall.

According to one embodiment, the control module 200 may extract a pair of points corresponding to the bow end and the stern end in the area corresponding to the ship. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may obtain a distance between one of a pair of points corresponding to a bow end and a stern end in an area corresponding to a ship and another ship in order to obtain a distance between the ship and the other ship. can Here, the control module 200 may obtain the distance between the other ships based on a point closer to the other ship among a pair of points corresponding to the bow end and the stern end.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191 and obtains a distance between the bow and stern of the ship and another ship, instead of obtaining a point of one arbitrary point of the target ship 191 Alternatively, it is also possible to obtain the distance between a ship and another ship by extracting two or more points, and it is also possible to obtain the distance between a ship and another ship based on the outline of the area or the area itself.

According to one embodiment, the control module 200 may obtain a relative speed between the target vessel 191 and another vessel 192 .

For example, the control module 200 may acquire the speed at which the bow of the target vessel 191 approaches another vessel 192 . Specifically, the control module 200 may determine one point corresponding to the bow of the target vessel 191 and acquire the speed at which the one point corresponding to the bow approaches another vessel 192 . Here, the control module 200 may obtain a speed at which the bow of the target vessel 191 approaches the other vessel 192 based on the distance between the bow of the target vessel 191 and the other vessel 192 . The control module 200 compares the distance between the bow of the target ship 191 and the other ship 192 in the current frame and the distance between the bow of the target ship 191 and the other ship 192 in the next frame. The speed at which the bow of the target vessel 191 approaches another vessel 192 may be obtained.

For another example, the control module 200 may obtain a speed at which the stern of the target ship 191 approaches another ship 192 . Specifically, the control module 200 may determine one point corresponding to the stern of the target ship 191 and acquire the speed at which the one point corresponding to the stern approaches another ship 192 . Here, the control module 200 may obtain a speed at which the stern of the target ship 191 approaches the other ship 192 based on the distance between the stern of the target ship 191 and the other ship 192 . The control module 200 compares the distance between the stern of the target ship 191 and the other ship 192 in the current frame and the distance between the stern of the target ship 191 and the other ship 192 in the next frame. The speed at which the stern of the target ship 191 approaches another ship 192 may be acquired.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191 and obtains the speed at which the bow and stern of the ship approach another ship 192, instead of the target ship 191 It is also possible to obtain the relative speed between the target vessel 191 and the other vessel 192 by extracting one point or two or more points of the target vessel 191 and the target vessel 191 based on the outline of the area or the area itself. It is okay to obtain the relative speed between other vessels 192.

23 is a flowchart of monitoring information output according to an embodiment.

Referring to FIG. 23 , eyepiece monitoring may further include outputting monitoring information (S30). The information output in the monitoring information output step is not limited as long as it is information related to image-based monitoring, such as the image of the ship's surroundings, the ocean, and the port, and the type and distance/speed of objects included in the image.

The device 10 may visually output monitoring information. For example, the device 10 may output monitoring information through an output module such as a display. For another example, the device 10 may transmit monitoring information to the user's terminal through the communication module 300 and transmit a control signal to output the monitoring information through the display of the user terminal.

The monitoring information output step ( S30 ) may include displaying an image acquired using the image generating unit in the image acquisition step. In addition to this, it may include displaying various images related to image-based monitoring, such as an image after preprocessing, an image after segmentation or detection, and an image after viewpoint conversion.

In addition, the step of outputting the monitoring information (S30) may include displaying the location/movement information estimated by the control module 200 in the image analysis step through an output module or a user terminal.

24 is a diagram related to an example of monitoring information output according to an embodiment.

Referring to FIG. 24 , an image and eyepiece guide information may be displayed together. As shown in FIG. 24 , the displayed berthing guide information may include the bow distance, bow speed, stern distance, and stern speed of the target ship.

The step of outputting the monitoring information (S30) may include providing the information to the user by the device 10 in other ways such as outputting sound or vibration in addition to the visual display. For example, the control module 200 sounds a warning sound when there is a risk of collision of the target ship with a quay, other ships, obstacles, etc., when the approaching speed to the quay at the time of berthing is greater than a reference speed, or when the ship departs from its course and operates. It can be output through an output module or a user terminal.

Image-based monitoring may include surveillance. Here, surveillance may mean providing users with security-related information, such as monitoring intruders or port access of unregistered ships, and information about emergency situations, such as fire occurrence.

The device 10 may monitor an intruder based on whether a person is included in the image and when the image was captured. For example, the control module 200 may determine that an intruder exists when a person is included in a port image captured at a point in time when no work is being performed in the port.

The device 10 may perform vessel monitoring based on whether or not a vessel is included in the image. For example, the control module 200 may monitor a vessel by providing information to a user when a vessel not registered with AIS is detected.

Also, the device 10 may perform surveillance by detecting a person or a vessel based on an image through segmentation or detection.

25 to 27 are diagrams related to other examples of outputting monitoring information according to an embodiment.

Referring to FIGS. 25 and 26 , an image and berthing guide information may be displayed together, and when there are a plurality of target ships, a plurality of berthing guide information may be output.

As shown in FIG. 25, a location where berthing guide information is output may correspond to a location of a target ship on an image. For example, berthing guide information of a left target ship may be displayed on the left side, and berthing guide information of a right target ship may be displayed on the right side.

Also, as shown in FIG. 26, berthing guide information may be output together with an identifier of a target ship. For example, an arbitrary identifier of 'ship 1' is assigned to the target ship on the left side of the image and 'ship 2' is assigned to the target ship on the right side of the image, respectively, and the assigned identifier and berthing guide information of each target ship can be output together. . In addition, the assigned identifier and berth guide information may be displayed around the target ship on the image.

Also, referring to FIG. 27 , the image and berthing guide information may be displayed together, and when there are a plurality of ships and only one target ship is determined, only berthing guide information of the target ship may be output. For example, when the target ship is determined to be the left ship in which berthing is in progress, only berthing guide information of the left ship determined as the target ship may be displayed on the screen.

In the above, we looked at image-based monitoring based on a single image. In addition to this, image-based monitoring may be performed based on a plurality of images. When image analysis is performed based on a plurality of images, the total monitoring area of the image-based monitoring device 10 may increase or the accuracy of monitoring may be improved.

28 is a diagram related to image-based monitoring based on a plurality of images according to an embodiment.

Referring to FIG. 28 , the image acquisition step may include a first image acquisition step (S11) and a second image acquisition step (S12), and the image analysis step (S20) is acquired in the first image acquisition step (S11). Image analysis may be performed based on the first image and the second image obtained in the second image acquisition step ( S12 ).

Image analysis may be performed after generating one image based on a plurality of images. For example, the control module 200 may generate a panoramic image in which a plurality of images are matched, and perform image analysis on the panoramic image. Specifically, the control module 200 may generate a registered image or a fusion image by matching or fusing the first image and the second image, and perform image analysis based on the generated registered image or the fused image.

Alternatively, a final analysis result may be calculated based on a result of performing image analysis based on each of a plurality of images. For example, after obtaining first monitoring information by performing image analysis on a first image and acquiring second monitoring information by performing image analysis on a second image, a final monitoring information is obtained based on the first monitoring information and the second monitoring information. Monitoring information can be obtained.

As an example of a method of obtaining final monitoring information from a plurality of pieces of monitoring information, there may be a method of calculating final monitoring information by considering a plurality of pieces of monitoring information for each weight.

Alternatively, final monitoring information may be calculated based on whether a plurality of pieces of monitoring information do not match each other or whether the difference is greater than or equal to a threshold equal to a specific value (hereinafter referred to as "whether or not an error has occurred"). For example, based on whether or not an error has occurred, final monitoring information is calculated by considering a plurality of monitoring information for each weight, specific monitoring information is prioritized to calculate final monitoring information, or specific monitoring information is compared to other monitoring information. There may be a method of correcting with information or ignoring corresponding monitoring information, but is not limited thereto.

A plurality of images may be images of the same type. For example, when a sensor module includes two identical image generating units or two identical sensor modules including one image generating unit are disposed to perform image-based monitoring, the first image and the second image are of the same type. can be

Monitoring areas of a plurality of images may be different from each other. For example, the first image may monitor a short distance from the image generating unit, and the second image may monitor a long distance. Alternatively, the first image may monitor the left side and the second image may monitor the right side from the image generating unit.

In addition, the device 10 may output monitoring information obtained by analyzing a plurality of images together with suitable images.

The device 10 may output the monitoring information obtained by analyzing a plurality of images together with an image including a target ship among a plurality of images. For example, the control module 200 image-analyzes a plurality of images acquired by a plurality of cameras that capture images of at least some of the different regions, and then outputs a target ship among the plurality of images through an output module or a user terminal. The captured first image may be output together with the first eyepiece guide information calculated from the first image.

29 and 30 are diagrams related to other examples of outputting monitoring information according to an embodiment.

Referring to FIGS. 29 and 30 , when a target ship is determined after analyzing a plurality of images, berthing guide information of the target ship may be displayed together with an image including the target ship among a plurality of images. For example, when the target ship is determined to be 'ship 1', berthing guide information of 'ship 1' may be displayed together with an image of a berth including 'ship 1'. In addition, when the target ship is determined to be 'ship 3', berthing guide information of 'ship 3' may be displayed along with an image of a berth including 'ship 3'.

The device 10 may output the monitoring information acquired through the analysis of a plurality of images together with an image of a ship's destination among a plurality of images. For example, the control module 200 image-analyzes a plurality of images acquired by a plurality of cameras that take images of regions where at least a portion overlaps, and then, when the target ship is captured in a plurality of images, a plurality of images Among them, a control signal may be generated so that an image acquired using a camera installed at a destination according to navigation information of a target ship and berthing guide information calculated from the image are output together.

31 and 32 are diagrams related to another example of outputting monitoring information according to an embodiment.

31 and 32 show that eyepiece guide information is displayed together with a first image and a second image, which are images of respective berths adjacent to each other, acquired by a plurality of cameras that capture images of regions at least partially overlapping.

Referring to Figures 31 and 32, the ship 252 is taken together on the two images at the same time point. When the destination of the ship 252 is not the berth of the first image but the berth of the second image, it is preferable to output the berthing guide information of the ship 252 passing toward another place together with the first image instead of the second image. may not As shown in FIG. 31 , it is preferable that the first image is output together with berthing guide information of the ship 251 where the berth of the first image is the destination according to the voyage information. In addition, as shown in FIG. 32 , it is preferable that berth guide information of the ship 252 to which the berth of the second image is the destination is output together with the second image according to the voyage information.

Of course, monitoring information obtained by analyzing a plurality of images may be output together with an arbitrary image among the plurality of images, and is not limited to the above description.

Image-based monitoring may further include a viewpoint conversion step.

In general, an image generated by an image generating unit such as a camera may appear as a perspective view. Converting this to a top view (planar viewpoint), a side view, or another perspective viewpoint may be referred to as viewpoint conversion. Of course, a top view or side view image may be converted to another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, the view conversion may not need to be performed.

33 and 34 are diagrams related to viewpoint conversion according to an exemplary embodiment.

Referring to FIG. 33 , another near perspective image may be obtained through conversion of a perspective view image. Here, viewpoint conversion may be performed so that the pier OBJ8 is positioned along the horizontal direction (left-right direction on the image) on the image. Referring to FIG. 34 , a top view image may be obtained through conversion of a perspective view image of a perspective view image. Here, the top view image may be a view looking down at the sea level from a direction perpendicular to the sea level. Also, as in FIG. 33 , viewpoint conversion may be performed so that the quay wall OBJ9 is positioned along the horizontal direction on the image.

After performing the viewpoint conversion after image acquisition, image analysis can be performed to acquire information necessary for the berthing guide of the ship. According to an example, acquisition of eyepiece guide information may be performed based on a segmented image in which a viewpoint is converted. Specifically, the view point may be converted from a perspective view segmentation image to a top view segmentation image, and berthing guide information of a ship may be obtained based on a region in which a class value of a pixel of the top view segmented image corresponds to a ship. Of course, the viewpoint-converted segmentation image may be a segmentation image in which viewpoints are converted to various viewpoints such as a side viewpoint rather than a top view. In addition, instead of extracting two points to obtain ship position/movement information, the outline of the region, Alternatively, it is okay to acquire the location/movement information of the vessel based on the area itself.

Of course, this is not necessarily the case, and it is also possible to perform the operation based on an image such as a captured video having a changed viewpoint.

In addition, monitoring information such as displaying an image to a user may be output after performing viewpoint conversion after image acquisition. In this case, it is possible to more easily provide information on the surrounding situation to the user through the change of viewpoint.

Converting the viewpoint of an image may be performed in various ways.

As an example of viewpoint transformation, inverse projective mapping (IPM) may be performed. A 2D image is generated when light reflected from a subject in a 3D space is incident on an image sensor through a lens of a camera, and the relationship between 2D and 3D depends on the image sensor and lens. For example, Equation 1 and can be expressed as

Here, the matrix on the left side denotes 2D image coordinates, the first matrix on the right side denotes intrinsic parameters, the second matrix denotes extrinsic parameters, and the third matrix denotes 3D coordinates. Specifically, fx and fy denote focal lengths, cx and cy denote principal points, and r and t denote rotation and translation parameters, respectively.

A viewpoint can be changed by projecting a 2D image onto an arbitrary 3D plane through back-projection transformation. For example, a perspective view image may be converted into a top view image through back projection transformation, or may be converted into another perspective view image.

Internal parameters may be required for viewpoint conversion. As an example of a method for obtaining internal parameters, the Zhang method may be used. The Zhang method is a kind of polynomial model, which acquires internal parameters by taking pictures of a grid having a known grid size from various angles and distances.

Information on the position and/or attitude of the image generating unit/sensor module that captured the image may be required for viewpoint conversion. This information may be obtained from the position measurement unit and the attitude measurement unit.

Alternatively, information on the position and/or posture may be obtained based on the position of the fixed body included in the image. For example, the image generating unit may generate a first image including a target fixture, which is a fixed object such as a terrain or a building, disposed in a first position and/or a first posture at a first viewpoint. Then, at a second viewpoint, the image generation unit may generate a second image including the target fixture. The position of the target fixture on the first image and the position of the target fixture on the second image are compared to calculate the second position and/or the second posture, which is the position and/or posture of the image generating unit at the second viewpoint. can do.

In addition, the accuracy of image analysis may vary depending on the selection of the reference plane when converting the viewpoint. For example, when converting a perspective view image into a top view image, the accuracy of image analysis based on the top view image may vary according to the height of the reference plane. In order to accurately calculate the distance between objects on the sea level, it may be preferable that the reference plane is the sea level when converting the viewpoint. Since the height of the sea level may change over time, it may be desirable to improve the accuracy of image analysis by performing viewpoint conversion in consideration of the height of the sea level.

The above-described viewpoint conversion method is only an example, and viewpoint conversion may be performed in a different way. contains the information necessary for

Image-based monitoring may include a pre-processing step. Preprocessing refers to all kinds of processing performed on images, such as image normalization, image equalization, histogram equalization, image resize, and upscaling to change the resolution/size of the image. and downscaling, cropping, denoising, and the like. Here, noise may include fog, rain, water droplets, sea fog, fine dust, direct sunlight, salt, and combinations thereof, and removing noise means removing or reducing noise components included in an image. can include

Looking at normalization as an example of preprocessing, normalization may mean obtaining an average of RGB values of all pixels of an RGB image and subtracting the average from the RGB image.

Looking at defogging as another example of preprocessing, defogging may mean converting an image of a foggy area to look like an image of a clear area through preprocessing. 13 is a view related to fog removal according to an embodiment. Referring to FIG. 13 , through fog removal, an image of a foggy region as shown in FIG. 13 (a) may be converted into a fog-free image as shown in FIG. 13 (b).

Looking at the removal of water droplets as another example of pre-processing, the removal of water droplets may mean converting an image of water droplets formed on the front of a camera to appear as if the water droplets have been removed through pre-processing.

According to an embodiment, after the image acquisition step (S10), the image analysis step (S20) may be performed through a preprocessing step. For example, image analysis may be performed after performing pre-processing on an image obtained by using the image generation unit. Image preprocessing may facilitate image analysis or improve accuracy.

Image pre-processing may be performed through an artificial neural network. For example, an image containing noise can be input to an artificial neural network to obtain a denoised image, such as inputting an image of a foggy area into an artificial neural network and converting it to look like an image of a sunny area. can Examples of artificial neural networks include, but are not limited to, GANs.

Alternatively, image preprocessing may be performed using an image mask. For example, an image of a foggy area can be converted to look like an image of a clear area by applying an image mask. Here, examples of the image mask include a deconvolution filter, a sharpen filter, and the like, and the image mask may be generated through an artificial neural network such as GAN, but is not limited thereto.

In the above, we looked at the case of performing image analysis after preprocessing the image. Alternatively, it may be possible to perform image analysis including a preprocessing process. For example, when the image analysis step includes segmentation or detection, a result of performing segmentation or detection of an image including noise may be implemented to be equivalent to a result of performing segmentation or detection of an image without noise.

35 is a flowchart of image-based monitoring according to an embodiment.

Referring to FIG. 35, berthing monitoring is an image acquisition step (S1010), a segmentation step (S1210), a ship tracking step (S1213), a tracking information acquisition step (S1216), a target ship determination step (S1219), a berthing guide information acquisition step. (S1220) and outputting monitoring information (S1030). Each step can be implemented as described above.

The device 10 may acquire an image including an object through an image acquisition step ( S1010 ). The device 10 may obtain a segmented image by performing a segmentation step ( S1210 ) based on the image. The device 10 may track at least one ship recognized based on the segmentation image by performing a ship tracking step (S1213). The device 10 may acquire tracking information by performing a tracking information acquisition step (S1216). Here, the tracking information may include at least one of ship location information, ship movement information, ship navigation information, and user input information. The apparatus 10 may perform a target ship determination step (S1219) to determine a target ship among ships to be tracked based on the acquired tracking information. The device 10 may obtain berthing guide information of the determined target ship based on the segmentation image by performing the berthing guide information acquisition step (S1220). Here, the berthing guide information may include at least one of a bow distance and a stern distance of the target ship. The device 10 may output the eyepiece guide information together with the image by performing the monitoring information output step (S1030).

The embodiment of FIG. 35 is only an example, and image-based monitoring may be performed in a different way.

For example, some steps may not be performed in the embodiment of FIG. 35 . Referring to FIG. 35 , outputting monitoring information (S1030) may not be performed.

As another example, another step may be added in the embodiment of FIG. 35 , such as adding a step of matching images or a step of converting a viewpoint of an image.

As another example, in the embodiment of FIG. 35 , such as a segmentation step being replaced with a detection step, some steps may be replaced with other steps.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In the above, the configuration and characteristics of the present invention have been described based on the examples, but the present invention is not limited thereto, and various changes or modifications can be made within the spirit and scope of the present invention to those skilled in the art. It is obvious, and therefore such changes or modifications are intended to fall within the scope of the appended claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication department
220: control unit
300: communication module

Claims (19)

In the vessel monitoring method performed by the computing means,
Obtaining a harbor image using a camera installed in the harbor to take an image;
An artificial neural network trained using a learning set in which class values indicating the sea, ship, and feature are labeled on pixels corresponding to the input image and objects including the sea, ship, and feature included in the input image, respectively generating a segmentation image for the objects from the harbor image using the image;
Acquiring at least one of VTS information and AIS information of a plurality of ships;
Determining a target vessel using at least one of the VTS information and the AIS information; and
Obtaining guide information including a bow distance, which is a distance between a bow and a pier of the target ship, and a stern distance, which is a distance between the stern and the pier of the target ship, based on the segmentation image; Including,
How to monitor ships. According to claim 1,
The step of determining the target ship,
Obtaining voyage information including destinations and scheduled docking times of each of the plurality of ships using at least one of the VTS information and the AIS information; and
Determining a target ship based on the destination and the scheduled docking time of each of the plurality of ships; including,
How to monitor ships. According to claim 2,
The step of determining the target ship,
identifying at least one ship whose destination is a berth where the camera is installed among the plurality of ships; and
Determining the target ship based on the expected berthing time among the identified at least one ship; including,
How to monitor ships. According to claim 3,
The step of determining the target ship,
Determining a ship identified through the segmentation image at the scheduled docking time among the identified at least one ship as a target ship; including,
How to monitor ships. According to claim 1,
The step of determining the target ship,
Tracking the plurality of ships using at least one of the VTS information and the AIS information; and
Determining the target ship based on location information of a representative point representing the ship on the segmentation image among the plurality of ships being tracked; including,
How to monitor ships. According to claim 5,
The step of determining the target ship,
Determining the target ship based on the distance between the representative point and the quay wall; including,
How to monitor ships. According to claim 6,
The step of determining the target ship,
Determining, among the plurality of ships to be tracked, a ship having a distance between the representative point representing the ship on the segmentation image and the quay wall that is less than a preset value as the target ship; including,
How to monitor ships. According to claim 5,
The step of determining the target ship,
Determining a ship in which the representative point is located on a predetermined area as the target ship; Including,
How to monitor ships. According to claim 5,
The step of determining the target ship,
obtaining movement information of the representative point based on location information of the representative point; and
Determining the target ship based on the movement information of the representative point; including,
How to monitor ships. According to claim 9,
The step of determining the target ship,
Determined based on the magnitude of the approach speed of the representative point to the quay,
How to monitor ships. According to claim 1,
The bow distance and the stern distance,
Obtained based on a pair of points corresponding to both ends of the bottom of the target vessel in contact with the sea level,
How to monitor ships. According to claim 1,
The guide information,
Including a bow speed, which is a speed at which the bow of the target ship approaches the quay, and a stern speed, which is a speed at which the stern of the target ship approaches the quay, obtained based on the bow distance and the stern distance,
How to monitor ships. According to claim 1,
The guide information,
Including the distance between the target ship and another ship obtained based on the segmentation image,
How to monitor ships. According to claim 13,
The guide information,
Including the relative speed between the target vessel and the other vessel obtained based on the distance between the target vessel and the other vessel,
How to monitor ships. According to claim 1,
The artificial neural network,
Class values indicating the sea, tugboat, ship excluding tugboat, and feature are labeled for pixels corresponding to objects including the sea, tugboat, ship excluding tugboat, and feature included in the input image, respectively. Learned using one learning set,
How to monitor ships. According to claim 1,
Outputting the guide information together with the harbor image; further comprising,
How to monitor ships. According to claim 1,
The image of the port,
Including a panoramic image in which a plurality of harbor images are matched,
How to monitor ships. Recording a program for executing the method according to any one of claims 1 to 17,
recording media. A camera installed in the port to capture images;
Obtaining a harbor image captured by the camera, and labeling class values indicating the sea, ship, and feature to pixels corresponding to objects including the sea, ship, and feature included in the input image and the input image, respectively Generating a segmentation image for the objects from the harbor image using an artificial neural network trained using one learning set, obtaining at least one of VTS information and AIS information of a plurality of ships, and the VTS information and AIS information A target ship is determined using at least one of the segmentation images, including a bow distance that is a distance between a bow and a pier of the target ship and a stern distance that is a distance between a stern and a pier of the target ship based on the segmentation image. A control module for acquiring guide information to; and
A communication module for transmitting the guide information to a terminal located remotely; including,
Ship monitoring device.

Images