KR102235787B1 - Device and method for monitoring a berthing - Google Patents

Abstract

The present invention relates to a berthing monitoring method performed by a computing means to monitor the vicinity of a ship and a harbor. According to one embodiment of the present invention, the berthing monitoring method comprises: a step of using a camera installed on a harbor to photograph an image to acquire a harbor image including the sea and a plurality of ships; a step of using an artificial neural network trained by using an input image and a learning set resulting from labeling pixels corresponding to objects including the sea, ships, and terrain features included in the input image with class values indicating the sea, ships, and terrain features to generate a segmentation image for the objects; a step of tracking the plurality of ships based on first pixels which are assigned a class value indicating a ship of the segmentation image to acquire tracking information including position information of representative points representing the plurality of ships; a step of determining a berthing-guided target ship among the tracked ships based on the tracking information; and a step of acquiring berthing guide information including a stem distance between the stem of the target ship and a pier and a stern distance between the stern of the target ship and the pier based on the segmentation image.

Description

Eyepiece monitoring device and method {DEVICE AND METHOD FOR MONITORING A BERTHING}

The present invention relates to an apparatus and method for eyepiece monitoring, and more particularly, to an apparatus and method for performing eyepiece monitoring based on an image.

Recently, many accidents have occurred in the operation of ships, berths and berths in ports. There are many accidents that occur mainly due to the inability to accurately check the situation around the ship or in the port through images when berthing.

Therefore, conventionally, the eyepiece is supported using various types of sensors such as ECDIS and radar. However, in the case of ECDIS, there are limitations due to the inaccuracy of GPS, the update period of AIS, and the non-registered mobile object, and in the case of radar, the There are limitations due to presence and noise.

Therefore, it is necessary to develop a technology for monitoring the situation around the ship or in the port through images when berthing is practically berthing.

One problem to be solved in the present specification is to provide an berthing monitoring device and method for monitoring the vicinity of the ship and the port.

Another task to be solved in the present specification is to provide a berth monitoring apparatus and method for determining a vessel that is a monitoring target, and calculating and outputting information on a monitoring target.

Another problem to be solved in the present specification is to provide an eyepiece monitoring apparatus and method using image segmentation.

The problem to be solved by the present invention is not limited to the above-described problems, and problems that are not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings. .

According to an aspect of the present specification, in the eyepiece monitoring method performed by a computing means, the step of acquiring a port image including the sea and a plurality of ships using a camera installed in a port to capture an image, an input image, and Using an artificial neural network learned using a learning set in which class values indicating the sea, ship, and terrain are respectively labeled in pixels corresponding to objects including sea, ship, and terrain included in the input image, the Generating a segmentation image for the objects from a port image, representing each of the plurality of vessels by tracking the plurality of vessels based on first pixels to which a class value indicating a vessel of the segmentation image is assigned Acquiring tracking information including location information of points, determining a target ship to be guided by the eyepiece among the tracked ships based on the tracking information, and between the bow and the quay wall of the target ship based on the segmentation image An eyepiece monitoring method comprising the step of acquiring eyepiece guide information including a bow distance, which is a distance, and a stern distance, which is a distance between a stern and a quay wall of the target vessel may be provided.

In addition, according to another aspect of the present specification, a camera installed in a port to capture an image, a port image including the sea and a ship captured by the camera is acquired, and an input image and a sea, a ship included in the input image And segmentation of the objects from the port image by using an artificial neural network learned using a learning set in which a pixel corresponding to objects including a feature is labeled sea, a ship, and a class value indicating a feature, respectively. An image is generated, and tracking information including location information of representative points representing each of the plurality of ships is provided by tracking the plurality of ships based on first pixels to which a class value indicating a ship of the segmentation image is assigned. It is obtained, based on the tracking information, determines a target ship that is guided by the eyepiece among the tracked ships, and based on the segmentation image, the bow distance, which is the distance between the bow and the quay wall, and the stern and quay wall of the target ship An eyepiece monitoring apparatus including a control module for acquiring eyepiece guide information including a stern distance, which is a distance therebetween, and a communication module for transmitting eyepiece guide information of the target vessel to a remotely located terminal may be provided.

The means for solving the problems of the present specification are not limited to the above-described solutions, and solutions that are not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings. I will be able to.

According to an embodiment of the present specification, by calculating the berthing guide information for monitoring the vicinity of the ship and the port, it is possible to perform monitoring for the ship and the port.

According to an exemplary embodiment of the present specification, it is possible to efficiently perform eyepiece monitoring by determining a vessel that is a monitoring target, and calculating and outputting information on the determined monitoring target.

According to an embodiment of the present specification, a vessel may be recognized using image segmentation, and a monitoring target may be determined based on this.

The effects of the present invention are not limited to the above-described effects, and effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings.

1 is a diagram for image-based monitoring according to an embodiment.
2 is a diagram of an eyepiece monitoring apparatus according to an exemplary embodiment.
3 and 4 are diagrams of an embodiment of an eyepiece monitoring apparatus according to an embodiment.
5 is a diagram illustrating a viewing angle and a depth of field according to an exemplary embodiment.
6 and 7 are diagrams illustrating an installation location of a sensor module according to an exemplary embodiment.
8 is a diagram illustrating image analysis according to an exemplary embodiment.
9 to 11 are diagrams illustrating an object recognition step according to an exemplary embodiment.
12 and 13 are diagrams illustrating a learning step and an inference step of an artificial neural network according to an embodiment.
14 and 15 are diagrams for estimating location/movement information of an object according to an exemplary embodiment.
16 is a flow chart for determining a target vessel according to an embodiment.
17 is a diagram illustrating an example of tracking of a ship according to an embodiment.
18 is a diagram illustrating an example of determining a target vessel based on location information according to an embodiment.
19 is a diagram illustrating an example of determining a target vessel based on movement information according to an embodiment.
20 is a diagram illustrating an example of determining a target ship based on voyage information according to an embodiment.
21 is a diagram illustrating an example of determining a target vessel based on user input information according to an embodiment.
22 is a diagram illustrating acquisition of eyepiece guide information of a target ship according to an exemplary embodiment.
23 is a flowchart illustrating outputting monitoring information according to an exemplary embodiment.
24 is a diagram illustrating an example of outputting monitoring information according to an embodiment.
25 to 27 are diagrams illustrating another example of outputting monitoring information according to an exemplary embodiment.
28 is a diagram illustrating image-based monitoring based on a plurality of images, according to an exemplary embodiment.
29 and 30 are diagrams illustrating another example of outputting monitoring information according to an embodiment.
31 and 32 are diagrams illustrating another example of outputting monitoring information according to an embodiment.
33 and 34 are diagrams for view conversion according to an embodiment.
35 is a flowchart of eyepiece monitoring according to an embodiment.

The embodiments described in the present specification are intended to clearly explain the spirit of the present invention to those of ordinary skill in the technical field to which the present invention pertains, so the present invention is not limited by the embodiments described herein, and the present invention The scope of should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as general terms currently widely used in consideration of functions in the present invention, but this varies according to the intention, custom, or the emergence of new technologies of those of ordinary skill in the technical field to which the present invention belongs. I can. However, if a specific term is defined and used in an arbitrary meaning unlike this, the meaning of the term will be separately described. Therefore, the terms used in the present specification should be interpreted based on the actual meaning of the term and the entire contents of the present specification, not a simple name of the term.

The drawings attached to the present specification are for easy explanation of the present invention, and the shapes shown in the drawings may be exaggerated and displayed as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a well-known configuration or function related to the present invention may obscure the subject matter of the present invention, a detailed description thereof will be omitted as necessary.

The term'port image' used in this specification can be understood as an image related to a port, for example, a port image includes an image captured by a camera installed in a port, an image including at least a part of the port, etc. can do.

According to an aspect of the present specification, in the eyepiece monitoring method performed by a computing means, the method comprising: acquiring a port image including the sea and a plurality of ships using a camera installed in a port to capture an image; An artificial neural network learned using a learning set in which the input image and the class values indicating the sea, the ship, and the topographic feature, respectively, are labeled in pixels corresponding to objects including the sea, ship, and feature included in the input image. Generating a segmentation image of the objects from the port image using the port image; Tracking the plurality of ships based on first pixels assigned a class value indicating a ship of the segmentation image to obtain tracking information including location information of representative points representing each of the plurality of ships; Determining a target vessel to be guided by the eyepiece among the tracked vessels based on the tracking information; And obtaining eyepiece guide information including a bow distance, which is a distance between a bow and quay wall of the target ship, and a stern distance, which is a distance between a stern and quay wall of the target ship, based on the segmentation image. Eyepiece monitoring method comprising a may be provided.

Here, the representative point may be one point on the segmentation area of each of the plurality of ships of the segmentation image.

Here, the representative point may be a point located at the center of the segmentation area of each of the plurality of ships.

Here, the obtaining of the tracking information includes obtaining a distance between the representative point and the quay wall, and the target ship may be determined based on the distance between the representative point and the quay wall.

Here, the target ship may be determined as a ship in which a distance between the representative point and the quay wall among the plurality of ships is less than a preset value.

Here, the target ship may be determined as a ship in which the representative point is located on a predetermined area among the plurality of ships.

Here, the obtaining of the tracking information includes obtaining movement information of the representative point based on the location information of the representative point, and the target ship may be determined based on the movement information of the representative point.

Here, the target vessel may be determined based on a magnitude of an approach speed of the representative point to the quay wall among the plurality of vessels.

Here, the obtaining of the tracking information includes obtaining at least one of VTS information and AIS information of the plurality of vessels, and the target vessel may be determined based on at least one of the VTS information and AIS information. .

Here, the target ship may be determined based on a destination according to at least one of the VTS information and AIS information among the plurality of ships.

Here, the step of obtaining the tracking information includes obtaining user input information for the plurality of vessels from a user terminal, and the target vessel may be determined based on user input information for the plurality of vessels.

Here, the target ship may be determined as a ship selected according to the user input information among the plurality of ships.

Here, the target vessel may be determined as one vessel that meets the condition later when there are a plurality of vessels that meet the conditions used for determining the target vessel.

Here, the artificial neural network includes the input image and the sea, the tugboat, the ship excluding the tugboat, and the terrain feature in pixels corresponding to the objects including the sea, the tugboat, the vessel excluding the tugboat, and the terrain feature, respectively. It can be learned using a running set labeled with the indicated class values.

Here, the step of obtaining the eyepiece guide information includes extracting a pair of points corresponding to both ends of the bottom surface in contact with the sea level of the target vessel, and calculating the bow distance and the stern distance based on the pair of points. It may include the step of obtaining.

Here, the step of obtaining the eyepiece guide information includes a bow speed, which is a speed at which the bow of the target ship approaches the quay wall, based on the bow distance and the stern distance, and a speed at which the stern of the target ship approaches the quay wall. It may include the step of obtaining the stern speed.

Here, the obtaining of the eyepiece guide information may include obtaining a distance between the target vessel and the other vessel based on the segmentation image.

Here, obtaining the eyepiece guide information may include obtaining a relative speed between the target vessel and the other vessel based on a distance between the target vessel and another vessel.

Here, the step of acquiring the eyepiece guide information may include acquiring information related to the risk of collision with another ship of the ship or the quay wall.

Here, the eyepiece monitoring method includes: outputting the eyepiece guide information together with the port image; It may further include.

Here, the port image may include a panoramic image in which a plurality of port images are matched.

Further, a recording medium on which a program for executing the above-described method is recorded may be provided.

In addition, according to an aspect of the present specification, a camera installed in a port to capture an image; A port image including the sea and a ship captured by the camera is acquired, and the sea, a ship, and a terrain feature are respectively included in the input image and the pixels corresponding to the objects including the sea, the ship, and the feature included in the input image. A first pixel to which a segmentation image for the objects is generated from the port image using an artificial neural network learned using a learning set labeled with the indicated class values, and a class value indicating a vessel of the segmentation image is assigned Based on the tracking information, tracking information including location information of representative points representing each of the plurality of ships is obtained by tracking the plurality of ships, and a target ship to be guided by the eyepiece among the tracked ships is determined based on the tracking information. And a control module for acquiring eyepiece guide information including a bow distance, which is a distance between a bow and a pier of the target ship, and a stern distance, which is a distance between a stern and a quay wall of the target ship, based on the segmentation image; And a communication module for transmitting the berthing guide information of the target vessel to a remotely located terminal. Eyepiece monitoring device comprising a may be provided.

Hereinafter, an apparatus and method for monitoring the eyepiece according to an embodiment will be described.

In this specification, monitoring refers to identifying or recognizing the surrounding situation, sensing targets such as a certain area or specific object using various sensors, and providing the detection result to the user, as well as through calculations based on the detection result. It should be interpreted broadly to include, for example, providing additional information.

Image-based monitoring may mean grasping or recognizing a surrounding situation based on an image. For example, monitoring may mean acquiring an image around the vessel during operation of the vessel and recognizing other vessels or obstacles therefrom, or obtaining information for calculating eyepiece guide information when berthing or berthing the vessel.

The berthing guide information refers to the surrounding environment required for berthing, such as recognition of other ships or obstacles, understanding the port situation, whether the berth is accessible, distance and speed from the quay wall, distance and speed from other ships, and whether there are obstacles in the navigation route. It can mean information about. In the present specification, the monitoring is mainly described when berthing is performed in ships and ports, but the present disclosure is not limited thereto, and may be applied to the case of driving a vehicle or operating an aircraft.

1 is a diagram for image-based monitoring according to an embodiment. Referring to FIG. 1, image-based monitoring may include an image acquisition step (S10) and an image analysis step (S20).

The image acquisition step S10 may mean a step in which the device 10 acquires an image. Here, the type of image may be various, such as an RGB image, an IR image, a depth image, a lidar image, and a radar image, and there is no limitation. In addition, not only 2D images but also 3D images are possible.

The image analysis step S20 may mean a step of obtaining an analysis result based on an image. For example, the image analysis step S20 may include calculating eyepiece guide information through an image. Alternatively, the image analysis step S20 may refer to a step of analyzing characteristics of an object included in the image. Alternatively, the image analysis step S20 may include determining a situation indicated by the image. or,

Details of the image acquisition step (S10) and the image analysis step (S20) will be described later. Hereinafter, information acquired through the image acquisition step S10 or the image analysis step S20 is referred to as monitoring information.

2 is a diagram of an eyepiece monitoring apparatus according to an exemplary embodiment. Referring to FIG. 2, the monitoring device 10 may include a sensor module 100, a control module 200, and a communication module 300.

The sensor module 100 may sense information on a ship or around a ship and a port. The sensor module 100 may include an automatic identification system (AIS), an image generation unit, a position measurement unit, an attitude measurement unit, a casing, and the like.

The image generating unit may generate an image. The image generating unit may include a camera, a lidar, a radar, an ultrasonic detector, and the like. Examples of cameras include monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras, but are not limited thereto.

The position measuring unit may measure the position of a component included in a sensor module, such as a sensor module or an image generating unit. For example, the location measurement unit may be a Global Positioning System (GPS). In particular, Real-Time Kinematic GPS (RTK-GPS) may be used to improve the accuracy of location measurement.

The location measurement unit may acquire location information at predetermined time intervals. Here, the time interval may vary depending on the installation position of the sensor module. For example, when the sensor module is installed on a moving object such as a ship, the position measurement unit may acquire position information at short time intervals. On the other hand, when the sensor module is installed in a fixture such as a port, the position measurement unit may acquire position information at long intervals. The time interval for obtaining the location information of the location measurement unit may be changed.

The posture measurement unit may measure the posture of a component included in a sensor module, such as a sensor module or an image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measurement unit may acquire posture information at predetermined time intervals. Here, the time interval may vary depending on the installation position of the sensor module. For example, when the sensor module is installed on a moving object such as a ship, the attitude measurement unit may acquire attitude information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the attitude measurement unit may acquire attitude information at every long time interval. The time interval for obtaining the posture information of the posture measurement unit may be changed.

The casing can protect sensor modules such as an image generating unit, a position measuring unit, and a posture measuring unit.

At least one of an image generating unit, a position measuring unit, and a posture measuring unit may be present inside the casing. The casing can prevent equipment such as an image generating unit existing inside from being corroded by salt water. Alternatively, the casing may protect the equipment by preventing or mitigating the impact applied to the equipment existing therein.

A cavity may be formed inside the casing to include an image generating unit or the like therein. For example, the casing may have a rectangular parallelepiped shape with an empty inside, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like may be disposed therein.

When the image generating unit is disposed inside the casing, an opening and closing opening may be formed in one area of the casing to secure a field of view of the image generating unit, or an area of the casing may be formed of a transparent material such as glass. The image generating unit may capture images around the ship and the port through the opening or the transparent area.

The casing may be provided with a tough material to protect the image generating unit or the like from external impact. Alternatively, the casing may be provided with a material such as an alloy for seawater in order to prevent corrosion due to salt.

The casing may include equipment for removing foreign matter from the image generating unit. As an example, foreign matter adhered to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to be removed from foreign substances. As another example, foreign substances may be removed by applying water or a washer liquid through a liquid spray included in the casing, or the foreign substances may be physically removed using a wiper after application.

Debris removal equipment can be operated manually, but it can also be operated automatically. For example, the foreign matter removal equipment may be operated at predetermined time intervals. Alternatively, the foreign material removal equipment may be operated using a sensor that detects whether the foreign material has adhered to the image generating unit. Alternatively, after determining whether a foreign material has been captured in the image using the image captured by the image generating unit, the foreign material removing equipment may be operated when it is determined that the foreign material is present. Here, whether or not a foreign object is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical devices, such as including two or more identical cameras.

The control module 200 may perform image analysis. In addition, the operation of receiving various data through the sensor module 100, outputting various outputs through an output device, storing various data in a memory or acquiring various data from the memory, etc. Can be done by control. Hereinafter, various operations or steps disclosed in the embodiments of the present specification may be interpreted as being performed by the control module 200 or performed by the control of the control module 200 unless otherwise stated.

Examples of the control module 200 include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, and an on-demand There may be an Application Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC), and a combination thereof.

The communication module 300 may transmit information from the device 10 to the outside or may receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or one-way communication. For example, the device 10 may transmit information to an external output device through the communication module 300 and output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive VTS information related to a vessel or Costal Intelligent Transport System (CITS) information from a vessel traffic control system (VTS) that controls a vessel.

The sensor module 100, the control module 200, and the communication module 300 may include a control unit. The control unit may process and operate various types of information within the module, and may control other components constituting the module. The control unit may be physically provided in the form of an electronic circuit that processes electrical signals. The module may physically include only a single control unit, but may otherwise include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be provided to processors mounted on a server and a terminal physically separated from each other to cooperate through communication. Examples of the control unit include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), a state machine, and an Application Specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100, the control module 200, and the communication module 300 may include a communication unit. The modules may transmit and receive information through a communication unit. For example, the sensor module 100 may transmit information obtained from the outside through its communication unit, and the control module 200 may receive information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100, the control module 200, and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing processing of the program, or processing result data. For example, the memory may store data necessary for learning and/or inference, an artificial neural network that is being trained or learned. Memory is a nonvolatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or any other tangible nonvolatile recording medium. It can be implemented as such.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100. For example, one device 10 may include two sensor modules 100, and each sensor module 100 may again include two cameras.

3 and 4 are diagrams of an embodiment of an eyepiece monitoring apparatus according to an embodiment.

Referring to FIG. 3, the monitoring device 10 may include a sensor module 100 and a control module 200. The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110. In addition, the control unit 120 of the sensor module 100 may convert a viewpoint of an image by performing viewpoint conversion to be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 and perform image analysis, such as estimation of position/movement information and image matching, which will be described later, through the control unit 220. In addition, the control module 200 may transmit an analysis result such as location/movement information and a matched image to the cloud server through the communication unit 210. The cloud server may transmit the analysis result received from the control module 200 to a user terminal such as a smartphone, a tablet, or a PC, or may receive an instruction from the user terminal.

Referring to FIG. 4, the monitoring device 10 may include a sensor module 100. The sensor module 100 may generate an image through the camera 130 and transmit the image to a cloud server through the communication unit 110. In addition, the control unit 120 of the sensor module 100 may convert a viewpoint of an image by performing viewpoint conversion to be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as location/movement information estimation and image matching, which will be described later. In addition, the cloud server may transmit the result of image analysis to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

The device 10 illustrated in FIGS. 2 to 4 is only an example, and the configuration of the device 10 is not limited thereto.

For example, the device 10 may include an output module 400. The output module 400 may output a result of an operation performed by the control module 200 and the like. For example, the output module 400 may output an analysis result. The output module 400 may be, for example, a display, a speaker, a signal output circuit, etc., but is not limited thereto. In this case, in addition to transmitting the information to an external output device such as a user terminal and outputting the information, the external output device may output information through the output module 400.

As another example, the device 10 may not include a sensor module. In this case, the control module 200 may perform an image-based monitoring operation, such as performing image analysis by receiving information from an external sensor device. For example, the control module 200 may perform image analysis by receiving information from an AIS, camera, lidar, radar, etc. already installed in a ship or port.

In addition, the steps performed by each configuration of FIGS. 2 to 4 are not necessarily performed by the corresponding configuration and may be performed by other configurations. For example, in FIG. 3 above, it is described that the control unit 120 of the sensor module 100 performs viewpoint conversion, but the control unit 220 of the control module 200 or a cloud server may perform viewpoint conversion. .

Hereinafter, the monitoring device 10 and method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100. For example, an image may be acquired through an image generating unit included in the sensor module 100. Alternatively, as described above, an image may be obtained from an external sensor device. Images for vessel and port monitoring will generally include seas, ships, buoys, obstacles, terrain, ports, sky, buildings, and the like. Hereinafter, a description will be given of mainly analyzing and monitoring an image acquired through a visible light camera, but is not limited thereto.

Depending on the image generating unit, a field of view and a depth of field may vary. 5 is a diagram illustrating a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 5, the field of view (FOV) may mean to what extent the image is included in the image horizontally or vertically, and is generally expressed as an angle (degree). The meaning of a larger viewing angle may mean generating an image including an area having a larger width left and right, or generating an image including an area having a larger width vertically. Depth of field may mean a distance range recognized as being in focus, and a deep depth of field may mean that an image has a wide range of distances recognized as being in focus. Referring to FIG. 5, the image may include an area A1 recognized as being in focus and other areas A2 according to a depth of field DOF. Hereinafter, the area included in the image is referred to as the imaging area (A1 + A2), and the area recognized as being in focus is referred to as the effective area (A1), and image analysis and monitoring can be performed based on the effective area, but the entire imaging area. Since it may be performed based on or may be performed based on a part of the imaging area, an area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and a shallow depth of field is a wide-angle camera. Examples of cameras with a small field of view and a deep depth of field include high-magnification cameras and zoom cameras.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, a crane, a ship, etc. in a port, and there is no limitation in the number of the sensor modules 100. However, the installation location or number of the sensor modules 100 may vary according to characteristics such as type and performance. For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more from the water surface for efficient monitoring, or a plurality of sensor modules 100 may be installed to have different imaging areas. In addition, the position and posture of the sensor module 100 may be manually or automatically adjusted during or after installation.

6 and 7 are diagrams illustrating an installation position of the sensor module 100 according to an exemplary embodiment. 6 and 7, the sensor module 100 may be installed in a fixed position such as a port or land, or may be installed in a moving object such as a ship. Here, when the sensor module 100 is installed on a ship, it may be installed on a vessel to be monitored (hereinafter referred to as “target vessel”) as shown in FIG. It may be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition to this, the sensor module 100 may be installed in a drone or the like to monitor a target ship.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or in a separate location.

As described above, image analysis for image-based monitoring may include obtaining object characteristics. Examples of objects may include ships, ports, buoys, sea, terrain, sky, buildings, people, animals, fire, smoke, and the like. Examples of object characteristics may include the type of the object, the position of the object, the distance to the object, and the absolute and relative speed and speed of the object. Another example,

Image analysis for image-based monitoring may include recognizing/determining the surrounding situation. For example, the image analysis may be to determine that a fire situation has occurred from an image of a fire in the port, or to determine that an intruder has entered from an image captured by a person who entered the port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in the presence of smoke.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each of the modules 100 and 200.

8 is a diagram illustrating image analysis according to an exemplary embodiment. Referring to FIG. 8, image analysis may include an object recognition step (S210) and a position/movement information estimation step (S220 ).

Image analysis may include an object recognition step (S210). The object recognition step S210 may include recognizing an object included in the image. For example, object recognition may be determining whether an object such as a ship, a tugboat, a sea, or a port is included in the image. Furthermore, object recognition may be determining where an object exists in an image.

9 to 11 are diagrams illustrating an object recognition step according to an exemplary embodiment.

9 is an image captured by a camera, and an object may be recognized as shown in FIG. 10 or 11 through the object recognition step.

Specifically, FIG. 10 shows which object a corresponding pixel corresponds to for each pixel of an image, and is also referred to as segmentation. In this case, the object recognition step may mean a segmentation step. Through segmentation, characteristics corresponding to pixels on an image can be assigned or calculated from an image. This could be said to have been assigned or labeled a characteristic to a pixel. 9 and 10, a segmentation image as shown in FIG. 10 may be obtained by performing segmentation based on the image captured by the camera of FIG. 9. In FIG. 10, a first pixel area P1 is an image area of a pixel corresponding to a ship, a second pixel area P2 is a sea, a third pixel area P3 is a quay wall of a port, and a fourth pixel area ( P4) is a terrain, and a fifth pixel area P5 is an area on an image of a pixel corresponding to the sky.

In FIG. 10, information on the type of object corresponding to each pixel on an image is calculated by performing segmentation, but information that can be obtained through segmentation is not limited thereto. For example, properties such as the location, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, different characteristics may be expressed independently or reflected at the same time.

Table 1 is a table for labeling that simultaneously reflects information on the type and distance of an object according to an embodiment. Referring to Table 1, a class may be set in consideration of information on the type and distance of an object, and an identification value may be assigned to each class. For example, the second identification value may be assigned in consideration of the terrain, which is information about the type of object, and the near distance, which is information about the distance. Table 1 is an example of a case in which information about type and information about distance are considered together, and other information such as direction information, obstacle movement direction, speed, and route indicator may also be considered together. In addition, not all identification values need to include a plurality of pieces of information, nor do they need to include the same type of information. In some cases, for example, certain identification values contain only information about type (e.g., identification value 1 does not contain information about distance) and other identification values contain information about type and distance, etc. It can be expressed in a variety of ways.

Identification value class 0 Sky and others One Sea 2 Terrain + close range 3 Terrain + medium distance 4 Terrain + Long Range 5 Fixed obstacle + close range 6 Fixed obstacle + medium distance 7 Fixed Obstacle + Long Range 8 Dynamic Obstacle + Close Range 9 Dynamic obstacle + medium distance 10 Dynamic Obstacle + Long Range

FIG. 11 shows the presence of an object at a certain position in the image as a bounding box, and is also referred to as detection. In this case, the object recognition step may mean a detection step. Compared with segmentation, detection can be seen as detecting in the form of a box where an object is included, rather than calculating characteristics for each pixel of an image. Referring to FIGS. 9 and 11, a detection image as shown in FIG. 11 may be obtained by performing detection based on an image captured by the camera of FIG. 9. In FIG. 11, it can be seen that the ship is detected on the image and the position of the ship is represented by a rectangular bounding box (BB). Although only one object is detected in FIG. 11, two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks, and the final result may be calculated by combining the results.

An artificial neural network is a type of algorithm modeled after the structure of a human neural network, and can include one or more nodes or one or more layers including neurons, and each node can be connected through a synapse. Data (input data) input to the artificial neural network may be output (output data) through a node through a synapse, and information may be obtained through this.

Types of artificial neural networks include a convolution neural network (CNN) that extracts features using a filter, and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back to the input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), and relational networks (RN) can be applied and have limitations. It is not.

Before using the artificial neural network, a step of training is required. Alternatively, it can be trained using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step and a step of using the artificial neural network as an inference step.

Artificial neural networks can be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

12 and 13 are diagrams illustrating a learning step and an inference step of an artificial neural network according to an embodiment.

12 is an embodiment of the learning step of an artificial neural network, wherein an untrained artificial neural network receives training data or training data, outputs output data, and compares the output data and labeling data. The artificial neural network can be trained through the backpropagation of the error. Training data, output data, and labeling data may be images. The labeling data may include ground truth. Alternatively, the labeling data may be data generated through a user or a program.

13 illustrates an embodiment of an inference step of an artificial neural network, and a learned artificial neural network may receive input data and output output data. Information that can be inferred in the inference stage may vary according to the information of the learning data in the learning stage. Also, the accuracy of the output data may vary according to the learning degree of the artificial neural network.

Image analysis may include the step of estimating location/movement information (S220). The location/movement information estimating step S220 may include estimating information on the location and/or movement of at least some of the objects recognized in the object recognition step S210. Here, the location information includes an absolute location such as the coordinates of an object, a relative location from a specific reference, a distance (distance from an arbitrary point, a distance range, etc.), a direction, and the like, and the movement information includes an absolute speed, a relative speed, It may include information on the movement of the object, such as speed.

The location/movement information of the object can be used when berthing or berthing the ship. For example, it is possible to assist or guide the safe berthing or berthing of the ship by using the distance to the berth or quay wall when berthing or berthing the ship, the approach speed based on these, and the distance and relative speed with other ships.

The location/movement information of the object can be used during the operation of the ship. For example, it is possible to assist or guide the safe operation of the ship by detecting other ships or obstacles around the ship, warning of a collision or creating/recommending a route using the distance to them and their moving speed. . Alternatively, autonomous navigation could be performed based on this information.

The location/movement information of the object may be calculated based on the image. For example, it is possible to calculate the location/movement information of a ship based on an image including a ship, sea, and land as an object. Hereinafter, an object for estimating position/movement information is referred to as a target object. For example, in the above example, the ship may be the target object. In addition, there may be a plurality of target objects. For example, when estimating the position or speed of each of a plurality of ships included in the image, the plurality of ships may be target objects.

The location/movement information of the object may be expressed in a plurality of categories having a certain range. For example, distance information may be expressed in a short distance, a medium distance, a far distance, and the like, and the direction information may be expressed in a left direction, a front direction, a right direction, and the like. It will be possible to combine these and express them in the near-left or far-right distance. Movement information can be expressed in high speed, low speed, and the like.

The location/movement information of the object may be expressed as an actual distance value, a direction value, and a speed value. For example, distance information may be expressed in meters (m), direction information may be expressed in degrees, and movement information may be expressed in cm/s.

The location/movement information of the object may be estimated based on an area or point. For example, the distance between the ship and the quay wall may be estimated by calculating a distance between one point of the ship and one point of the quay wall, or may be estimated by calculating the shortest distance between a point of the ship and the quay wall. As another example, the distance between ships may be estimated by calculating a distance between one point of the first ship and one point of the second ship. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

The location/movement information of the object may be estimated based on the image pixel. As described above, when estimating location/movement information based on a point, a point on the image may correspond to a pixel. Accordingly, the position/movement information of the object may be calculated based on the interval between image pixels.

The distance information between points may be calculated based on the distance between pixels. For example, a predetermined distance may be allocated for each pixel interval, and the distance between points may be calculated in proportion to the interval between pixels. As another example, a distance between pixels may be calculated based on a coordinate value on an image of a pixel, and the distance between points may be calculated based on this.

Movement information between points may be calculated based on a change in distance information between points. In this case, movement information may be calculated based on a plurality of images or image frames. For example, movement information between points may be calculated based on a distance between points in a previous frame, a distance between points in a current frame, and a time interval between frames.

14 and 15 are diagrams for estimating location/movement information of an object according to an exemplary embodiment.

Referring to FIG. 14, the step of estimating location/movement information includes location/movement information (f1, f2) with the quay wall (OBJ2) when berthing or berthing of the ship (OBJ1), or location/movement with other ships (OBJ3, OBJ4). It may include estimating the information f3 and f4. As shown in FIG. 14, the position/movement information f1 and f2 between the ship OBJ1 and the quay wall OBJ2 may be calculated for two points of the ship OBJ1. In this case, the two points may correspond to a point where the ship OBJ1 contacts the sea. In addition, the distance between the ship OBJ1 and the quay wall OBJ2 may be the shortest distance between the two points and the quay wall OBJ2. The location/movement information (f3, f4) between the ship OBJ1 and other ships (OBJ3, OBJ4) may be location/movement information between points corresponding to the bow or stern of the ships (OBJ1, OBJ3, OBJ4). . In this way, when the location/movement information is used in the berthing or the berth of the ship to assist or guide it, it may be referred to as berthing guide information or berthing guide information.

Referring to FIG. 15, the step of estimating position/movement information may include estimating position/movement information f5, f6 with another vessel OBJ6 or an obstacle OBJ7 such as a buoy during the operation of the vessel OBJ5. have.

Port operation or management can be performed based on the data calculated in the step of estimating location/movement information. For example, when a ship collides with a fender, it is possible to predict the replacement timing of the insect repellent by calculating the amount of impact from movement information such as the speed of the ship.

In the above, an embodiment of image analysis in a method of estimating location/movement information after performing the object recognition step has been described. Alternatively, object recognition and location/movement information estimation may be performed in one step. For example, it is possible to recognize an object by performing segmentation or detection and estimate the location/movement information of the object.

When all ships on the image are monitored, the amount of data computation for acquiring the ship's berthing guide information increases, and the processing speed of the apparatus 10 may be lowered. In addition, by determining the target vessel to be monitored, berthing guide information only for the target vessel is obtained (for example, head distance, stern distance, etc.), and outputting the obtained berthing guide information, users can exclude other unnecessary information. You can be provided with information necessary for berthing. To this end, the control module 200 may perform image analysis to determine a target ship when a plurality of ships are included in at least one image acquired using the image generating unit.

16 is a flow chart for determining a target vessel according to an embodiment.

Referring to FIG. 16, the image-based monitoring may include a ship tracking step (S213), a tracking information acquisition step (S216), and a target ship determination step (S219).

According to an embodiment, the control module 200 may track a vessel on an image (S213).

Tracking refers to the act of tracking feature points (objects, tracking targets) that change temporally and temporally on an image. For example, tracking may include tracking the location, shape, movement, trajectory, information, etc. of feature points that change spatiotemporally on an image. In addition, tracking may include continuing to preserve an image of a feature point on the image.

According to an embodiment, the control module 200 may perform tracking of a ship based on the segmented image. Specifically, the control module 200 may track a ship based on an area in which a class value of a pixel corresponds to a ship in the segmented image. For example, the control module 200 may track a class value of segmented images (consecutive) of a continuously photographed image based on a pixel corresponding to a ship. Here, the control module 200 may track a representative point (pixel) of a ship within a ship area, in which a class value is formed by a pixel corresponding to a ship, between successive frames.

17 is a diagram illustrating an example of tracking of a ship according to an embodiment.

Referring to FIG. 17, the control module 200 may track the vessels of the image by tracking representative points 141, 142, and 143 representing the vessel in the continuously photographed image. For example, the control module 200 may track one point on an area corresponding to each of the ships of the segmentation image.

The representative points 141, 142, and 143 of the ship can be any feature point, but since the segmented image has a single class value, for example, the control module 200 is a point located at the center of the area corresponding to the ship. (141) can be determined as a representative point.

Of course, the control module 200 may determine the representative points (141, 142, 143) of the ship as points at other locations such as the top point, the bottom point 142, the leftmost point, and the rightmost point 143 of the ship area. In addition, the control module 200 may determine one pixel as a representative point of the ship and track the outline of the area corresponding to the ship or the area itself instead of tracking it.

Various techniques can be used for tracking, for example Kalman filter, Extended Kalman filter, Unscented Kalman filter, Particle filter, Information filter ), a histogram filter, and the like, and at least one of them may be applied for tracking.

Of course, this is not necessarily the case, and the control module 200 can track the ship based on images such as captured images, such as tracking the ship using the background image and the difference image. You can track the ship.

In addition, the tracking step (S213) may include the control module 200 tracking a vessel other than a tugboat among vessels included in the image. For example, the control module 200 may track a vessel other than a tugboat among vessels based on the segmentation image. Specifically, the control module 200 includes an input image and a ship and a terrain site excluding the sea, a tugboat, and a tugboat, respectively, in pixels corresponding to the input image and objects including the sea, tugboat, and tugboat excluding ships and terrain features. Image segmentation can be performed using an artificial neural network learned using a learning set labeled with class values indicating water, and vessels other than a tugboat can be tracked. Here, the generated segmentation image may include a pixel labeled to correspond to a ship excluding a tugboat, and the control module 200 may track a pixel corresponding to a ship excluding the tugboat.

According to an embodiment, the control module 200 may acquire tracking information of a vessel tracked on an image (S216).

The tracking information may mean information on a tracked ship, and the tracking information may be used to determine a target ship. For example, without limitation, the tracking information may include location information of a ship, movement information of a ship, VTS information of a ship, AIS information of a ship, user input information, and the like.

According to an embodiment, the step of acquiring tracking information (S216) may include the control module 200 acquiring the position information of the ship from the image. For example, the control module 200 may acquire location information of a representative point of a ship to be tracked from a continuously photographed image. Here, the location information of the representative point of the ship may be information about the distance between the representative point and the quay wall, the distance between the representative point and other objects, coordinates on the image of the representative point, and the like.

According to an embodiment, the step of acquiring tracking information (S216) may include the control module 200 acquiring movement information of the vessel from the image. For example, the control module 200 may acquire movement information of a representative point of a ship to be tracked from a continuously photographed image. Here, the control module 200 may obtain movement information of the representative point based on the location information of the representative point. Here, the movement information of the representative point of the ship may be information about an approach speed of the representative point to the quay wall, an approach speed of the representative point to another object, a change in speed of the representative point, and the like.

According to an embodiment, the step of acquiring tracking information (S216) may include acquiring voyage information such as VTS information and AIS information of a vessel for which the control module 200 is tracked. For example, the control module 200 may acquire at least one of VTS information and AIS information. Here, the control module 200 may obtain information on the destination of the ship according to the voyage information, a scheduled berthing time of the ship, and the like.

According to an embodiment, the step of acquiring tracking information (S216) may include acquiring user input information on a vessel tracked by the control module 200. For example, the control module 200 may obtain user input information used to determine a target vessel from a user terminal.

However, the acquisition of tracking information need not be limited to being performed by the above-described method, and may be performed in various ways.

According to an embodiment, the control module 200 may determine a target vessel based on the acquired tracking information (S219).

According to an embodiment, the control module 200 may determine a target ship based on the acquired position information of the ship. For example, the control module 200 may determine a target ship based on the distance between the ship and the quay wall.

18 is a diagram illustrating an example of determining a target vessel based on location information according to an embodiment.

Referring to FIG. 18, when there are a plurality of ships on an image, the control module 200 may determine a target ship to be monitored based on location information.

According to an embodiment, the control module 200 may determine a target vessel based on a distance from the quay wall of the tracked vessel. Here, the distance from the quay wall of the tracked ship may be a distance between the representative point of the tracked ship and the quay wall.

For example, the control module 200 may determine a ship whose distance from the quay wall is less than the critical distance among the tracked ships as the target ship. Referring to FIG. 18, the control module 200 may determine a vessel whose distance to the quay wall _{is smaller than a preset critical distance (d 0} ) among the tracked vessels as the target vessel. The control module 200 may determine that the vessel 151 is not a target vessel when the vessel 151 determined as the target vessel moves and is located in a place where _{the distance from the quay wall is greater than d 0.} In addition, the control module 200 may determine the vessel 152 as a target vessel when the vessel 152 moves and is located at a location closer _{to the quay wall than d 0.}

The control module 200 may variously set a critical distance used to determine a target vessel. For example, the control module 200 may set the critical distance based on the recognized size of the ship. Specifically, the control module 200 may set the critical distance larger as the size of the ship increases. In addition, the control module 200 may determine a vessel whose distance to a specific object (structure, etc.) other than the quay wall among the vessels in the image is less than the critical distance as the target vessel.

According to an embodiment, the control module 200 may determine a target vessel based on a position on an image of the tracked vessel. Here, the position on the image of the tracked ship may be a position on the image of the representative point of the tracked ship. For example, the control module 200 may determine a ship located on a predetermined specific area of the image among ships on the image as the target ship. Referring to FIG. 18, the control module 200 may determine a vessel 151 located within a predetermined square-shaped area 153 on an image among tracked vessels as a target vessel. The control module 200 may determine the vessel 152 as a target vessel when the vessel 152 moves and is located within the region 153. In addition, the control module 200 may exclude the vessel 151 from the target vessel when the vessel 151 determined as the target vessel moves and is located outside the region 153.

The control module 200 may set the region 153 in various ways. For example, the control module 200 may variously determine the size, shape, and number of the regions 153. Specifically, the control module 200 may determine the area 153 as an area around a path where a ship mainly enters from a berth where an image is captured, an area that the ship mainly berths, and the like. In addition, on the contrary, the control module 200 may determine a vessel located in an area other than the predetermined specific area 153 among the vessels on the image as the target vessel.

According to an embodiment, the control module 200 may determine a target ship based on the acquired movement information of the ship. For example, the control module 200 may determine the target ship based on the speed at which the ship approaches the quay wall.

19 is a diagram illustrating an example of determining a target vessel based on movement information according to an embodiment.

Referring to FIG. 19, when there are a plurality of ships on an image, the control module 200 may determine a target ship based on movement information.

According to an embodiment, the control module 200 may determine a target vessel based on the speed of the tracked vessel. For example, the control module 200 may determine, as a target ship, a ship having a speed (absolute value of speed) greater than a critical speed among ships in the image. Referring to FIG. 16, the control module 200 may determine a vessel 161 approaching at a speed of _{v o} , which is a value greater than a threshold speed value among the tracked vessels, as a target vessel. The control module 200 may determine that the vessel 161 is not the target vessel when the vessel 161 determined as the target vessel is stopped still in the berth after the berthing is completed. In addition, the control module 200 may determine the vessel 162 as a target vessel when the vessel 162 that was moored moves and moves at a speed (regardless of direction) greater than the critical speed.

The control module 200 may variously set the size of the critical speed used to determine the target vessel. For example, the control module 200 may determine a critical speed used for determining a target vessel based on the size of the vessel. Specifically, the control module 200 may determine that the larger the size of the vessel, the larger the critical speed.

In addition, the control module 200 may determine a target vessel based on conditions related to movement information, such as a direction, magnitude, and speed change (acceleration) of the speed among the tracked vessels. For example, the control module 200 may determine a vessel whose acceleration magnitude (absolute value of acceleration) is larger than a preset value among vessels on the image as the target vessel.

According to an embodiment, the control module 200 may determine a target ship based on the acquired navigation information of the ship. Here, the voyage information may include information related to the sailing of the ship. For example, the voyage information may be AIS information, VTS information, CITS information, and the like. The voyage information may include information such as a ship's departure point, destination, mooring point, and navigation route in relation to the ship's voyage.

For example, the control module 200 may determine the target ship based on the destination according to the navigation information of the tracked ship.

20 is a diagram illustrating an example of determining a target ship based on voyage information according to an embodiment.

Referring to FIG. 20, when there are a plurality of ships on the image, the control module 200 may determine a target ship based on navigation information.

According to an embodiment, the control module 200 may determine a target ship based on a destination according to navigation information of a tracked ship. For example, the control module 200 may determine as a target ship a ship in which a camera on which an image is captured is installed at a destination according to the voyage information among ships on the image. Referring to FIG. 17, a ship 171, which is a berth in which a camera photographing a corresponding image is installed, at a destination according to navigation information among the tracked ships, may be determined as a target ship. The control module 200 may determine that the ship 171 is not a target ship when the ship 171 determined as the target ship is departing or outgoing according to the voyage information. In addition, the control module 200 may determine the ship 172 as a target ship when the ship 172 is moored at another place and then docks with a berth in which a camera photographing a corresponding image is installed according to new voyage information. .

In addition, the control module 200 may determine a target vessel based on other conditions related to navigation information. For example, the control module 200 may determine a vessel recognized on the image as a target vessel at a time scheduled to be berthing in the berth according to the navigation information.

According to an embodiment, the control module 200 may determine a target vessel based on the acquired user input information. The user input information may include information input and received from the user's device. For example, the user input information may include information input from the user in connection with selection of a vessel to perform monitoring, selection of a berth, and the like.

21 is a diagram illustrating an example of determining a target vessel based on user input information according to an embodiment.

Referring to FIG. 21, when there are a plurality of ships on an image, the control module 200 may determine a target ship based on user input information. For example, the control module 200 may determine a vessel selected as a target for monitoring according to user input information among the tracked vessels as the target vessel. Referring to FIG. 18, the control module 200 may determine a vessel 181 selected from an indicator 193 displayed on a screen according to user input information among tracked vessels as a target vessel. The control module 200 may determine that the vessel 181 that is not selected according to the user input information is not the target vessel after the vessel 181 determined as the target vessel is docked. In addition, the control module 200 may determine the vessel 182 as a target vessel when an unselected vessel 182 is selected by a user according to user input information.

Here, the user input information used to determine the target vessel may be provided in various ways. For example, the user input information may be information for determining a target ship by selecting a ship from a list of ships entering and leaving a port. In addition, the user input information may be information for determining a target vessel for a vessel at a location selected on an output port image (eg, an output image of a user terminal).

In addition, the control module 200 may determine an image of a berth to which monitoring information is output among images captured from a plurality of berths. For example, the control module 200 may output an image of a berth selected according to user input information among images captured from a plurality of berths, and determine a ship included in the image as a target ship.

In addition, determining a target vessel based on the obtained tracking information (S216) may include the control module 200 determining one target vessel on the image.

According to an embodiment, the control module 200 may determine one of a plurality of ships meeting a condition determined as the target ship on the image as the target ship. For example, the control module 200 may determine one vessel determined as the target vessel as the target vessel later than when there are a plurality of vessels meeting the condition determined as the target vessel on the image. For example, referring to FIG. 18, when the ship 151 and the ship 152 are sequentially located within the area 153, the ship 151 was initially a target ship, but the ship 152 After) is located within the region 153, the target vessel may be determined as the vessel 152. Conversely, the control module 200 may determine one vessel determined as the target vessel as the target vessel prior to the case where there are a plurality of vessels meeting the condition determined as the target vessel on the image.

However, the determination of the target ship is not necessarily limited to being performed by the above-described method, and may be performed in various ways.

22 is a diagram illustrating acquisition of eyepiece guide information of a target ship according to an exemplary embodiment.

Referring to FIG. 22, the control module 200 may acquire eyepiece guide information of the target vessel 191 among the tracked vessels 191 and 192. Here, the berthing guide information may include a bow distance, a stern distance, a bow speed, a stern speed, a distance between a target ship and another ship, a relative speed between the target ship and another ship, and the like. The bow distance may mean the distance from the quay wall of the bow of the ship, and the stern distance may mean the distance from the quay wall of the ship's stern. The bow speed may refer to the speed at which the bow of the ship approaches the quay wall, and the stern speed may refer to the speed at which the stern of the ship approaches the quay wall.

According to an embodiment, the control module 200 may acquire a distance between the target vessel 191 and the quay wall.

For example, the control module 200 may acquire a bow distance 193 which is a distance between a bow and a quay wall of the target ship 191. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship, and obtain a distance between the one point corresponding to the bow and the quay wall. For another example, the control module 200 may obtain a stern distance 194 that is a distance between the stern of the target vessel 191 and the quay wall. Specifically, the control module 200 may determine one point corresponding to the stern of the target ship, and obtain a distance between the one point corresponding to the stern and the quay wall.

According to an embodiment, the control module 200 may extract a pair of points corresponding to both ends of the bottom of the ship in contact with the sea level. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may acquire the distance between the quay wall and one of a pair of points corresponding to both ends of the bottom surface in contact with the sea level in order to obtain the bow distance. In addition, the control module 200 may acquire the distance between the quay wall and the other of a pair of points corresponding to both ends of the bottom surface in contact with the sea level in order to obtain the stern distance.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target vessel 191 to obtain the bow distance and the stern distance of the target vessel, instead of obtaining one point or two or more points of the target vessel 191. It is possible to obtain the distance between the ship and the quay wall by extracting points, and it is also possible to obtain the distance between the ship and the quay wall based on the outline of the area or the area itself.

According to an embodiment, the control module 200 may acquire a speed at which the target vessel 191 approaches the quay wall.

For example, the control module 200 may acquire a bow speed, which is a speed at which the bow of the target ship 191 approaches the quay wall. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship 191, and obtain a speed at which one point corresponding to the bow approaches the quay wall. Here, the control module 200 may acquire a speed at which the bow of the target ship 191 approaches the quay wall based on the bow distance 193 of the target ship 191. The control module 200 compares the bow distance 193 of the target ship 191 in the current frame and the bow distance of the target ship 191 in the subsequent frame, and the speed at which the bow of the target ship 191 approaches the quay wall. Can be obtained.

For another example, the control module 200 may acquire a stern speed, which is a speed at which the stern of the target vessel 191 approaches the quay wall. Specifically, the control module 200 may determine one point corresponding to the stern of the target vessel 191 and obtain a speed at which one point corresponding to the stern approaches the quay wall. Here, the control module 200 may acquire a speed at which the stern of the target vessel 191 approaches the quay wall based on the stern distance 194 of the target vessel 191. The control module 200 compares the stern distance 194 of the target vessel 191 in the current frame with the stern distance of the target vessel 191 in a subsequent frame, and the speed at which the stern of the target vessel 191 approaches the quay wall. Can be obtained.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target vessel 191 to obtain the bow speed and stern speed of the vessel, instead of obtaining one point or two or more of the target vessel 191. It is also possible to obtain the speed at which the ship approaches the quay wall by extracting points, and it is also possible to acquire the speed at which the ship approaches the quay wall based on the outline of the area or the area itself.

According to an embodiment, the control module 200 may acquire a distance between the target vessel 191 and another vessel 192.

For example, the control module 200 may acquire a distance between a bow of the target ship 191 and another ship 192. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship 191 and obtain a distance between the one point corresponding to the bow and the other ship 192.

For another example, the control module 200 may acquire a distance between the stern of the target vessel 191 and another vessel 192. Specifically, the control module 200 may determine one point corresponding to the stern of the target vessel 191 and obtain a distance between the one point corresponding to the stern and the quay wall.

According to an embodiment, the control module 200 may extract a pair of points corresponding to the fore end and the stern end from the area corresponding to the ship. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may acquire the distance between one of the pair of points corresponding to the fore end and the stern end in the area corresponding to the ship and another ship in order to obtain the distance between the ship and another ship. I can. Here, the control module 200 may acquire a distance between other ships based on a point closer to another ship among a pair of points corresponding to the fore end and the stern end.

Of course, the control module 200 extracts points corresponding to the fore and stern of the target vessel 191 to obtain a distance between the fore and stern of the vessel and other vessels, instead of one arbitrary point of the target vessel 191 Alternatively, it is possible to obtain the distance between the ship and other ships by extracting two or more points, and it is also possible to obtain the distance between the ship and other ships based on the outline of the area or the area itself.

According to an embodiment, the control module 200 may acquire a relative speed between the target ship 191 and another ship 192.

For example, the control module 200 may acquire a speed at which a player of the target vessel 191 approaches another vessel 192. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship 191 and acquire a speed at which one point corresponding to the bow approaches the other ship 192. Here, the control module 200 may acquire the speed at which the bow of the target ship 191 approaches the other ship 192 based on the distance between the bow of the target ship 191 and the other ship 192. The control module 200 compares the distance between the bow of the target ship 191 and the other ship 192 in the current frame and the distance between the bow of the target ship 191 and the other ship 192 in the subsequent frame. It is possible to acquire the speed at which the athlete of the target ship 191 approaches another ship 192.

For another example, the control module 200 may acquire a speed at which the stern of the target vessel 191 approaches another vessel 192. Specifically, the control module 200 may determine one point corresponding to the stern of the target vessel 191 and obtain a speed at which one point corresponding to the stern approaches the other vessel 192. Here, the control module 200 may acquire a speed at which the stern of the target vessel 191 approaches the other vessel 192 based on the distance between the stern of the target vessel 191 and the other vessel 192. The control module 200 compares the distance between the stern of the target vessel 191 and the other vessel 192 in the current frame and the distance between the stern of the target vessel 191 and the other vessel 192 in the subsequent frame. The speed at which the stern of the target vessel 191 approaches another vessel 192 may be obtained.

Of course, the control module 200 extracts points corresponding to the fore and stern of the target vessel 191 to obtain the speed at which the fore and stern of the vessel approaches other vessels 192, It is also possible to obtain the relative speed between the target ship 191 and the other ship 192 by extracting one point or two or more points of, and the target ship 191 and the target ship 191 based on the outline of the area or the area itself It is okay to acquire the relative speed between the different ships 192.

23 is a flowchart illustrating outputting monitoring information according to an exemplary embodiment.

Referring to FIG. 23, eyepiece monitoring may further include outputting monitoring information (S30). The information output in the monitoring information output stage is not limited as long as it is information related to image-based monitoring, such as the characteristics of the image around the ship, the sea, the port, the type and distance/speed of the object included in the image.

The device 10 may visually output monitoring information. For example, the device 10 may output monitoring information through an output module such as a display. For another example, the device 10 may transmit monitoring information to the user's terminal through the communication module 300 and transmit a control signal to output monitoring information through the display of the user terminal.

The monitoring information output step S30 may include displaying an image acquired by using the image generating unit in the image acquisition step. In addition, it may include displaying various images related to image-based monitoring, such as an image that has undergone a pre-processing step, an image after segmentation or detection, and an image after viewpoint conversion.

In addition, the monitoring information output step S30 may include displaying the position/movement information estimated in the image analysis step by the control module 200 through an output module or a user terminal.

24 is a diagram illustrating an example of outputting monitoring information according to an embodiment.

Referring to FIG. 24, the image and eyepiece guide information may be displayed together. As shown in FIG. 24, the displayed eyepiece guide information may include a bow distance, bow speed, aft distance, and aft speed of the target ship.

The monitoring information output step S30 may include providing information to the user in other ways, such as outputting sound or vibration in addition to the visual display by the device 10. For example, the control module 200 may sound a warning sound, such as when the target ship is in danger of colliding with the quay wall, other ships, obstacles, etc., or when the approach speed to the quay wall is higher than the reference speed when berthing, or when the ship deviates from the route and operates. It can be output through an output module or a user terminal.

Image-based monitoring may include surveillance. Here, the surveillance may mean providing security-related information such as monitoring an intruder or monitoring access to a port of an unregistered vessel, and information on an emergency situation such as a fire occurrence to a user.

The device 10 may monitor an intruder based on whether or not a person is included in the image and when the image is captured. For example, the control module 200 may determine that an intruder exists when a person is included in a port image captured at a point in time when a work is not performed in the port.

The device 10 may monitor the vessel based on whether the image contains a vessel. For example, when a vessel not registered in AIS is detected, the control module 200 may monitor a vessel in a manner that provides information on this to a user.

In addition, the device 10 may perform surveillance by detecting a person or a ship based on an image through segmentation or detection.

25 to 27 are diagrams illustrating another example of outputting monitoring information according to an exemplary embodiment.

Referring to FIGS. 25 and 26, the image and eyepiece guide information may be displayed together, and when there are a plurality of target vessels, a plurality of eyepiece guide information may be output.

As shown in FIG. 25, the position where the eyepiece guide information is output may correspond to the position of the target vessel on the image. For example, the eyepiece guide information of the left target ship may be displayed on the left, and the eyepiece guide information of the right target ship may be displayed on the right.

In addition, as shown in FIG. 26, the eyepiece guide information may be output together with the identifier of the target vessel. For example, a random identifier of'Ship 1'to the left target vessel on the image and'Ship 2'to the right target vessel on the image may be assigned, and the assigned identifier and berthing guide information of each target vessel may be output together. . In addition, the assigned identifier and eyepiece guide information may be displayed around the target vessel on the image.

In addition, referring to FIG. 27, the image and the eyepiece guide information may be displayed together. In the case of a plurality of vessels, if only one target vessel is determined, only the eyepiece guide information of the target vessel may be output. For example, when the target ship is determined to be a left ship in which berthing is in progress, only berthing guide information of the left ship determined as the target ship may be displayed on the screen.

In the above, we have looked at the image-based monitoring based on a single image. In addition to this, image-based monitoring may be performed based on a plurality of images. When image analysis is performed based on a plurality of images, the total monitoring area of the image-based monitoring device 10 may increase or the accuracy of monitoring may be improved.

28 is a diagram illustrating image-based monitoring based on a plurality of images, according to an exemplary embodiment.

Referring to FIG. 28, the image acquisition step may include a first image acquisition step (S11) and a second image acquisition step (S12), and the image analysis step (S20) is acquired in the first image acquisition step (S11). An image analysis may be performed based on the first image and the second image acquired in the second image acquisition step S12.

After generating one image based on a plurality of images, image analysis may be performed. For example, the control module 200 may generate a panoramic image in which a plurality of images are matched, and perform image analysis of the panoramic image. Specifically, the control module 200 may match or fuse the first image and the second image to generate a matched image or a fused image, and perform image analysis based on the generated matched image or fused image.

Alternatively, a final analysis result may be calculated based on a result of performing image analysis based on each of the plurality of images. For example, after performing image analysis from a first image to obtain first monitoring information and performing image analysis from a second image to obtain second monitoring information, a final Monitoring information can be obtained.

As an example of a method of obtaining final monitoring information from a plurality of monitoring information, there may be a method of calculating final monitoring information by considering a plurality of monitoring information by weight.

Alternatively, the final monitoring information may be calculated based on whether a plurality of monitoring information does not match each other or whether the difference is equal to or greater than a threshold equal to a specific value (hereinafter referred to as “whether or not an error has occurred”). For example, based on whether an error has occurred, the final monitoring information is calculated by considering a plurality of monitoring information by weight, or by prioritizing specific monitoring information among a plurality of monitoring information to calculate the final monitoring information, or to perform other monitoring of specific monitoring information. There may be a method of correcting with information or ignoring the monitoring information, but is not limited thereto.

The plurality of images may be the same type of image. For example, when the sensor module includes two identical image generating units, or two identical sensor modules including one image generating unit are disposed to perform image-based monitoring, the first image and the second image are of the same type. Can be

Monitoring areas of the plurality of images may be different from each other. For example, the first image may monitor a short distance from the image generating unit, and the second image may monitor a far distance. Alternatively, the first image may monitor the left side from the image generating unit, and the second image may monitor the right side.

In addition, the device 10 may output monitoring information acquired by analyzing a plurality of images together with an appropriate image.

The device 10 may output monitoring information obtained by analyzing a plurality of images together with an image including a target vessel among the plurality of images. For example, the control module 200 analyzes a plurality of images acquired by a plurality of cameras that at least partially capture images of different areas, and then the target ship among the plurality of images through an output module or a user terminal The captured first image may be output together with the first eyepiece guide information calculated from the first image.

29 and 30 are diagrams illustrating another example of outputting monitoring information according to an embodiment.

Referring to FIGS. 29 and 30, when a target ship is determined after analyzing a plurality of images, the eyepiece guide information of the target ship may be displayed together with an image including the target ship among the plurality of images. For example, when the target ship is determined to be'Ship 1', berthing guide information of'Ship 1'may be displayed together with an image of a berth including'Ship 1'. In addition, when the target ship is determined to be'Ship 3', the berthing guide information of'Ship 3'may be displayed together with the image of the berth including'Ship 3'.

The device 10 may output monitoring information acquired by analyzing a plurality of images together with an image of a destination of a ship among the plurality of images. For example, the control module 200 analyzes a plurality of images acquired by a plurality of cameras that capture an image of an area in which at least a portion overlaps, and then, when the target ship is photographed with a plurality of images, the plurality of images Among them, a control signal may be generated so that an image obtained using a camera installed at a destination according to the navigation information of the target ship and the eyepiece guide information calculated from the image are output together.

31 and 32 are diagrams illustrating another example of outputting monitoring information according to an embodiment.

31 and 32 show that eyepiece guide information is displayed together with a first image and a second image, which are images of each berth that are adjacent to each other, obtained by a plurality of cameras that capture images of an area where at least a portion overlaps.

31 and 32, the ship 252 is photographed together on two images at the same time point. When the destination of the ship 252 is not the berth of the first image but the berth of the second image, it is preferable to output the eyepiece guide information of the ship 252 passing toward another place together with the first image other than the second image. I can't. As shown in FIG. 31, it is preferable that the first image is output along with the berthing guide information of the ship 251 where the berth of the first image is destination according to the navigation information. In addition, as shown in FIG. 32, it is preferable that the berth guide information of the ship 252 to which the berth of the second image is destination is output together with the second image according to the voyage information.

Of course, monitoring information obtained by analyzing a plurality of images may be output together with an arbitrary image among a plurality of images, and is not limited to the above description.

The image-based monitoring may further include a viewpoint conversion step.

In general, an image generated by an image generating unit such as a camera may appear as a perspective view. Converting this into a top view, a side view, or another perspective may be referred to as a viewpoint conversion. Of course, a top view or a side view image may be converted to another view, and the image generation unit may generate a top view image or a side view image, and in this case, the view conversion may not need to be performed.

33 and 34 are diagrams for view conversion according to an embodiment.

Referring to FIG. 33, another perspective view image may be obtained through a viewpoint conversion of the perspective view image. Here, viewpoint conversion may be performed so that the quay wall OBJ8 is positioned along the horizontal direction (left and right directions on the image) on the image. Referring to FIG. 34, a top view image may be obtained through a viewpoint conversion of a perspective view image. Here, the top view image may be a view looking down at the sea level in a direction perpendicular to the sea level. Also, as in FIG. 33, viewpoint conversion may be performed so that the quay wall OBJ9 is positioned along the horizontal direction on the image.

After acquiring the image and performing viewpoint conversion, image analysis to obtain necessary information for the ship's berthing guide can be performed. According to an example, the acquisition of the eyepiece guide information may be performed based on the segmented image in which the viewpoint is converted. Specifically, the viewpoint may be converted from a segmentation image of a perspective view to a top view segmentation image, and eyepiece guide information of a ship may be obtained based on an area in which a class value of a pixel corresponds to a ship among the top view segmented images. Of course, the segmentation image converted from viewpoint may not be a top view, but may be a segmentation image in which the viewpoint is converted into various viewpoints such as a side viewpoint.In addition, instead of obtaining the location/movement information of the ship by extracting two points, Alternatively, it is also possible to acquire the location/movement information of the ship based on the area itself.

Of course, this is not necessarily the case, and the viewpoint may be performed based on an image such as a converted photographed image.

In addition, it is possible to output monitoring information such as displaying an image to a user after performing a viewpoint conversion after acquiring an image. In this case, information about the surrounding situation can be more easily provided to the user through the viewpoint conversion.

Transformation of the viewpoint of an image can be performed in various ways.

As an example of viewpoint transformation, inverse projection transformation (IPM) may be performed. In a two-dimensional image, light reflected from a subject in a three-dimensional space is incident on the image sensor through the lens of the camera, and the relationship between the two-dimensional and three-dimensional depends on the image sensor and the lens, for example, Equation 1 and It can be expressed together.

Here, the matrix on the left side is a 2D image coordinate, the first matrix on the right side is an intrinsic parameter, the second matrix is an extrinsic parameter, and the third matrix is 3D coordinates. Specifically, fx and fy denote a focal length, cx and cy denote a principal point, and r and t denote rotation and translation transformation parameters, respectively.

It is possible to change the viewpoint by projecting a 2D image onto an arbitrary plane on the 3D through a back projection transformation. For example, a perspective view image may be converted into a top view image through reverse projection transformation, or another perspective view image may be converted.

Internal parameters may be required for viewpoint conversion. As an example of how to obtain an internal parameter, the Zhang method can be used. The Zhang method is a type of polynomial model. It is a method of obtaining internal parameters by photographing a grid plate with a known size of the grid at various angles and distances.

Information on the position and/or posture of the image generating unit/sensor module that has captured an image may be required for viewpoint conversion. This information can be obtained from the position measuring unit and the posture measuring unit.

Alternatively, information on the position and/or posture may be obtained based on the position of the fixture included in the image. For example, at a first viewpoint, the image generating unit may generate a first image including a target fixture, which is disposed in a first position and/or a first posture, and is a fixed object such as a terrain or a building. Thereafter, at a second viewpoint, the image generating unit may generate a second image including the target fixture. By comparing the position of the target fixture on the first image and the position of the target fixture on the second image, a second position and/or a second posture, which is the position and/or posture of the image generating unit at the second viewpoint, is calculated. can do.

In addition, the accuracy of image analysis may vary depending on the selection of the reference plane during the viewpoint conversion. For example, when converting a perspective view image to a top view image, the accuracy of image analysis based on the top view image may vary according to the height of the reference plane. In order to accurately calculate the distance between objects on the sea level, it may be preferable that the reference plane is sea level when the viewpoint is converted. Since the height of the sea level may change over time, it may be desirable to perform a viewpoint conversion in consideration of the height of the sea level to improve the accuracy of image analysis.

The above-described viewpoint transformation method is only an example, and viewpoint transformation may be performed in a different method. The viewpoint transformation information includes viewpoint transformation such as information on the matrix, parameters, coordinates, position and/or posture of Equation 1 above. Include the information you need for it.

Image-based monitoring may include a pre-processing step. Preprocessing refers to all kinds of processing performed on an image, image normalization, image equalization, histogram equalization, image resize, and upscaling to change the resolution/size of an image. And downscaling, cropping, noise removal, and the like. Here, the noise may include fog, rain, water droplets, sea clutter, fine dust, direct sunlight, salt, and combinations thereof, and removing noise means removing or reducing noise components included in the image. Can include.

Looking at the normalization as an example of the preprocessing, normalization may mean obtaining an average of RGB values of all pixels of the RGB image and subtracting this from the RGB image.

Looking at defogging as another example of the pre-processing, fog removal may mean converting an image of a foggy area to look like an image of a clear area through pre-processing. 13 is a diagram illustrating fog removal according to an exemplary embodiment. Referring to FIG. 13, an image of a foggy area as shown in (a) of FIG. 13 may be converted into an image from which the fog is removed as shown in (b) of FIG. 13 through fog removal.

Looking at the removal of water droplets as another example of the pre-processing, the water droplet removal may mean converting the water droplets formed on the front of the camera to appear as if the water droplets have been removed through pre-processing in a photographed image.

According to an embodiment, after the image acquisition step S10, the image analysis step S20 may be performed through a pre-processing step. For example, image analysis may be performed after performing pre-processing on an image acquired using the image generating unit. Image analysis may be facilitated or accuracy may be improved through image preprocessing.

Image preprocessing can be performed through an artificial neural network. For example, the image containing the noise can be input into the artificial neural network by inputting an image of a foggy area into an artificial neural network and transformed to look like an image of a clear area. I can. Examples of artificial neural networks include GAN, but are not limited thereto.

Alternatively, image preprocessing may be performed using an image mask. For example, by applying an image mask to an image of a foggy area, the clear area can be converted to look like a captured image. Here, examples of the image mask include an inverse convolution filter, a sharpen filter, and the like, and an image mask may be generated through an artificial neural network such as GAN, but is not limited thereto.

In the above, we have looked at the case of performing image analysis after preprocessing the image. Alternatively, it may be possible to perform image analysis including pre-processing. For example, when the image analysis step includes segmentation or detection, the result of performing segmentation or detection of an image including noise may be implemented to be equivalent to a result of performing segmentation or detection of an image that does not contain noise.

35 is a flowchart of image-based monitoring according to an embodiment.

Referring to Figure 35, the eyepiece monitoring is an image acquisition step (S1010), segmentation step (S1210), ship tracking step (S1213), tracking information acquisition step (S1216), target ship determination step (S1219), eyepiece guide information acquisition step It may include (S1220) and the step of outputting monitoring information (S1030). Each step can be implemented as described above.

The device 10 may acquire an image including the object through the image acquisition step S1010. The device 10 may obtain a segmentation image by performing the segmentation operation S1210 based on the image. The apparatus 10 may track at least one vessel recognized based on the segmentation image by performing the vessel tracking step S1213. The device 10 may acquire tracking information by performing the tracking information acquisition step (S1216). Here, the tracking information may include at least one of location information of a ship, movement information of a ship, navigation information of a ship, and user input information. The apparatus 10 may determine a target vessel among the tracked vessels based on the acquired tracking information by performing the target vessel determination step (S1219). The apparatus 10 may obtain the determined eyepiece guide information of the target vessel based on the segmentation image by performing the step of obtaining eyepiece guide information (S1220 ). Here, the eyepiece guide information may include at least one of a bow distance and a stern distance of the target vessel. The device 10 may output the eyepiece guide information together with the image by performing the monitoring information output step (S1030).

The embodiment of FIG. 35 is only an example, and image-based monitoring may be performed in a different way.

For example, some steps may not be performed in the embodiment of FIG. 35. Referring to FIG. 35, the step of outputting monitoring information (S1030) may not be performed.

As another example, another step may be added in the embodiment of FIG. 35, such as adding an image matching step or adding an image viewpoint conversion step.

As another example, some steps in the embodiment of FIG. 35 may be replaced with other steps, such as the segmentation step being replaced by the detection step.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

In the above, the configuration and features of the present invention have been described based on the embodiments, but the present invention is not limited thereto, and that various changes or modifications can be made within the spirit and scope of the present invention to those skilled in the art. It is obvious, and therefore, such changes or modifications are found to be within the scope of the appended claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication department
220: control unit
300: communication module

Claims (23)

In the eyepiece monitoring method performed by the computing means,
Acquiring a port image including the sea and a ship using a camera installed in the port to capture an image;
An artificial neural network learned using a learning set in which the input image and the class values indicating the sea, the ship, and the topographic feature, respectively, are labeled in pixels corresponding to objects including the sea, ship, and feature included in the input image. Generating a segmentation image of the objects from the port image using the port image;
Tracking a plurality of ships based on first pixels assigned a class value indicating a ship of the segmentation image to obtain tracking information including location information of representative points representing each of the plurality of ships;
Determining a target vessel to be guided by the eyepiece among the tracked vessels based on the tracking information; And
Obtaining eyepiece guide information including a bow distance, which is a distance between a bow and a pier of the target ship, and a stern distance, which is a distance between a stern and a quay wall of the target ship based on the segmentation image; Including,
The step of obtaining the tracking information
Including the step of obtaining at least one of the VTS information and AIS information of the plurality of ships,
The target vessel is determined based on at least one of the VTS information and AIS information
Eyepiece monitoring method.
The method of claim 1,
The representative points above are,
One point on the segmentation area of each of the plurality of ships of the segmentation image
Eyepiece monitoring method.
The method of claim 2,
The representative points above are,
A point located at the center of the segmentation area of each of the plurality of ships,
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the tracking information
Including the step of obtaining a distance between the representative point and the quay wall,
The target ship is determined based on the distance between the representative point and the quay wall
Eyepiece monitoring method.
The method of claim 4,
The target ship is determined as a ship in which the distance between the representative point and the quay wall among the plurality of ships is less than a preset value.
Eyepiece monitoring method.
The method of claim 1,
The target ship is determined as a ship in which the representative point is located on a preset area among the plurality of ships.
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the tracking information,
Including the step of obtaining movement information of the representative point based on the location information of the representative point,
The target ship is determined based on the movement information of the representative point
Eyepiece monitoring method.
The method of claim 7,
The target ship is determined based on the magnitude of the approach speed of the representative point to the quay wall among the plurality of ships.
Eyepiece monitoring method.
delete The method of claim 1,
The target ship is determined based on a destination according to at least one of the VTS information and AIS information among the plurality of ships.
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the tracking information
Including the step of obtaining user input information for the plurality of ships from a user terminal,
The target ship is determined based on user input information for the plurality of ships
Eyepiece monitoring method.
The method of claim 11,
The target ship is determined as a ship selected according to the user input information among the plurality of ships.
Eyepiece monitoring method.
The method of claim 1,
The target ship is determined as one ship that meets the conditions later when there are a plurality of ships meeting the conditions used to determine the target ship.
Eyepiece monitoring method.
The method of claim 1,
The artificial neural network,
Class values indicating the sea, the tugboat, the ship excluding the tugboat and the terrain feature are labeled on the pixels corresponding to the input image and the objects including the sea, the tugboat, and the vessel excluding the tugboat and the feature included in the input image, respectively. Learned using one running set,
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the eyepiece guide information,
Extracting a pair of points corresponding to both ends of the bottom surface of the target ship in contact with the sea level, and
Comprising the step of obtaining the bow distance and the stern distance based on the pair of points
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the eyepiece guide information,
And acquiring a bow speed that is a speed at which the bow of the target ship approaches the quay wall and a stern speed that is a speed at which the stern of the target ship approaches the quay wall based on the bow distance and the stern distance.
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the eyepiece guide information,
Comprising the step of obtaining a distance between the target vessel and another vessel based on the segmentation image
Eyepiece monitoring method.
The method of claim 17,
The step of obtaining the eyepiece guide information,
Comprising the step of obtaining a relative speed between the target vessel and the other vessel based on the distance between the target vessel and another vessel
Eyepiece monitoring method.
The method of claim 1,
The step of obtaining the eyepiece guide information,
Comprising the step of acquiring information related to the risk of collision with another vessel of the vessel or the quay wall
Eyepiece monitoring method.
The method of claim 1,
Outputting the eyepiece guide information together with the port image; Further comprising
Eyepiece monitoring method.
The method of claim 1,
The above port image is
Including a panoramic image in which a plurality of port images are matched
Eyepiece monitoring method.
A recording medium on which a program for executing the method according to any one of claims 1 to 8 and 10 to 21 is recorded.
A camera installed in a port to capture an image;
A port image including the sea and a ship captured by the camera is acquired, and the sea, a ship, and a terrain feature are respectively included in the input image and the pixels corresponding to the objects including the sea, the ship, and the feature included in the input image. A first pixel to which a segmentation image for the objects is generated from the port image using an artificial neural network learned using a learning set labeled with the indicated class values, and a class value indicating a vessel of the segmentation image is assigned A plurality of vessels are tracked based on the field to obtain tracking information including location information of representative points representing each of the plurality of vessels, and a target vessel to be guided by the eyepiece among the tracked vessels is determined based on the tracking information. A control module for acquiring eyepiece guide information including a bow distance, which is a distance between a bow and a pier of the target ship, and a stern distance, which is a distance between a stern and a quay wall of the target ship, based on the segmentation image; And
A communication module for transmitting the berthing guide information of the target vessel to a remotely located terminal; Including,
The control module
Acquiring at least one of the VTS information and AIS information of the plurality of vessels to obtain the tracking information, and determining the target vessel based on at least one of the obtained VTS information and AIS information
Eyepiece monitoring device.

Images