KR20220055556A - Device and method for monitoring ship and port - Google Patents

Abstract

The present invention relates to a device and method for monitoring a ship and a harbor. According to an embodiment of the present invention, provided is a harbor monitoring method performed by computing means. According to an embodiment of the present invention, a distance calculation method based on feature points of the ship includes the steps of: acquiring sensor data (S1000); generating ship feature points (S2000); and calculating a distance based on the feature points (S2000). According to the present invention, the feature points of the ship are extracted by efficiently converging camera and Lidar data, and information for monitoring can be accurately provided using the same.

Description

Vessel and port monitoring device and method {DEVICE AND METHOD FOR MONITORING SHIP AND PORT}

The present invention relates to a vessel and port monitoring apparatus and method, and more particularly, to a vessel and port monitoring apparatus and method for estimating lidar data related to a vessel using a camera image.

Many accidents occur in the operation of ships and berthing and berthing in ports, and it is known that the main cause of the accidents is human negligence. Here, the negligence of navigation is mainly caused by the fact that the situation around the ship or in the port cannot be accurately monitored through the naked eye. Currently, various types of obstacle sensors are used to supplement this problem, but there are still limitations.

Recently, a technology for monitoring the situation around a ship or in a port through video has been developed, but the use of lidar is also being used to improve the monitoring accuracy. However, in the case of lidar used in a port, it has a problem in that it is difficult to obtain the minimum lidar data required for monitoring because it has low performance due to cost problems and the size of the vessel to be monitored is large.

Accordingly, there is a need to develop a technology for estimating additional lidar data required for monitoring from the obtained lidar data.

One problem to be solved by the present application is to provide an apparatus and method for monitoring a ship and a port that efficiently fuse data of a camera and a lidar.

An object of the present application is to provide an apparatus and method for monitoring a ship and a port for estimating lidar data related to a ship using a camera image.

The problem to be solved of the present application is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to an embodiment, as a port monitoring method performed by a computing means, preparing an artificial neural network to be trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, wherein the first index is located in pixels corresponding to an area of the vessel in the plurality of training images. labeled, and the second index is labeled to pixels corresponding to the region of the sea in the plurality of training images, obtaining an image captured by a camera, a viewing angle that at least partially overlaps a viewing angle of the camera Acquiring lidar data including a plurality of lidar points obtained by a lidar sensor having generating a transformed image by projecting a vessel region onto a specific reference plane, a lidar point associated with a lidar beam reflected from the target vessel in consideration of pixel positions of pixels included in the projected target vessel region of the transformed image selecting , generating estimated lidar points using the selected lidar points, determining a characteristic point of the target vessel from among the selected lidar points and the generated estimated lidar points; A port monitoring method including calculating a distance between the target vessel and another object by using a feature point may be provided.

According to an embodiment there is a method for monitoring a port performed by computing means, comprising: acquiring an image captured by a camera; acquiring lidar data including points, generating a segmentation image from the image, wherein the segmentation image includes a vessel region corresponding to a vessel, wherein the vessel region includes a side region corresponding to the side of the vessel and comprising a deck area corresponding to a deck area of the vessel, projecting the vessel area onto any first and second reference planes to produce at least one of a first transformed image or a second transformed image; At least one of the first transformed image and the second transformed image is a lidar point related to a lidar beam reflected from the vessel among the plurality of lidar points, the height of the first reference plane and the second reference plane being different from each other. aligning to one, interpolating and/or extrapolating the lidar points associated with the lidar beam reflected from the vessel according to the shape of the vessel region projected in at least one of the first transformed image and the second transformed image generating an estimated lidar point; determining a characteristic point of the vessel among lidar points and the estimated lidar point associated with a lidar beam reflected from the vessel; and using the characteristic point to be different from the target vessel A port monitoring method comprising calculating a distance to an object may be provided.

According to an embodiment, as a port monitoring method performed by a computing means, preparing an artificial neural network that is learned using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images step - the object information has a first index indicating that the type of object is a vessel and a second index indicating that the type of object is a sea, wherein the first index includes pixels corresponding to an area of the vessel in the plurality of training images and the second index is labeled in pixels corresponding to the region of the sea in the plurality of training images - a maritime image comprising a first vessel and a second vessel, the decks of which are located at different heights from each other obtaining, detecting a first vessel region corresponding to the first vessel and a second vessel region corresponding to the second vessel from the acquired sea image by using the artificial neural network; obtaining the position of the first vessel and the position of the second vessel based on a first height corresponding to the deck height and a second height corresponding to the deck height of the second vessel and the position of the first vessel and A port monitoring method comprising a; calculating a distance between the first vessel and the second vessel based on the position of the second vessel may be provided.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to an embodiment of the present application, it is possible to efficiently fuse data of a camera and a lidar to extract a feature point of a ship, and use it to accurately provide information for monitoring.

According to another embodiment of the present application, it is possible to solve the problem of lack of lidar data by estimating lidar data related to a ship using a camera image.

The effect of the present application is not limited to the above-described effects, and the effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings.

1 is a view of a port monitoring apparatus according to an embodiment.
2 and 3 are diagrams related to an embodiment of a monitoring device according to an embodiment.
4 is a view of a viewing angle and a depth of field according to an exemplary embodiment.
5 and 6 are views of an installation position of a sensor module according to an embodiment.
7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.
10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
12 is a view for explaining eyepiece guide information according to an embodiment.
13 is a view for explaining information on the berthing guide between the ship and the quay wall according to an embodiment.
14 is a view for explaining a method of obtaining eyepiece guide information according to an embodiment.
15 is a diagram of viewpoint transformation according to an embodiment.
16 is a flowchart of a method for calculating a distance based on a feature point of a ship according to an exemplary embodiment.
17 is an example of acquisition of sensor data according to an embodiment.
18 is a flowchart of a method for generating feature points of a ship using an image in which a viewpoint is converted according to an exemplary embodiment.
19 is an example of a converted image according to an embodiment.
20 is a diagram for explaining calculation of a distance between ships according to an exemplary embodiment.
21 is a flowchart of a method for estimating a lidar point according to an embodiment.
22 is an example of registration of a camera image and lidar data according to an embodiment.
23 is a flowchart of a method for generating feature points of a ship in consideration of an estimated lidar point according to an embodiment.
24 is an example of lidar points associated with a lidar beam reflected from a vessel according to an embodiment.
25 and 26 are diagrams for explaining generation of an estimated lidar point using a lidar point matched to a ship area projected on a reference plane according to an embodiment.
27 is a flowchart of a method for acquiring feature points of a ship using images in which a ship area is projected on a plurality of reference planes, according to an exemplary embodiment.
28 and 29 are diagrams for explaining the alignment of images projected on mutually different arbitrary reference planes according to an exemplary embodiment.

The embodiments described herein are for clearly explaining the spirit of the present invention to those of ordinary skill in the art to which the present invention pertains, so the present invention is not limited by the embodiments described herein, and the present invention It should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as widely used general terms as possible in consideration of the functions in the present invention, but they may vary depending on the intention, custom, or emergence of new technology of those of ordinary skill in the art to which the present invention belongs. can However, if a specific term is defined and used in an arbitrary sense, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the terms and the contents of the entire specification, rather than the names of simple terms.

The drawings attached to this specification are for easily explaining the present invention, and the shapes shown in the drawings may be exaggerated as necessary to help understand the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

In the present specification, the sea level height should be broadly interpreted to include not only the absolute sea level height, but also the relative sea level height and the relative sea level height compared with the average sea level height of a specific sea area. For example, sea level height may vary with changes in sea level height, such as the distance between sea level and a quay wall, the distance between sea level and a monitoring device (e.g., an image generating unit), and the length of objects exposed above sea level. may include

Monitoring in the present specification refers to understanding or recognizing a surrounding situation, detecting a detection target such as a certain area or a specific object using various sensors and providing the detection result to the user through calculations based on the detection result, etc. It should be construed broadly to include, for example, providing additional information.

Image-based monitoring may mean identifying or recognizing a surrounding situation based on an image. For example, monitoring may mean acquiring information for calculating berthing guide information during berthing or berthing of a ship or recognizing other ships or obstacles therefrom by acquiring images around the ship when the ship is operating.

Berthing guide information is information about the environment, such as recognizing other ships or obstacles, understanding the port situation, whether the berth is accessible, the distance and speed from the quay wall, the distance and speed from other ships, and the presence of obstacles in the navigation route. can mean In this specification, although mainly described for monitoring when berthing is performed in a ship and a port, it is not limited thereto and may be applied to the case of driving of a vehicle, operation of an aircraft, and the like.

According to an aspect of the present invention, as a port monitoring method performed by a computing means, an artificial neural network trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images preparing - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, wherein the first index corresponds to an area of the vessel in the plurality of training images labeled pixels, wherein the second index is labeled to pixels corresponding to the region of the sea in the plurality of training images, obtaining an image captured by a camera, the angle of view of the camera and at least in part Acquiring lidar data including a plurality of lidar points obtained by lidar sensors having overlapping viewing angles, detecting a target vessel area corresponding to a target vessel from the acquired image using the artificial neural network , generating a transformed image by projecting the target vessel region to a specific reference plane, taking into account pixel positions of pixels included in the projected target vessel region of the transformed image, related to the lidar beam reflected from the target vessel selecting lidar points; generating estimated lidar points using the selected lidar points; determining a feature point of the target vessel among the selected lidar points and the generated estimated lidar points; A port monitoring method may be provided, comprising: calculating a distance between the target vessel and another object by using the step and the feature point.

In an embodiment, the specific reference plane may include a reference plane at sea level.

In an embodiment, the estimated lidar point may be generated from lidar points located in a line where the target vessel and the sea level meet in the converted image among the selected lidar points.

In an embodiment, the estimated lidar point may be generated by extrapolating and/or interpolating in consideration of the relative positions of the lidar points located on the line where the target vessel and the sea level are in contact.

In an embodiment, the specific reference plane may comprise a reference plane at the deck level of the target vessel.

In an embodiment, the estimated lidar point may be generated from lidar points located on a line in which the side surface of the target vessel and the deck contact the converted image among the selected lidar points.

In an embodiment, the estimated lidar point may be generated by extrapolating and/or interpolating in consideration of the relative positions of the lidar points located on the line in which the side surface of the target vessel and the deck are in contact.

In an embodiment, the specific reference plane comprises any first and second reference planes, the transformed image comprises a first transformed image and a second transformed image, and the first transformed image comprises the target The vessel region may be generated by projecting the first reference plane, and the second transformed image may be generated by projecting the target vessel region onto the second reference plane.

In an embodiment, the first transformed image and the second transformed image may be aligned based on positions of the plurality of lidar points.

In an embodiment, the feature point of the target ship may include a point corresponding to at least one of a bow and a stern of the target ship.

In an embodiment, the feature point of the target vessel may include at least one arbitrary point among LiDAR points located on a line in which the target vessel and the sea level of the converted image are in contact.

In an embodiment, the feature point of the target vessel may include lidar points located at both ends of the target vessel area of the converted image.

In an embodiment, the artificial neural network may further include a third index indicating that the type of the object is the side of the vessel, and a fourth index indicating that the type of the object is the deck of the vessel.

According to another aspect of the present invention, there is provided a method for monitoring a port performed by a computing means, comprising: acquiring an image captured by a camera; obtaining lidar data including lidar points of, generating a segmentation image from the image, wherein the segmentation image includes a vessel region corresponding to a vessel, the vessel region corresponding to a side of the vessel comprising a side area and a deck area corresponding to a deck area of the vessel, projecting the vessel area onto any first and second reference planes to produce at least one of a first transformed image or a second transformed image the first transform image and the second transform of LiDAR points related to the LIDAR beam reflected from the vessel among the plurality of LIDAR points, the heights of the first reference plane and the second reference plane being different from each other. aligning to at least one of the images, interpolating and/or interpolating lidar points associated with the lidar beam reflected from the vessel according to the shape of the vessel area projected in at least one of the first transformed image and the second transformed image; or extrapolating to generate an estimated lidar point, determining lidar points associated with a lidar beam reflected from the vessel and a characteristic point of the vessel among the estimated lidar points, and using the characteristic point to determine the target A port monitoring method may be provided that includes calculating a distance between a vessel and another object.

According to another aspect of the present invention, as a port monitoring method performed by a computing means, an artificial neural network trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images. preparing - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, wherein the first index corresponds to a region of the vessel in the plurality of training images and the second index is labeled with pixels corresponding to the region of the sea in the plurality of training images, comprising a first vessel and a second vessel, the decks of which are located at different heights from each other. obtaining a sea image, detecting a first vessel region corresponding to the first vessel and a second vessel region corresponding to the second vessel from the acquired sea image using the artificial neural network; acquiring the position of the first vessel and the position of the second vessel based on a first height corresponding to the deck height of the first vessel and a second height corresponding to the deck height of the second vessel, and the first vessel A port monitoring method comprising a; calculating a distance between the first vessel and the second vessel based on the position of the second vessel and the position of the second vessel may be provided.

In one embodiment, acquiring the position of the first vessel and the position of the second vessel includes generating a first transformed image by projecting the region of the first vessel to a reference plane at the first height; obtaining the position of the first vessel by using a first transformed image; projecting the second vessel region onto a reference plane at the second height to generate a second transformed image; and the second transformed image It may include the step of obtaining the position of the second vessel by using.

In an embodiment, the obtaining of the position of the first vessel includes a first position of at least one of a bow and a stern of the first vessel by using a pixel position of the region of the first vessel projected from the first transformed image. and acquiring the position of the second vessel, wherein the acquiring of the position of the second vessel includes at least one of a bow and a stern of the second vessel using a pixel position of the region of the second vessel projected from the second transformed image. acquiring the second location.

According to another aspect of the present invention, a computer-readable recording medium in which a program for performing the above-described methods is recorded may be provided.

Hereinafter, a vessel and port monitoring apparatus and method according to an embodiment will be described.

1 is a view of a port monitoring apparatus according to an embodiment.

Referring to FIG. 1 , the monitoring device 10 may include a sensor module 100 , a control module 200 , and a communication module 300 .

The sensor module 100 may sense information about a ship or a ship's vicinity and a port. The sensor module 100 may include an automatic identification system (AIS), an image generation unit, a LIDAR sensor, a position measurement unit, an attitude measurement unit, a casing, and the like.

The image generating unit may generate an image. The image generating unit may include a camera, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

The lidar sensor is a sensor for detecting a distance and a position of an object using a laser. For example, the distance between the lidar sensor and the object and the position of the object with respect to the lidar sensor may be represented by a three-dimensional coordinate system. For example, the distance between the lidar sensor and the object and the position of the object with respect to the lidar sensor may be expressed in a rectangular coordinate system, a spherical coordinate system, a cylindrical coordinate system, or the like. The lidar sensor may have a plurality of channels in a vertical or horizontal direction, and may be, for example, a lidar sensor having 32 or 64 channels.

The lidar sensor may use a laser reflected from the object to determine the distance R from the object. For example, the lidar sensor may use a time of flight (TOF), which is a time difference between an emitted laser and a detected laser, to determine the distance to the object. To this end, the lidar sensor may include a laser output unit for outputting a laser and a receiving unit for detecting the reflected laser. The lidar sensor determines the time the laser is output from the laser output unit, checks the time the receiver detects the laser reflected from the object, and determines the distance to the object based on the difference between the emitted time and the sensed time can do. Of course, the LiDAR sensor uses other methods such as triangulation based on the detected position of the detected laser to determine the distance (R) from the object, such as using a phase shift of the detected laser. R) may be determined.

The lidar sensor may determine the position of the object by using the angle of the irradiated laser. For example, when the irradiation angle of one laser irradiated from the lidar sensor toward the scan region of the lidar sensor is known, if a laser reflected from an object existing on the scan region is detected by the receiver, lidar The sensor may determine the position of the object based on the irradiation angle of the irradiated laser.

The lidar sensor may have a scan area including the object in order to detect the position of an arbitrary object in the vicinity. Here, the scan area represents a detectable area as one screen, and may mean a set of points, lines, and planes that form one screen during one frame. In addition, the scan area may mean the irradiation area of the laser irradiated from the lidar sensor, and the irradiation area may mean a set of points, lines, and surfaces where the laser irradiated during one frame meets a sphere at the same distance (R). can In addition, the field of view (FOV) means a detectable area, and may be defined as an angular range of the scan area when the lidar sensor is viewed as the origin.

The position measuring unit may measure the position of components included in the sensor module 100 , such as the sensor module 100 or the image generating unit. As an example, the location measurement unit may be a Global Positioning System (GPS). In particular, Real-Time Kinematic GPS (RTK-GPS) may be used to improve the accuracy of location measurement.

The position measuring unit may acquire position information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module 100 . For example, when the sensor module 100 is installed in a moving object such as a ship, the position measuring unit may acquire position information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a port, the position measurement unit may acquire position information at long time intervals. The time interval for obtaining the location information of the location measuring unit may be changed.

The posture measuring unit may measure the posture of components included in the sensor module 100 such as the sensor module 100 or the image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measuring unit may acquire posture information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module. For example, when the sensor module 100 is installed in a moving object such as a ship, the posture measuring unit may acquire posture information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a port, the posture measuring unit may acquire posture information at long time intervals. The time interval for acquiring the posture information of the posture measuring unit may be changed.

The casing may protect a sensor module such as an image generating unit, a position measuring unit, and a posture measuring unit.

At least one of an image generating unit, a position measuring unit, and a posture measuring unit may be present inside the casing. The casing may prevent equipment such as an image generating unit existing therein from being corroded by salt water. Alternatively, the casing may protect it by preventing or mitigating the impact applied to the equipment existing therein.

A cavity may be formed in the interior of the casing to contain an image generating unit or the like therein. For example, the casing may have a rectangular parallelepiped shape with an empty interior, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like can be disposed.

When the image generating unit is disposed inside the casing, an opening may be formed in one area of the casing or a transparent material such as glass may be formed in one area of the casing to secure a view of the image generating unit. The image generating unit may image the perimeter of the vessel and the harbor through the opening or transparent area.

The casing may be provided with a strong material to protect the image generating unit or the like from external impact. Alternatively, the casing may be provided with a material such as an alloy for seawater in order to prevent corrosion due to salt.

The casing may include equipment for removing foreign substances from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove foreign substances. As another example, foreign substances may be removed by applying water or washer liquid through a liquid spray included in the casing, or the foreign substances may be physically removed using a wiper after application.

The debris removal equipment can be operated manually, but can also be operated automatically. For example, the foreign material removal equipment may operate at a predetermined time interval. Alternatively, the foreign material removal device may be operated using a sensor that detects whether a foreign material has adhered to the image generating unit. Alternatively, after determining whether a foreign material is captured in the image by using the image captured by the image generating unit, the foreign material removal device may be operated when it is determined that the foreign material is present. Here, whether a foreign material is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as including two or more identical cameras.

The control module 200 may perform image analysis. In addition, the operation of receiving various data through the sensor module 100, the operation of outputting various outputs through the output device, the operation of storing various data in the memory or acquiring various data from the memory, etc. It can be done by control. Hereinafter, various operations or steps disclosed in the embodiments of the present specification may be interpreted as being performed by the control module 200 or being performed by the control of the control module 200 unless otherwise specified.

Examples of the control module 200 include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, an on-demand There may be a semiconductor (Application Specific Integrated Circuit, ASIC), a Radio-Frequency Integrated Circuit (RFIC), and a combination thereof.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or unidirectional communication. For example, the device 10 may transmit information to an external output device through the communication module 300 to output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.

The sensor module 100 , the control module 200 , and the communication module 300 may include a control unit. The control unit may process and calculate various types of information within the module, and may control other components constituting the module. The control unit may be provided in the form of an electronic circuit that physically processes an electrical signal. A module may include only a single control unit physically, or may include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be mounted on a physically separated server and a terminal and provided to processors that cooperate through communication. Examples of the control unit include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an application specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100 , the control module 200 , and the communication module 300 may include a communication unit. The modules may transmit/receive information through a communication unit. For example, the sensor module 100 may transmit information obtained from the outside through the communication unit, and the control module 200 may receive information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100 , the control module 200 , and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing processing of the programs, or data as a result of such processing. For example, the memory may store data required for learning and/or inference, an artificial neural network in progress or learned, and the like. Memory includes non-volatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or other tangible, non-volatile recording media. etc. can be implemented.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100 , and each sensor module 100 may include two cameras again.

2 and 3 are diagrams related to an embodiment of a monitoring device according to an embodiment.

Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100 and a control module 200 . The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 , and perform image analysis such as estimating position/movement information and image matching, which will be described later, through the control unit 220 . Also, the control module 200 may transmit an analysis result such as location/movement information and a matched image to the cloud server through the communication unit 210 . The cloud server may transmit the analysis result received from the control module 200 to a user terminal, such as a smartphone, tablet, or PC, or receive an instruction from the user terminal.

Referring to FIG. 3 , the monitoring device 10 may include a sensor module 100 . The sensor module 100 may generate an image through the camera 130 and transmit the image to the cloud server through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as position/movement information estimation and image registration, which will be described later. In addition, the cloud server may transmit the image analysis result to a user terminal such as a smartphone, tablet, or PC or receive an instruction from the user terminal.

The apparatus 10 illustrated in FIGS. 1 to 3 is merely an example, and the configuration of the apparatus 10 is not limited thereto.

As an example, the device 10 may include an output module 400 . The output module 400 may output a result of an operation performed by the control module 200 , and the like. For example, the output module 400 may output an analysis result. The output module 400 may be, for example, a display, a speaker, a signal output circuit, and the like, but is not limited thereto. In this case, information may be output through the output module 400 in addition to transmitting information to an external output device such as a user terminal and the external output device outputting information.

As another example, the device 10 may not include a sensor module. In this case, the control module 200 may receive information from an external sensor device and perform an image-based monitoring operation, such as performing image analysis. For example, the control module 200 may perform image analysis by receiving information from an AIS, a camera, a lidar, a radar, etc. already installed in a ship or a port.

In addition, the steps performed by each configuration of FIGS. 1 to 3 are not necessarily performed by the corresponding configuration and may be performed by other configurations. For example, in FIG. 2 above, although it has been described that the control unit 120 of the sensor module 100 performs viewpoint transformation, the control unit 220 of the control module 200 or the cloud server may perform viewpoint transformation. .

Hereinafter, the monitoring apparatus 10 and the method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100 . For example, an image may be acquired through an image generating unit included in the sensor module 100 . Alternatively, as described above, an image may be acquired from an external sensor device. Images for vessel and port monitoring will generally include sea, vessel, buoy, obstacle, terrain, port, sky, building, and the like. Hereinafter, the analysis and monitoring of an image acquired through a visible light camera will be mainly described, but the present invention is not limited thereto.

A field of view (FOV) and a depth of field may vary according to an image generating unit. 4 is a view of a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 4 , a field of view (FOV) may mean to what extent an image is included in an image horizontally or vertically, and is generally expressed as an angle (degree, degree). A larger viewing angle may mean generating an image including areas having a larger width left and right or generating an image including areas having a larger width up and down. The depth of field may mean a range of distances recognized as being in focus of the image, and the meaning of a deep depth of field may mean that the range of distances recognized as being in focus of the image is wide. Referring to FIG. 4 , according to the depth of field (DOF), the image may include an area A1 recognized as being in focus and an area A2 other than that. Hereinafter, an area included in the image is referred to as an imaging area (A1 + A2), and an area recognized as being in focus is referred to as an effective area (A1). Image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on , or based on a part of the imaging area, an area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and a shallow depth of field is a wide-angle camera. Examples of cameras with a small field of view and a deep depth of field include high magnification cameras and zoom cameras.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, a crane, a ship, etc. in a port, and there is no limitation in the number. However, the installation location or number of the sensor module 100 may vary according to characteristics such as the type and performance of the sensor module 100 . For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more above the water surface for efficient monitoring, or a plurality of sensor modules may be installed to have different imaging areas. In addition, the position and posture of the sensor module 100 may be manually or automatically adjusted during or after installation.

The viewing angle of the lidar may at least partially overlap with the viewing angle of the image generating unit (eg, a camera). For example, the field of view (FOV) of the lidar may be smaller than the field of view of the camera. As will be described later, in this case, lidar data outside the viewing angle of lidar for which lidar data is not acquired may be estimated using an image acquired from a camera.

5 and 6 are views of an installation position of a sensor module according to an embodiment. 5 and 6 , the sensor module 100 may be installed at a fixed location, such as a port or on land, or installed on a moving object, such as a ship. Here, when the sensor module 100 is installed in a vessel, it may be installed in a vessel to be monitored as shown in FIG. 6 (hereinafter referred to as a “target vessel”), and berthing or berthing of the target vessel as shown in FIG. 5 It may be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition, the sensor module 100 may be installed on a drone or the like to monitor a target vessel.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or at a location separate from it.

As described above, image analysis for image-based monitoring may include acquiring object characteristics. Examples of the object may include a ship, port, buoy, sea, terrain, sky, building, person, animal, fire, smoke, and the like. Examples of object characteristics may include the type of object, the position of the object, the distance to the object, absolute and relative speed and speed of the object, and the like.

Image analysis for image-based monitoring may include recognizing/judging a surrounding situation. For example, the image analysis may be to determine that a fire situation has occurred from an image of a fire in the port, or to determine that an intruder has entered the port from an image captured by a person entering the port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each of the modules 100 and 200 .

The device 10 may recognize an object. For example, the device 10 may recognize an object included in an image. For example, the device 10 may determine whether an object such as a ship, a tugboat, the sea, or a port is included in the image. Here, the object recognition may be determining at which position in the image the object exists.

7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.

7 is an image captured by a camera, and an object may be recognized as in FIG. 8 or 9 through the object recognition step.

Specifically, FIG. 8 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation. In this case, the object recognition step may mean a segmentation step. Through segmentation, a characteristic corresponding to a pixel on the image may be assigned or calculated from the image. It could be said that a characteristic has been assigned or labeled to a pixel. 7 and 8 , a segmentation image as shown in FIG. 8 may be obtained by performing segmentation based on an image captured by the camera of FIG. 7 . In FIG. 8 , the first pixel area P1 is the area on the image of the pixel corresponding to the ship, the second pixel area P2 is the sea, the third pixel area P3 is the quay wall of the harbor, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area P5 is an area on the image of a pixel corresponding to the sky.

Although FIG. 8 illustrates that information on the type of object corresponding to each pixel on an image is calculated by performing segmentation, information obtainable through segmentation is not limited thereto. For example, characteristics such as position, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, different characteristics may be expressed independently or may be expressed by reflecting them at the same time.

Table 1 is a table related to labeling in which information on the type and distance of an object is simultaneously reflected, according to an embodiment. Referring to Table 1, a class may be set in consideration of information on the type and distance of an object, and an identification value may be assigned to each class. For example, the second identification value may be assigned in consideration of both topography, which is information about the type of an object, and short distance, which is information about distance. Table 1 is an example in which type information and distance information are considered together. In addition, other information such as direction information, obstacle movement direction, speed, and route mark may also be considered. Also, not all identification values need to include a plurality of pieces of information, nor do they need to include the same type of information. For example, a specific identification value contains only information about the type (eg, identification value 1 does not include information about the distance) and another identification value contains information about the type and distance, etc. It can be expressed in various ways. For another example, other classes such as tugs, ropes, sides and decks of ships may be added or modified to other classes.

identification value class 0 sky and others One Sea 2 terrain + close range 3 Terrain + Medium 4 Terrain + Ranged 5 Fixed obstacle + close range 6 Fixed obstacle + medium range 7 Fixed Obstacle + Ranged 8 Dynamic obstacles + close range 9 Dynamic Obstacle + Medium Range 10 Dynamic Obstacle + Ranged

9 is a diagram showing a position of an object in an image with a bounding box, which is also referred to as detection. In this case, the object recognition step may mean a detection step. Compared with segmentation, detection can be viewed as detecting a position in which an object is included in a box shape, rather than calculating a characteristic for each pixel of an image. 7 and 9 , a detection image as shown in FIG. 9 may be obtained by performing detection based on an image captured by the camera of FIG. 7 . In FIG. 9 , it can be seen that a vessel is detected on the image and the position of the vessel is expressed as a rectangular bounding box BB. Although it is illustrated in FIG. 9 that only one object is detected, two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks, and the final result may be calculated by combining the results.

An artificial neural network is a kind of algorithm modeled after the structure of a human neural network, and may include one or more nodes or one or more layers including neurons, and each node may be connected through a synapse. Data input to the artificial neural network (input data) may be output (output data) through a node through a synapse, and information may be obtained through this.

Types of artificial neural networks include a convolutional neural network (CNN) that extracts features using a filter and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back as an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), relational network (RN), etc. can be applied and have limitations. it is not

Before using the artificial neural network, it is necessary to train it. Alternatively, training may be performed using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and the step of using the artificial neural network will be expressed as an inference step.

The artificial neural network may be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.

10 is an embodiment of the learning step of the artificial neural network, in which the untrained artificial neural network receives learning data or training data and outputs the output data, and compares the output data with the labeling data. The artificial neural network can be trained through the backpropagation of the error. The training data, output data, and labeling data may be images. The labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or a program.

11 is an example of an inference step of an artificial neural network, and a learned artificial neural network may receive input data and output output data. According to the information of the learning data in the learning stage, information that can be inferred in the inference stage may vary. Also, the accuracy of output data may vary according to the learning degree of the artificial neural network.

The recognition of the object is not limited to the above description and may be implemented in other ways. For example, although it has been described that the identification value is used for object recognition for convenience of description, the identification value is only used as one type of index. For example, a vector type index may be used for object recognition, and the training data, output data, and labeling data of the artificial neural network may be vector type data.

According to one embodiment, device 10 may provide berthing guide information for a vessel. The berthing guide information may be used in the berthing of a ship and may refer to information for assisting or guiding the berthing of a user such as a pilot or a captain. The eyepiece guide information may include information about distance/velocity. Examples of the distance/velocity information include, but are not limited to, an absolute position such as coordinates, a relative position from a specific reference, a distance from an arbitrary point, a distance range, a direction, an absolute velocity, a relative velocity, a velocity, and the like.

The distance/velocity information may be estimated based on a predetermined area or point. For example, the distance between the ship and the quay wall may be estimated by calculating the distance between one point of the ship and one point of the quay wall, or may be estimated by calculating the shortest distance between one point of the ship and the quay wall. As another example, the distance between the vessels may be estimated by calculating a distance between a point of the first vessel and a point of the second vessel. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

The distance/speed information may be expressed as a predetermined distance value, a direction value, a speed value, and the like. For example, distance information may be expressed as 1 m, 2 m, etc., direction information may be expressed as 10°, 20°, etc., and speed information may be expressed as 5 cm/s, 10 cm/s, or the like.

The distance/velocity information may be expressed as an index corresponding to a plurality of categories having a predetermined range. For example, distance information may be expressed in short-range, medium-distance, and long-distance, direction information may be expressed in a left direction, a front direction, and a right direction, and speed information may be expressed in low speed, medium speed, high speed, etc. there is. It may be possible to combine these to express the left near field, the right far field, and the like.

12 is a view for explaining eyepiece guide information according to an embodiment. Referring to Figure 12 (a), the berthing guide information is the berthing guide information between the vessel (OBJ1) and the quay wall (OBJ2) (f1, f2) and the vessel (OBJ1) and the other vessel (OBJ3, OBJ4) between the berthing guide It may include information f3 and f4. Specifically, the berthing guide information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 may include information about the distance/speed with the quay wall OBJ2 of the vessel OBJ1, and the vessel OBJ1 The berthing guide information (f3, f4) between the and other vessels (OBJ3, OBJ4) may include information about the distance/speed of the vessel (OBJ1) and the other vessels (OBJ3, OBJ4). Referring to FIG. 12 (b), the berthing guide information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 is an area corresponding to an area in contact with the sea level of the vessel OBJ1, a boundary region and a quay wall OBJ2 ) may be information between The boundary area may include not only a line or point where the ship and the sea level come into contact with each other, but also a predetermined area near the line or point. In addition, the berthing guide information between the vessel (OBJ1) and the other vessels (OBJ3, OBJ4) may be information between a predetermined area corresponding to the bow/stern of the vessel (OBJ1, OBJ3, OBJ4), where the predetermined area may mean not only the bow/stern point of the ships OBJ1, OBJ3, OBJ4, but also a certain area in the vicinity thereof.

The berthing guide information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 may include, but is not limited to, two eyepiece guide information as shown in FIG. 12, but one eyepiece guide information or three or more It may include berthing guide information. In addition, the berthing guide information f3, f4 between the vessel OBJ1 and the other vessels OBJ3, OBJ4 may include one berthing guide information between the two vessels as shown in FIG. 12, but is limited thereto No, it may include two or more berthing guide information between two ships. The above-described information may be referred to as eye guide information when used in the eye of the vessel.

13 is a view for explaining information on the berthing guide between the ship and the quay wall according to an embodiment.

The eyepiece guide information may be information on sea level/sea level. For example, the berthing guide information f5 may include information about the distance/speed between the vessel OBJ1 and the quay wall OBJ2 at sea level.

The eyepiece guide information may be information at the ground level (or the height of the quay wall). For example, the eyepiece guide information f6 may include information about the distance/velocity between the vessel OBJ1 and the quay wall OBJ2 at the height of the quay wall OBJ2 .

The eyepiece guide information may be information at a predetermined height. Here, the predetermined height may be between the height of the sea level and the height of the quay wall, as in the eyepiece guide information f7 shown in FIG. 13, but is not limited thereto.

The berthing guide information between the vessel and the quay wall may be information f8 between the fenders (fender, OBJ6) installed on the vessel (OBJ1) and the quay wall (OBJ2). Since the ship (OBJ1) collides with the insect repellent (OBJ6) when berthing or comes into contact with the insect repellent (OBJ6) when anchored, it is important to obtain information (f8) on the distance/speed between the ship (OBJ1) and the insect repellent (OBJ6). can be advantageous In FIG. 13, although the insect repellent (OBJ6) is shown to be installed at a position higher than sea level, at least a portion of the insect repellent (OBJ6) may be installed to be submerged in the sea (OBJ5), in this case the berthing guide information is at sea level. It may include information about the distance/speed between the vessel and the insect repellent.

The berthing guide information between the vessel and the other vessel may include information on the distance/speed between the vessel and the other vessel at sea level. The berthing guide information between the vessel and the other vessel may include information on the distance/speed between the vessel and the other vessel at a predetermined height. Here, the predetermined height may be determined in consideration of the shape of the hull. For example, the predetermined height may be the height of a region in which the vessel protrudes in the direction of another vessel. In this case, it may be advantageous to the berthing guide by more accurately grasping the possibility of collision between ships.

According to an embodiment, the device 10 may calculate the eyepiece guide information based on the lidar data. For example, the device 10 may calculate the distance/velocity to the object by using the three-dimensional coordinates of the lidar points included in the obtained lidar data. Examples of objects may include ships, seas and land, ports, quays, buoys, terrain, sky, buildings, people, animals, and the like.

According to an embodiment, the eyepiece guide information may be calculated based on the image. Here, the image may be an image generated by an image generating unit such as a camera or an image processed through image segmentation therefrom. For example, information on the distance/velocity of a ship is calculated based on an image including a ship, sea, and land as objects, or information on distance/speed of a ship based on a segmentation image generated from the image through segmentation can be calculated. Examples of the object may include a port, a quay wall, a buoy, a terrain, a sky, a building, a person, an animal, etc. in addition to a ship, sea, and land.

Hereinafter, an object for estimating eyepiece guide information is referred to as a target object. For example, in the above example, a ship may be a target object. Also, there may be a plurality of target objects. For example, when estimating a distance or speed of each of a plurality of ships included in an image, the plurality of ships may be a target object.

According to an embodiment, the eyepiece guide information may be calculated based on image pixels. As described above, when the eyepiece guide information is calculated based on the point, the point on the image may correspond to a pixel. Accordingly, the eyepiece guide information may be calculated based on the spacing between image pixels.

Information on the distance between points may be calculated based on the distance between pixels. For example, a predetermined distance may be allocated to each pixel interval, and the distance between points may be calculated in proportion to the interval between pixels. As another example, the distance between pixels may be calculated based on the coordinate values of the pixels on the image, and the distance between points may be calculated based on this.

The information on the speed between points may be calculated based on a change in the information on the distance between the points. In this case, movement information may be calculated based on a plurality of images or image frames. For example, information on the speed between points may be calculated based on the distance between points in the previous frame, the distance between points in the current frame, and the time interval between frames.

14 is a view for explaining a method of obtaining eyepiece guide information according to an embodiment. The step of acquiring the eyepiece guide information is a step of acquiring an image generated by an image generating unit such as a camera (FIG. 14 (a)), and performing image segmentation on the image to perform image segmentation (or an image obtained by visualizing the segmentation image) generating ((b) of FIG. 14), finding a point for calculating eyepiece guide information based on the segmentation image ((c) of FIG. 14) and calculating eyepiece guide information corresponding to the point may include In FIG. 14 , a method for calculating eyepiece guide information from a segmentation image has been described, but this is only an embodiment, and without a step of generating a segmentation image, a point is found based on the image generated by the image generating unit and the eyepiece guide information may be calculated. There will be.

Acquiring the eyepiece guide information may include converting a viewpoint of the image. For example, the step of acquiring the eyepiece guide information may include, after the step of acquiring the image generated by the image generating unit, performing viewpoint transformation on the image, and performing segmentation on the viewpoint transformed image to obtain a segmented image. Or, after the step of generating the segmented image, by performing a viewpoint transformation on the segmented image, it is possible to calculate eyepiece guide information for the viewpoint-converted segmented image. Hereinafter, viewpoint transformation will be described.

In general, an image generated by an image generating unit such as a camera may be displayed as a perspective view. Converting this into a top view (planar view), a side view, another perspective view, etc. may be referred to as view transformation. Of course, a top view or a side view image may be converted into another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, it may not be necessary to perform view point conversion.

15 is a diagram of viewpoint transformation according to an embodiment. Referring to (a) of FIG. 15 , another perspective view image may be acquired through viewpoint transformation of the perspective view image. Here, viewpoint conversion may be performed so that the quay wall OBJ2 is positioned along a horizontal direction (left and right direction on the image) on the image. Referring to (b) of FIG. 15 , a top view image may be obtained through viewpoint transformation of a perspective viewpoint image. Here, the top view image may be a view looking down at the sea level in a direction perpendicular to the sea level. Also, as in (a) of FIG. 15 , viewpoint conversion may be performed so that the quay wall OBJ2 is positioned along the horizontal direction on the image.

It is possible to improve the ease, convenience, and accuracy when calculating the eyepiece guide information through viewpoint conversion. For example, in the case of pixel-based distance calculation, if a top-view image is used, a distance corresponding to an interval between pixels may be the same for the entire image or at least a partial region.

As an example of the viewpoint transformation, inverse projection transformation (IPM) may be performed. A two-dimensional image is generated when light reflected from a subject in a three-dimensional space is incident on an image sensor through the lens of the camera, and the relationship between two dimensions and three dimensions depends on the image sensor and lens, for example, Equation 1 and can be expressed together.

Here, the matrix on the left side means two-dimensional image coordinates, the first matrix on the right side means internal parameters, the second matrix means external parameters, and the third matrix means three-dimensional coordinates. Specifically, fx and fy denote focal lengths, cx and cy denote principal points, and r and t denote rotation and translation transformation parameters, respectively.

By projecting a two-dimensional image on an arbitrary plane in three dimensions through inverse projection transformation, the viewpoint can be changed. For example, a perspective view image may be converted into a top view image through inverse projection transformation, or may be converted into another perspective view image.

Internal parameters may be required for viewpoint transformation. As an example of a method for obtaining an internal parameter, the Zhang method may be used. The Zhang method is a type of polynomial model, and is a method of acquiring internal parameters by photographing a grid with a known size at various angles and distances.

Information on the position and/or posture of the image generating unit/sensor module capturing the image may be required for viewpoint transformation. Such information may be obtained from the position measuring unit and the posture measuring unit.

Alternatively, information on the position and/or posture may be acquired based on the position of the fixture included in the image. For example, at a first time point, the image generating unit may generate a first image including a target fixture that is disposed at a first position and/or a first posture and is a fixed object such as a terrain or a building. Thereafter, at a second time point, the image generating unit may generate a second image including the target fixture. Comparing the position of the target fixture on the first image and the position of the target fixture on the second image to calculate a second position and/or a second posture that is the position and/or posture of the image generating unit at the second time point can do.

Acquisition of information on a position and/or posture for viewpoint transformation may be performed at predetermined time intervals. Here, the time interval may depend on an installation position of the image generating unit/sensor module. For example, when the image generating unit/sensor module is installed in a moving object such as a ship, there may be a need to obtain information on the position and/or posture at short time intervals. On the other hand, when the image generating unit/sensor module is installed in a fixed body such as a port, information on the position and/or posture may be acquired at relatively long time intervals or may be initially acquired only once. When moving and fixing are repeated like a crane, it may be implemented in a manner that acquires information about a position and/or posture only after moving. In addition, the time interval for obtaining information on such a position and/or posture may be changed.

The device 10 may transform the image based on the reference plane. For example, the device 10 may view-convert an image from a plane in which a quay wall is located and parallel to the sea level to a reference plane. Here, the reference plane may depend on the calculated sea level height. Of course, the device 10 is not limited to the above description, and the view of the image may be converted into a reference plane on another plane such as sea level, a part of the ship other than the plane where the quay wall is located (for example, the reference plane of the deck height).

The device 10 may convert the image viewpoint in consideration of the sea level. For example, the device 10 may calculate the sea level height by using data obtained from a lidar sensor or a camera, and convert the viewpoint of the image in consideration of the calculated sea level height.

The above-described viewpoint transformation method is merely an example, and viewpoint transformation may be performed in a different way. The viewpoint transformation information is for viewpoint transformation such as information on the matrix, parameters, coordinates, position and/or posture of Equation 1 above. Include the necessary information.

Device 10 may calculate distances to other vessels and/or quay walls for vessel and/or port monitoring. For example, the device 10 may generate a feature point of a vessel for calculating a distance from sensor data, and calculate a distance to another vessel and/or a quay wall based on the generated feature point.

A feature point of a ship is a concept including a specific point in an image, a specific lidar point of lidar data, and the like. It may include a point corresponding to , a point corresponding to a portion where the ship and the sea level are in contact.

16 is a flowchart of a method for calculating a distance based on a feature point of a ship according to an exemplary embodiment.

Referring to FIG. 16 , the method for calculating a distance based on a feature point of a ship according to an exemplary embodiment includes acquiring sensor data ( S1000 ), generating a feature point of the ship ( S2000 ), and calculating a distance based on the feature point It may include a step (S2000) of doing.

The device 10 may acquire sensor data (S1000).

According to an embodiment, the device 10 may obtain an image from a camera. For example, the device 10 may acquire an image from a camera installed on a berth toward the sea. For example, the device 10 may acquire an image of the sea, and if there is a ship, it may also acquire an image of the ship.

According to an embodiment, the device 10 may obtain lidar data from a lidar sensor. For example, the device 10 may acquire lidar data from a lidar sensor installed on a berth toward the sea. For example, the device 10 may acquire lidar data for the sea, and when the vessel enters the berth, it may also acquire lidar data for the vessel.

According to an embodiment, the lidar sensor may acquire lidar data for an area corresponding to an area captured by a camera acquiring an image. For example, the lidar sensor may have a field of view that at least partially overlaps with a field of view of a camera that acquires the image.

17 is an example of acquisition of sensor data according to an embodiment. Referring to FIG. 17 , the device 10 according to an embodiment may acquire sensor data for the same area from the lidar and the camera installed around the berth. For example, the lidar and the camera may be installed on the pier toward or looking at the berth to acquire data on an area where the berth is located. Here, the viewing angle of the lidar may at least partially overlap with the viewing angle of the camera.

Referring to FIG. 17( a ), the device 10 may acquire an image of the sea, and if there is a ship, it may also acquire an image of the ship. The device 10 may use the acquired camera images to calculate a distance between a vessel and another vessel and/or a distance between a vessel and a quay wall.

Referring to FIG. 17(b) , the device 10 may acquire lidar data for the sea, and when the vessel enters the berth, it may also acquire lidar data for the vessel. The lidar data may include a plurality of lidar points captured by lidar sensors. For example, the lidar data may include a plurality of lidar points for each vertical or horizontal channel. The device 10 may use the obtained lidar data to calculate a distance between a vessel and another vessel and/or a distance between a vessel and a quay wall.

It will be described again by returning to FIG. 16 .

The device 10 may generate a feature point of the vessel (S2000).

The device 10 may use the acquired sensor data to generate a feature point of a ship used to calculate a distance between the ship and another object. For example, the device 10 may generate a feature point of the vessel by using at least one of a camera installed on a berth toward the sea and sensor data obtained from a lidar sensor. As an example, the device 10 may detect an area of the vessel from the camera image and generate a feature point of the vessel based on the detected area of the vessel. For example, the device 10 may detect LiDAR points related to a LIDAR beam reflected from the vessel from LIDAR data, and generate a characteristic point of the vessel based on the detected LIDAR points. The device 10 is not limited to the above description, and may generate the feature points of the vessel in other ways, such as fusing a camera image and lidar data. Various embodiments of the generation of the feature point of the vessel will be described later.

The device 10 may calculate a distance between the ship and another object based on the feature point of the ship ( S3000 ).

The device 10 may calculate a distance between a vessel and another vessel and/or a distance between a vessel and a quay wall based on the vessel's characteristic points. For example, the device 10 may calculate a distance between a vessel and another vessel based on a position of a point corresponding to at least one of a bow and a stern of the vessel. For another example, the device 10 may calculate the distance between the ship and the quay wall based on the location of the point corresponding to the portion where the ship and the sea level come into contact. For example, the distance calculation using the camera image may be calculated based on the distance between image pixels, and the distance calculation using the lidar data may be calculated based on the coordinate values of the lidar data. It can be applied, so the details will be omitted.

According to an embodiment, the device 10 may calculate the distance between the ship and another object by using the feature points of the ship obtained from the camera image. For example, the device 10 may calculate a distance between the ship and another ship or a quay wall by using the feature points of the ship obtained from the camera image.

According to an embodiment, the device 10 may generate a feature point of a vessel using an image in which a viewpoint is converted. For example, the device 10 may generate feature points of the vessel using a transformed image in which the vessel zero in the image is projected onto a reference plane.

18 is a flowchart of a method for generating feature points of a ship using an image in which a viewpoint is converted according to an exemplary embodiment. Referring to FIG. 18 , the method for generating feature points of a ship using an image in which a viewpoint is converted according to an embodiment includes detecting a ship area from a camera image (S2010), and generating a converted image by projecting the ship area on a reference plane It may include the step (S2020) and the step (S2030) of obtaining the feature point of the vessel from the converted image.

The device 10 may detect the vessel area from the camera image (S2010).

For example, the device 10 may detect a region corresponding to the vessel in the image using an artificial neural network. For example, the device 10 may generate a segmented image from an image using an artificial neural network, and detect an area in which a pixel labeled with object information indicating a ship of an object type is located as an area corresponding to the ship. The device 10 is not limited to the above description, and may detect a region corresponding to the vessel in the image in another method, such as determining an region corresponding to the vessel through image detection.

The apparatus 10 may generate a converted image by projecting the vessel area on the reference plane (S2020).

For example, the device 10 may generate a transformed image in which the area of the vessel is projected as a reference plane at the deck level of the vessel. As an example, the device 10 generates a segmentation image from the image using an artificial neural network, and based on the area in which the pixel is located, the object type is the deck of the ship or the object information indicating the side of the ship. It is possible to acquire the position of the part where the deck is in contact, and project the area of the ship to the obtained position of the deck as a reference plane. As another example, the device 10 may acquire the position of the deck of the vessel in the image using lidar data matched to the camera image, and project the area of the vessel to the reference plane based on the acquired position of the deck. .

As another example, the device 10 may generate a converted image in which the sea level is a reference plane and the area of the ship is projected. As an example, the device 10 generates a segmented image from the image using an artificial neural network, and based on the regions in which pixels labeled with object information representing a ship or sea of an object type are located, the position of the portion where the ship and the sea level come into contact can be obtained and the area of the ship can be projected on the reference plane based on the obtained sea level position.

The apparatus 10 is not limited to the above description, and may generate the converted image in other ways, such as acquiring the position of the deck through image detection or projecting the ship area using the quay wall as a reference plane.

The device 10 may acquire the feature point of the vessel from the converted image (S2030).

For example, the device 10 may determine a feature point of a vessel for calculating a distance between the vessels in the transformed image. For example, the device 10 obtains a bow and/or a point corresponding to the bow of the ship from the converted image in which the area of the ship is projected as a reference plane at the deck height of the ship, and uses this to determine the distance between the ships can be calculated

For another example, the device 10 may determine a feature point of the vessel for calculating the distance between the vessel and the quay wall in the transformed image. As an example, the device 10 may obtain a bow and/or a point corresponding to the bow of the ship from the converted image in which the area of the ship is projected with the sea level as a reference plane, and calculate the distance between the ship and the quay wall using this. .

The apparatus 10 is not limited to the above description, and may generate the converted image in other ways, such as acquiring the position of the deck through image detection or projecting the ship area using the quay wall as a reference plane.

19 is an example of a converted image according to an embodiment. In the converted image, a distortion phenomenon may occur due to the projection of the vessel area onto the reference plane, the distortion may become more severe as the distance from the reference plane increases, and distortion may not occur in the reference plane.

Referring to FIG. 19 ( a ), the apparatus 10 according to an embodiment generates a converted image in which the area of the ship is projected to the reference plane 401 at sea level, and uses this to calculate the ship's feature points for distance calculation. can be decided

For example, the apparatus 10 may determine an arbitrary point of a line where the ship and the sea level in the converted image meet as a feature point of the ship. For example, the device 10 may calculate the distance between the bow of the vessel and the quay wall or another vessel by using an arbitrary point 403 on the bow side of the line where the vessel and the sea level meet in the converted image. As another example, the device 10 may calculate the distance between the stern of the vessel and the quay wall or another vessel by using an arbitrary point 402 on the stern side of the line where the vessel and the sea level meet in the converted image.

As another example, the device 10 may determine points located at the ends of the bow and stern of the ship in the converted image as the feature points of the ship. As an example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the point 405 of the bow end of the vessel in the transformed image. As another example, the device 10 may use the point 406 of the stern end of the vessel in the transformed image to calculate the distance between the vessel and the quay wall or other vessel.

The apparatus 10 is not limited to the above description, and the distance between the vessel and another vessel or quay wall is calculated by using the vessel feature points extracted by polygonalizing the region of the vessel in the converted image, such as calculating the distance in other ways. It is free to calculate

Referring to FIG. 19( b ), the device 10 according to an embodiment generates a converted image in which the area of the ship is projected to the reference plane 411 at the deck height of the ship, and uses this to generate a ship for distance calculation It is possible to determine the characteristic points of

For example, the device 10 may determine an arbitrary point of a line in which the side surface of the ship and the deck area of the ship in the converted image meet as the feature point of the ship. As an example, the device 10 may calculate the distance between the bow of the ship and the quay wall or another ship using any point 413 on the bow side of the line where the side of the ship and the deck area of the ship in the converted image meet. there is. As another example, the device 10 calculates the distance between the stern of the vessel and the quay wall or another vessel using an arbitrary point 414 on the stern side of the line where the side of the vessel and the deck area of the vessel in the converted image meet. can

As another example, the device 10 may determine points located at the ends of the bow and stern of the ship in the converted image as the feature points of the ship. As an example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the point 415 of the bow end of the vessel in the transformed image. As another example, the device 10 may use the point 416 of the stern end of the vessel in the transformed image to calculate the distance between the vessel and the quay wall or other vessel.

The apparatus 10 is not limited to the above description, and the distance between the vessel and another vessel or quay wall is calculated by using the vessel feature points extracted by polygonalizing the region of the vessel in the converted image, such as calculating the distance in other ways. It is free to calculate

20 is a diagram for explaining calculation of a distance between ships according to an exemplary embodiment. Referring to FIG. 20 , the apparatus 10 may calculate the distance between the ships by projecting the area of each ship in the image onto different reference planes.

The device 10 may detect the first vessel region 421 and the second vessel region 422 from the camera image. For example, the device 10 may detect the first vessel region 421 and the second vessel region 422 using an artificial neural network. For this, since the above description may be applied, a detailed description thereof will be omitted.

The device 10 may project the detected first vessel region 421 and the second vessel region 422 onto each reference plane at the deck height of each vessel to generate a transformed image. For example, the device 10 may project the detected first vessel area 421 onto a first reference plane 425 at the deck level of the first vessel to generate a first transformed image 423 , The device 10 may project the detected second vessel region 422 onto a second reference plane 426 at the deck height of the second vessel to generate a second transformed image 424 . For this, since the above description may be applied, a detailed description thereof will be omitted.

The device 10 may obtain a feature point of each vessel by using each of the transformed images.

The device 10 may determine a feature point using a point at a position corresponding to the bow and/or stern of the first ship in order to calculate the distance between the first ship and the second ship in the first converted image 423 . For example, the device 10 may acquire one of the points at both ends of the first vessel region in the first transformed image 423 as a feature point. Here, a point 427 located near the second ship among the points at both ends of the first ship area may be determined as a feature point of the ship.

The device 10 may determine a feature point by using a point corresponding to the bow and/or stern of the second ship in order to calculate the distance between the second ship and the first ship in the second converted image 424 . For example, the device 10 may acquire one of the points at both ends of the second vessel region in the second transformed image 424 as a feature point. Here, among the points at both ends of the second vessel area, a point 428 located closer to the first vessel may be determined as a characteristic point of the vessel. For this, since the above description may be applied, a detailed description thereof will be omitted.

The apparatus 10 may calculate a distance between the first vessel and the second vessel using feature points of the first vessel and the second vessel obtained from each of the transformed images 423 and 424 . For example, the device 10 may calculate a distance between the first vessel and the second vessel based on the number of pixels between the feature points 427 , 428 of the first vessel and the second vessel. As another example, the device 10 may calculate a distance between the first vessel and the second vessel based on lidar data matched to the feature points 427 and 428 of the first vessel and the second vessel. For this, since the above description may be applied, a detailed description thereof will be omitted.

According to an embodiment, the device 10 may calculate a distance between the ship and another object by using the feature points of the ship obtained from the lidar data. For example, the device 10 may calculate the distance between the ship and another ship or a quay wall using the feature points of the ship obtained from the lidar data. For example, the device 10 may calculate the distance between the vessel and another vessel by using a lidar point related to a lidar beam reflected from both ends of the vessel. As another example, the device 10 may calculate the distance between the ship and the quay wall using a lidar point related to a lidar beam reflected from a portion where the vessel and the sea level meet.

However, if the lidar sensor has low performance or there are restrictions on the place where the lidar sensor is installed, insufficient lidar data may be obtained for monitoring a ship or a port. Therefore, it may be necessary to supplement insufficient lidar data for monitoring of ships or ports.

To this end, according to an embodiment, the device 10 may estimate new lidar data from the acquired lidar data. For example, the device 10 may newly estimate a lidar point associated with a ship by using the acquired lidar data. As an example, the device 10 may estimate the lidar point associated with the vessel by interpolating or extrapolating lidar points associated with the lidar beam reflected from the vessel.

21 is a flowchart of a method for estimating a lidar point according to an embodiment. Referring to FIG. 21 , the method for estimating a lidar point according to an embodiment includes matching a camera image and lidar data (S1010) and estimating a lidar point using the matched camera image and lidar data (S1020). ) may be included.

The device 10 may match the camera image and lidar data (S1010).

According to an embodiment, the device 10 may match the camera image and the lidar data using the information for matching. For example, the device 10 may match the camera image and the lidar data by matching the coordinate system on the camera image and the coordinate system on the lidar data. That is, the coordinate system of the camera image and the lidar data may be converted to each other.

The device 10 may match the camera image and lidar data in consideration of the installation position of the camera, the installation angle of the camera, the installation position of the lidar sensor, the installation angle of the lidar sensor, and the like. Here, the device 10 may realign the camera image and lidar data by reflecting the calculated sea level height. As an example, when the lidar data and the camera image are matched, the device 10 may use the calculated sea level height as a data conversion variable to realign the camera image and the lidar data.

22 is an example of matching a camera image and lidar data according to an embodiment. Referring to FIG. 22( a ), it can be seen that the lidar data acquired from the lidar sensor scanning the same area as the image acquired from the camera installed on the berth are matched.

According to one embodiment, the device 10 may match the segmented image with the lidar data. For example, the device 10 may generate a segmentation image from an image obtained from a camera using an artificial neural network, and may match the generated segmentation image with lidar data. Referring to FIG. 22(b), it can be seen that the lidar data acquired from the lidar sensor scanning the same area as the image acquired from the camera installed on the berth are matched.

The matching of the camera image and the lidar data is not limited to the above description, and may be implemented in other ways, such as matching the image detected using an artificial neural network with the lidar data.

It will be described again by returning to FIG. 21 .

The device 10 may estimate a lidar point using the matched camera image and lidar data (S1020).

According to an embodiment, the device 10 may generate an estimated lidar point by using an image matched with the lidar data among a plurality of lidar points of the lidar data. For example, the device 10 may newly generate an estimated LiDAR point related to the sea by using a lidar point matched to an area corresponding to the sea in the image. As another example, the device 10 may newly generate an estimated lidar point associated with the vessel by using the lidar point matched to the region of the vessel in the image.

23 is a flowchart of a method for generating feature points of a ship in consideration of an estimated lidar point according to an embodiment. Referring to FIG. 23 , in the method for generating feature points of a ship in consideration of the estimated LiDAR point according to an embodiment, detecting a ship area from a camera image ( S2110 ), and a lidar beam reflected from the ship based on the detected ship area Selecting LiDAR points related to (S2120), estimating a LiDAR point related to a vessel using the selected LiDAR points (S2130), and generating a feature point of the vessel in consideration of the estimated LiDAR point (S2130) S2140) may be included.

The device 10 may detect the vessel area from the camera image ( S2110 ).

The camera image and lidar data may correspond to each other and may be matched. For this, the above-described contents may be applied, and a detailed description thereof will be omitted.

The device 10 may select lidar points related to the lidar beam reflected from the vessel based on the detected vessel area ( S2120 ).

The device 10 may use an area corresponding to the vessel in the detected image to select lidar points associated with the lidar beam reflected from the vessel. For example, the device 10 may select lidar points associated with a lidar beam reflected from the vessel in consideration of pixel positions of pixels included in an area corresponding to the vessel in the image.

24 is an example of lidar points associated with a lidar beam reflected from a vessel according to an embodiment. Referring to FIG. 24A , the device 10 may select lidar points 431 related to the lidar beam reflected from the vessel from among lidar points matched with the image. For example, the apparatus 10 may select lidar points 431 associated with a lidar beam reflected from the vessel that are matched to pixels included in an area corresponding to the vessel in the image among the plurality of lidar points. ) can be selected.

Referring to FIG. 24( b ), the apparatus 10 may select lidar points 432 related to the lidar beam reflected from the vessel from among lidar points matched with the segmentation image.

Also, as will be described later, it is also possible for the apparatus 10 to select LiDAR points related to the LIDAR beam reflected from the vessel among LIDAR points that are matched with the transformed image in which the vessel area is projected on the reference plane.

24 , selection of lidar points related to a lidar beam reflected from a ship, such as when a detected image is used, is not limited to the above description and may be implemented in another manner. According to one embodiment, the device 10 may select lidar points associated with a lidar beam reflected from a vessel using only lidar data. For example, the device 10 may select lidar points related to a lidar beam reflected from a ship in consideration of the distribution and number of lidar data.

It will be described again by returning to FIG. 23 .

The device 10 may estimate a LiDAR point related to a ship by using the selected LiDAR points (S2130).

For example, device 10 may estimate a new lidar point by interpolating or extrapolating lidar points associated with a lidar beam reflected from a vessel. As an example, the apparatus 10 may estimate a new lidar point by interpolating or extrapolating each lidar point in consideration of a relative position using coordinates of lidar points related to a lidar beam reflected from a ship.

According to an embodiment, the apparatus 10 may estimate a new lidar point by using a transformed image in which a vessel area is projected as a reference plane. For example, the device 10 may estimate a new LiDAR point using a converted image in which a ship area is projected with the sea level as a reference plane. As another example, the apparatus 10 may estimate a new lidar point using a converted image in which the ship area is projected with the area at the deck height of the ship as a reference plane.

25 and 26 are diagrams for explaining generation of an estimated lidar point using a lidar point matched to a ship area projected on a reference plane according to an embodiment.

Referring to FIG. 25 , the converted image in which the ship area is projected with the sea level 441 as the reference plane and the lidar point may be matched with each other. Since the lidar points 442 located in the area where the ship and the sea level contact each other have the same height value, the estimated lidar points 443 are interpolated and/or extrapolated by interpolating and/or extrapolating the lidar points 442 located in the same area. , 444, 445). For example, the apparatus 10 may generate the estimated lidar point 443 by interpolating lidar points 442 located in the region 441 where the ship and the sea level meet. As another example, the apparatus 10 may generate the estimated lidar points 444 and 445 by extrapolating the lidar points 442 located in the region 441 where the ship and the sea level meet. Here, the device 10 may estimate the lidar points in consideration of the shape of the vessel in the image.

Referring to FIG. 26 , the converted image in which the ship area is projected with the area 451 at the deck height of the ship as a reference plane and the lidar point may be matched with each other. Since the lidar points 452 located in the deck height region of the ship have the same height value, the estimated lidar points 453, 453, by interpolating and/or extrapolating the lidar points 452 located in the same region 454, 455) can be generated. For example, the device 10 may interpolate the lidar points located in the deck-height region 451 of the vessel to generate an estimated lidar point 453 . As another example, device 10 may extrapolate lidar points located in deck-height region 451 of the vessel to generate estimated lidar points 454 , 455 . Here, the device 10 may estimate the lidar points in consideration of the shape of the vessel in the image.

Of course, the apparatus 10 is not limited to the above description, and the estimated lidar points may be generated by other methods, such as estimating lidar points located in an area that does not have the same height value.

It will be described again by returning to FIG. 23 .

The device 10 may generate a feature point of the vessel in consideration of the estimated lidar point ( S2140 ).

According to an embodiment, the device 10 may select a feature point of the vessel for calculating the distance between the vessels from among the generated estimated lidar points and the existing lidar points related to the lidar beam reflected from the vessel. there is. That is, the device 10 is an estimated lidar generated by extrapolating and/or interpolating the existing lidar points and the existing lidar points matched with the transformed image in which the region of the vessel is projected to the reference plane at the deck height of the vessel. Among the points, the bow and/or lidar points corresponding to the bow of the ship may be acquired, and the distance between the ships may be calculated using the obtained.

According to one embodiment, the device 10 determines the characteristic point of the vessel for calculating the distance between the vessel and the quay wall among the existing lidar points associated with the lidar beam reflected from the vessel and the generated estimated lidar points. can That is, the device 10 sets the existing lidar points matched with the transformed image in which the region of the vessel is projected with the sea level as the reference plane, and the vessel among the estimated lidar points generated by extrapolating and/or interpolating the existing lidar points. It is possible to obtain points corresponding to the bow and/or bow of , and use them to calculate the distance between the vessel and the quay wall.

For example, the device 10 may determine any LiDAR point matched to a line where the ship and the sea level in the converted image meet as the feature point of the ship. For example, the device 10 may calculate the distance between the bow of the vessel and the quay wall or another vessel using any lidar points matched to the bow side of the line where the vessel and the sea level in the converted image meet. As another example, the device 10 may calculate the distance between the stern of the vessel and the quay wall or another vessel using any lidar point matched to the stern side of the line where the vessel and the sea level meet in the converted image. Here, the lidar point may include an existing lidar point and an estimated lidar point.

For another example, the device 10 may determine the lidar points matched to the ends of the bow and stern of the ship in the converted image as the feature points of the ship. As an example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the lidar point matched to the bow end of the vessel in the converted image. As another example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the lidar point matched to the stern end of the vessel in the converted image. Here, the lidar point may include an existing lidar point and an estimated lidar point.

The device 10 is not limited to the above description, and may use other methods, such as polygonalizing an area of the vessel in the converted image, and calculating the distance between the vessel and another vessel or quay wall using the lidar point matched thereto. It is free to calculate the distance.

The device 10 may obtain the height (deck height) of the vessel by using the lidar data, and use the acquired deck height of the vessel to generate feature points of the vessel. However, there may be cases where it is not possible to accurately obtain the deck height of the ship due to the lack of lidar data. In this case, the device 10 cannot determine which plane to use the camera image as the reference plane, If you do, the converted image may be severely distorted. Accordingly, when the device 10 according to an exemplary embodiment does not accurately obtain the deck height of the vessel, it may acquire the characteristic point of the vessel by using the converted images in which the region of the vessel is projected onto an arbitrary reference plane.

According to an embodiment, the apparatus 10 may acquire the feature point of the ship by using the converted images in which the area of the ship is projected on an arbitrary reference plane. For example, the apparatus 10 may align transformed images projected on an arbitrary reference plane having different regions of the vessel in the image using lidar data, and may acquire feature points of the vessel using the LIDAR data. Here, the device 10 may use the segmented image.

27 is a flowchart of a method of acquiring feature points of a ship using images in which a ship area is projected on a plurality of reference planes, according to an exemplary embodiment. Referring to FIG. 27 , in the method for acquiring feature points of a ship using images in which the ship area is projected on a plurality of reference planes according to an embodiment, detecting the ship area from a camera image (S2210), Projecting to a reference plane of (S2220), arranging images projected on arbitrary reference planes different from each other (S2230), and generating feature points of the vessel using the aligned images and lidar points (S2240) ) may be included.

The device 10 may detect the vessel area from the camera image (S2210).

For example, the device 10 may generate a segmented image by segmenting a camera image using an artificial neural network. Specifically, the device 10 may generate a segmentation image that detects the deck area and the side area of the vessel. As an example, the device 10 may generate a segmented image using panoptic segmentation.

The apparatus 10 may project the vessel area onto arbitrary reference planes different from each other ( S2220 ).

For example, the device 10 may project the segmented image for detecting the deck area and the side area of the ship on different reference planes using an artificial neural network. That is, the apparatus 10 may generate at least one of the first image and the second image by projecting the segmented image onto different first and second reference planes.

The apparatus 10 may align the images projected on arbitrary reference planes that are different from each other ( S2230 ).

According to an embodiment, the device 10 may align the images projected on any reference planes different from each other by using the lidar data. For example, the device 10 may align at least one of the images projected onto any different reference planes based on the positions of the lidar points relative to the lidar beam reflected from the vessel. Here, the projected images may be a projected image of a segmentation image that detects a deck area and a side area of a ship using an artificial neural network.

28 and 29 are diagrams for explaining alignment of images projected on arbitrary reference planes different from each other, according to an exemplary embodiment.

Referring to FIG. 28 , the device 10 may align the LIDAR points with images projected on arbitrary different reference planes while changing the reference plane. As the reference plane is changed, the relative position of the lidar point that is matched with the projected images may also change. The device 10 may align the images and the LiDAR points so that the LiDAR points are located in a specific area of the images projected on any different reference planes.

Referring to FIG. 28( a ), a first image in which a segmentation image for detecting the deck area 461 and the side area 462 of the ship is projected onto an arbitrary first reference plane can be seen. The device 10 may align the lidar points and the first image such that the lidar points associated with the lidar beam reflected from the vessel are located in a specific area in the first image. For example, when the device 10 aligns the first image with lidar points associated with a lidar beam reflected from the vessel, in the first image the lidar points associated with the lidar beam reflected from the vessel are segmented into the segmented image. It is possible to align the lidar points and the first image to be located in the projected area corresponding to the side area 462 of the vessel of . As an example, the device 10 may align the lidar points and the first image so that some of the lidar points related to the lidar beam reflected from the vessel are located in the region where the vessel and the sea level meet in the first image. .

Referring to FIG. 28( b ), a second image in which the segmentation image for detecting the deck area 463 and the side area 464 of the ship is projected on an arbitrary second reference plane different from the first reference plane can be seen. there is. The device 10 may align the lidar points and the second image such that the lidar points associated with the lidar beam reflected from the vessel are located in a specific area in the second image. For example, when the device 10 aligns the second image with lidar points associated with a lidar beam reflected from the vessel, in the second image the lidar points associated with the lidar beam reflected from the vessel are segmented into the segmented image. It is possible to align the lidar points and the second image so that they are located in the projected area corresponding to the deck area 463 of the ship. As an example, the device 10 may align the lidar points and the second image so that some of the lidar points associated with the lidar beam reflected from the vessel are located in the region where the deck of the vessel begins in the second image. there is.

Referring to FIG. 29 , it can be seen that images projected on different arbitrary reference planes aligned based on the position of the lidar point are fused. The device 10 may align the lidar points with the first image and the second image based on the relative positions of the lidar points in the first image and the second image.

For example, the device 10 may be configured such that in the first image the lidar points associated with the lidar beam reflected from the vessel are located below the projected region corresponding to the side region 462 of the vessel in the segmentation image, and in the second image may align the first image and the second image with the lidar points such that the lidar points associated with the lidar beam reflected from the vessel are located in a projected region corresponding to the deck region 463 of the vessel in the segmentation image. .

It will be described again by returning to FIG. 27 .

The device 10 may generate a feature point of the vessel using the aligned images and lidar points ( S2240 ).

For example, the device 10 may use the aligned lidar points and the projected first and second images to calculate the distance to another object (eg, another vessel, quay wall, etc.) You can earn points.

According to an embodiment, the apparatus 10 may acquire the feature points of the vessel in consideration of the LIDAR points estimated using the aligned LIDAR points and the projected first and second images.

For example, the device 10 extrapolates and/or interpolates the LiDAR points aligned with the projected first image and the second image to generate an estimated LiDAR point, the generated estimated LIDAR point and the existing LIDAR point can be used to acquire the ship's characteristic points. Referring to FIG. 29 , the device 10 may generate an estimated lidar point 465 by interpolating the lidar points, and may generate estimated lidar points 466 and 467 by extrapolating the lidar points. there is. The above description may be applied to the estimation of the lidar point or the acquisition of the characteristic point of the ship, and thus the detailed description will be omitted.

The device 10 may obtain the deck height of a ship in a variety of ways.

In one embodiment, the device 10 may use the sensor data to obtain the deck height of the vessel. For example, the device 10 may obtain the deck height of the vessel using at least one of an image and lidar data. For example, the apparatus 10 may obtain the deck height of the ship by using the coordinates of the LiDAR points corresponding to the portion in which the distance values of the LIDAR points related to the LIDAR beam reflected from the side of the vessel change rapidly. As another example, the device 10 detects the side and deck area of the ship using an artificial neural network capable of detecting the side and deck area of the ship, and a lidar corresponding to a portion where the side area and the deck area of the ship come into contact with each other. The deck height of the ship can be obtained using the coordinates of the points.

In an embodiment, the device 10 may obtain the deck height of the vessel through the communication module 300 . For example, the device 10 may receive AIS information of the vessel through the communication module 300 , and obtain the deck height of the vessel by using the received AIS information. As another example, the device 10 may obtain the deck height of the ship by receiving the deck height of the ship input from the external device through the communication module 300 .

Of course, the acquisition of the deck height of the ship is not limited to the above description and may be implemented in other ways, such as being obtained in another way.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication department
220: control unit
300: communication module

Claims (18)

A port monitoring method performed by computing means, comprising:
preparing an artificial neural network to be trained using a training set including a plurality of training images and object information labeled to pixels of the plurality of training images, wherein the object information includes a first index indicating that the type of object is a ship; having a second index indicating that the type of object is a sea, wherein the first index is labeled pixels corresponding to an area of the vessel in the plurality of training images, and wherein the second index is the sea in the plurality of training images. labeled pixels corresponding to the region of -;
acquiring an image captured by a camera;
acquiring lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera;
detecting a target vessel area corresponding to the target vessel from the acquired image using the artificial neural network;
generating a transformed image by projecting the target vessel area onto a specific reference plane;
selecting LiDAR points related to the LIDAR beam reflected from the target vessel in consideration of pixel positions of pixels included in the projected area of the target vessel of the converted image;
generating estimated lidar points using the selected lidar points;
determining a characteristic point of the target vessel among the selected lidar points and the generated estimated lidar points; and
Calculating a distance between the target ship and another object by using the feature point; including
How to monitor ports.
According to claim 1,
The specific reference plane includes a reference plane at sea level
How to monitor ports.
3. The method of claim 2,
The estimated lidar point is
generated from LiDAR points located on a line where the target ship and the sea level meet in the converted image among the selected LiDAR points.
How to monitor ports.
4. The method of claim 3,
The estimated lidar point is
generated by extrapolation and/or interpolation in consideration of the relative positions of the lidar points located on the line where the target ship and the sea level are in contact
How to monitor ports.
According to claim 1,
wherein the specific reference plane comprises a reference plane at the deck level of the target vessel.
How to monitor ports.
6. The method of claim 5,
The estimated lidar point is
Among the selected LiDAR points, generated from LiDAR points located on a line in which the side and deck of the target ship in the converted image are in contact.
How to monitor ports.
7. The method of claim 6,
The estimated lidar point is
generated by extrapolating and/or interpolating in consideration of the relative positions of the lidar points located on the line where the side surface of the target ship and the deck are in contact
How to monitor ports.
According to claim 1,
the specific reference plane includes any first and second reference planes,
The converted image includes a first converted image and a second converted image,
The first transformed image is generated by projecting the target vessel area onto the first reference plane,
The second converted image is generated by projecting the target vessel area onto the second reference plane.
How to monitor ports.
9. The method of claim 8,
The first transformed image and the second transformed image are aligned based on the positions of the plurality of lidar points.
How to monitor ports.
According to claim 1,
The characteristic point of the target vessel includes a point corresponding to at least one of a bow and a stern of the target vessel.
How to monitor ports.
11. The method of claim 10,
The feature point of the target vessel includes at least one arbitrary point among LiDAR points located on a line in which the target vessel and the sea level of the converted image are in contact.
How to monitor ports.
11. The method of claim 10,
The feature point of the target vessel includes LiDAR points located at both ends of the target vessel area of the converted image.
How to monitor ports.
According to claim 1,
The artificial neural network is
Further comprising a third index indicating that the type of the object is a side of the vessel, and a fourth index indicating that the type of the object is a deck of the vessel
How to monitor ports.
A port monitoring method performed by computing means, comprising:
acquiring an image captured by a camera;
acquiring lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera;
generating a segmentation image from the image, the segmentation image comprising a vessel region corresponding to a vessel, wherein the vessel region comprises a side region corresponding to a side of the vessel and a deck region corresponding to a deck region of the vessel box-;
Projecting the vessel region onto any first and second reference planes to generate at least one of a first transformed image or a second transformed image, the heights of the first and second reference planes being different from each other ;
aligning lidar points related to a lidar beam reflected from the vessel among the plurality of lidar points to at least one of the first transformed image and the second transformed image;
Interpolating and/or extrapolating LiDAR points associated with a LiDAR beam reflected from the vessel according to the shape of the vessel area projected in at least one of the first transformed image and the second transformed image to generate an estimated lidar point to do;
determining a characteristic point of the vessel among lidar points related to the lidar beam reflected from the vessel and the estimated lidar point; and
Calculating a distance between the target ship and another object by using the feature point; including
How to monitor ports.
preparing an artificial neural network to be trained using a training set including a plurality of training images and object information labeled to pixels of the plurality of training images, wherein the object information includes a first index indicating that the type of object is a ship; having a second index indicating that the type of object is a sea, wherein the first index is labeled pixels corresponding to an area of the vessel in the plurality of training images, and wherein the second index is the sea in the plurality of training images. labeled pixels corresponding to the region of -;
acquiring a sea image including a first vessel and a second vessel in which the deck is located at different heights;
detecting a first vessel region corresponding to the first vessel and a second vessel region corresponding to the second vessel from the acquired sea image using the artificial neural network;
obtaining a position of the first vessel and a position of the second vessel based on a first height corresponding to the deck height of the first vessel and a second height corresponding to the deck height of the second vessel; and
Calculating a distance between the first vessel and the second vessel based on the position of the first vessel and the position of the second vessel; including
How to monitor ports.
16. The method of claim 15,
The step of obtaining the position of the first vessel and the position of the second vessel is
generating a first transformed image by projecting the first vessel region onto a reference plane at the first height;
obtaining the position of the first vessel by using the first converted image;
projecting the second vessel area onto a reference plane at the second height to generate a second transformed image; and
Comprising the step of obtaining the position of the second vessel by using the second converted image
How to monitor ports.
17. The method of claim 16,
The step of obtaining the position of the first vessel is
obtaining a first position of at least one of a bow and a stern of the first vessel by using a pixel position of the region of the first vessel projected from the first transformed image;
The step of obtaining the position of the second vessel is
Acquiring a second position of at least one of a bow and a stern of the second vessel by using the pixel position of the second vessel region projected from the second transformed image
How to monitor ports.
A computer-readable recording medium in which a program for performing the method of any one of claims 1 to 17 is recorded.

Images