KR20230074438A - Device and method for monitoring ship and port - Google Patents

Abstract

The present invention relates to an apparatus and method for monitoring a ship and a harbor, and according to an embodiment, a harbor monitoring method performed by a computing means includes a plurality of training images and object information labeled on pixels of the plurality of training images. Preparing an artificial neural network to be learned using a learning set to be trained - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, and the first index is the plurality of pixels corresponding to the region of the ship in the training image of, and the second index is labeled to the pixels corresponding to the region of the sea in the plurality of training images-, obtaining an image captured by the camera Acquiring lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera, using the artificial neural network to target a target from the acquired image Detecting a target ship region corresponding to the ship, generating a converted image by projecting the target ship region onto a specific reference plane, considering the pixel positions of pixels included in the projected target ship region of the converted image Selecting lidar points related to the lidar beam reflected from the target ship, generating estimated lidar points using the selected lidar points, the selected lidar points and the generated estimated lidar points Among them, a harbor monitoring method may be provided, including determining a feature point of the target ship and calculating a distance between the target ship and another object using the feature point.

Description

Ship and port monitoring device and method {DEVICE AND METHOD FOR MONITORING SHIP AND PORT}

The present invention relates to an apparatus and method for monitoring a ship and a port, and more particularly, to an apparatus and method for monitoring a ship and a harbor for estimating lidar data related to a ship using a camera image.

Many accidents occur in the operation of ships, berthing and unloading in ports, and the main cause of these accidents is known to be carelessness of human operation. Here, negligence in operation is mainly caused by the fact that the situation around the ship or in the port cannot be accurately monitored through the naked eye. Currently, various types of obstacle sensors are used to supplement this, but there are still limitations.

Recently, a technology for monitoring the situation around a ship or in a port through images has been developed, but it is being used together with LIDAR to improve the accuracy of monitoring. However, in the case of lidar used in ports, there is a problem that it is difficult to obtain even the minimum lidar data required for monitoring because of low performance due to cost issues and the large size of the ship to be monitored.

Therefore, it is necessary to develop a technology for estimating additional lidar data required for monitoring from the obtained lidar data.

One problem to be solved by the present application is to provide a ship and port monitoring device and method that efficiently converges camera and lidar data.

One problem to be solved by the present application is to provide a ship and port monitoring apparatus and method for estimating lidar data related to the ship using camera images.

The problem to be solved in the present application is not limited to the above-described problem, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. .

According to an embodiment, a method of monitoring a port performed by a computing means, preparing an artificial neural network to be learned using a plurality of training images and a learning set including object information labeled on pixels of the plurality of training images. - The object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, the first index being applied to pixels corresponding to the region of the ship in the plurality of training images. labeled, and the second index is labeled pixels corresponding to the area of the sea in the plurality of training images—obtaining an image captured by a camera, a field of view that at least partially overlaps with a field of view of the camera Acquiring lidar data including a plurality of lidar points obtained by a lidar sensor having ; detecting a target ship area corresponding to a target vessel from the acquired image using the artificial neural network; Projecting a ship area to a specific reference plane to generate a converted image, a lidar point related to a lidar beam reflected from the target ship in consideration of a pixel position of a pixel included in the projected target ship area of the converted image selecting, generating estimated lidar points using the selected lidar points, determining feature points of the target ship among the selected lidar points and the generated estimated lidar points, and the A harbor monitoring method including calculating a distance between the target ship and another object using a feature point may be provided.

According to an embodiment, as a harbor monitoring method performed by a computing means, obtaining an image captured by a camera, a plurality of lidars obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Acquiring lidar data including points, generating a segmentation image from the image -The segmentation image includes a ship area corresponding to a ship, and the ship area includes a side area corresponding to a side of the ship and including a deck area corresponding to the deck area of the vessel, generating at least one of a first transformed image or a second transformed image by projecting the vessel area onto an arbitrary first reference plane and a second reference plane; Among the plurality of lidar points, which have different heights from the first reference plane and the second reference plane, lidar points related to the lidar beam reflected from the ship are selected from at least one of the first converted image and the second converted image. aligning to one, interpolating and/or extrapolating lidar points associated with a lidar beam reflected from the vessel according to the shape of the vessel area projected in at least one of the first transformed image and the second transformed image Generating an estimated lidar point, determining a feature point of a ship among lidar points related to a lidar beam reflected from the ship and the estimated lidar point, and using the feature point to determine a different point from the target ship. A harbor monitoring method including calculating a distance to an object may be provided.

According to one embodiment, a method of monitoring a seaport performed by a computing means, preparing an artificial neural network to be learned using a learning set including a plurality of training images and object information labeled on pixels of the plurality of training images Step - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, the first index being pixels corresponding to an area of the ship in the plurality of training images. is labeled, and the second index is labeled pixels corresponding to the areas of the sea in the plurality of training images-, a marine image including a first ship and a second ship whose decks are located at different heights from each other. Obtaining a first ship area corresponding to the first ship and a second ship area corresponding to the second ship from the acquired maritime image using the artificial neural network; Obtaining a location of the first vessel and a location of the second vessel based on a first height corresponding to a deck height and a second height corresponding to a deck height of the second vessel, and a location of the first vessel, and Based on the location of the second vessel, calculating a distance between the first vessel and the second vessel; may be provided with a harbor monitoring method comprising the.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. You will be able to.

According to an embodiment of the present application, it is possible to extract feature points of a ship by efficiently converging camera and lidar data, and accurately provide information for monitoring using this.

According to another embodiment of the present application, it is possible to solve the problem of insufficient lidar data by estimating lidar data related to a ship using a camera image.

The effects of the present application are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

1 is a diagram of a port monitoring device according to an embodiment.
2 and 3 are diagrams of an embodiment of a monitoring device according to an embodiment.
4 is a view related to a viewing angle and a depth of field according to an exemplary embodiment.
5 and 6 are diagrams of an installation position of a sensor module according to an embodiment.
7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.
10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
12 is a diagram for explaining eyepiece guide information according to an embodiment.
13 is a diagram for explaining berthing guide information between a ship and a quay according to an embodiment.
14 is a diagram for explaining a method of acquiring eyepiece guide information according to an embodiment.
15 is a diagram related to viewpoint conversion according to an exemplary embodiment.
16 is a flowchart of a method for calculating a distance based on feature points of a vessel according to an embodiment.
17 is an example of acquisition of sensor data according to an embodiment.
18 is a flowchart of a method of generating feature points of a ship using an image in which a viewpoint is converted according to an embodiment.
19 is an example of a converted image according to an embodiment.
20 is a diagram for explaining distance calculation between ships according to an embodiment.
21 is a flowchart of a lidar point estimation method according to an embodiment.
22 is an example of matching camera images and lidar data according to an embodiment.
23 is a flowchart of a method of generating feature points of a vessel considering estimated lidar points according to an embodiment.
24 is an example of lidar points related to a lidar beam reflected from a ship according to an embodiment.
25 and 26 are diagrams for explaining generation of estimated lidar points using lidar points matched to a ship area projected on a reference plane according to an embodiment.
27 is a flowchart of a method for acquiring feature points of a ship using images in which a ship area is projected onto a plurality of reference planes according to an embodiment.
28 and 29 are views for explaining alignment of images projected on different arbitrary reference planes according to an exemplary embodiment.

The embodiments described in this specification are intended to clearly explain the spirit of the present invention to those skilled in the art to which the present invention belongs, and the present invention is not limited by the embodiments described in this specification, and the present invention The scope of should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as general terms that are currently widely used as much as possible in consideration of the functions in the present invention, but they may vary depending on the intention, custom, or the emergence of new technologies of those skilled in the art in the technical field to which the present invention belongs. can However, in the case where a specific term is defined and used in an arbitrary meaning, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not the simple name of the term.

The drawings accompanying this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

If it is determined that a detailed description of a known configuration or function related to the present invention in this specification may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

In this specification, sea level height should be broadly interpreted to include not only absolute sea level height, but also relative sea level height, relative sea level height compared to the average sea level height of a specific sea area, and the like. For example, the sea level height may include a variety of information that may vary depending on the change in sea level height, such as the distance between the sea level and the quay wall, the distance between the sea level and the monitoring device (e.g., an image generating unit), and the length of an object exposed above the sea level. may also include

In this specification, monitoring refers to grasping or recognizing a surrounding situation, and not only detecting a sensing object such as a certain area or a specific object using various sensors and providing the sensing result to the user, but also through calculation based on the sensing result. It should be interpreted broadly, including providing additional information.

Image-based monitoring may mean grasping or recognizing a surrounding situation based on an image. For example, monitoring may mean obtaining information for recognizing other vessels or obstacles by acquiring images around the vessel during operation of the vessel, or calculating berthing guide information when the vessel is berthing or unloading.

Berthing guide information is information about the environment, such as recognition of other ships or obstacles, identification of port conditions, access to berths, distance and speed from the quay, distance and speed from other ships, and identification of obstacles on the navigation route. can mean In the present specification, monitoring is mainly described when berthing is performed in a ship and a port, but is not limited thereto and may be applied to driving of a vehicle, operation of an aircraft, and the like.

According to one aspect of the present invention, as a method for monitoring ports performed by a computing means, an artificial neural network learned using a learning set including a plurality of training images and object information labeled on pixels of the plurality of training images Preparing - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, the first index corresponding to an area of the ship in the plurality of training images pixels are labeled, and the second index is labeled pixels corresponding to the area of the sea in the plurality of training images—obtaining an image taken by a camera, at least partially at a field of view of the camera Acquiring lidar data including a plurality of lidar points obtained by lidar sensors having overlapping viewing angles, detecting a target ship area corresponding to the target vessel from the acquired image using the artificial neural network , generating a conversion image by projecting the target ship region onto a specific reference plane, in consideration of pixel positions of pixels included in the projected target ship region of the conversion image, related to the lidar beam reflected from the target ship. Selecting lidar points, generating estimated lidar points using the selected lidar points, determining a feature point of the target ship among the selected lidar points and the generated estimated lidar points and calculating a distance between the target ship and another object using the feature point may be provided.

In one embodiment, the specific reference plane may include a reference plane at sea level.

In one embodiment, the estimated lidar point may be generated from lidar points located on a line where the target ship and the sea level touch in the converted image among the selected lidar points.

In one embodiment, the estimated lidar point may be generated by extrapolation and/or interpolation in consideration of relative positions of the lidar points located on the line where the target ship and the sea level come into contact.

In one embodiment, the specific reference plane may include a reference plane at deck height of the target ship.

In one embodiment, the estimated lidar point may be generated from lidar points located on a line where the side of the target ship and the deck come into contact in the converted image among the selected lidar points.

In one embodiment, the estimated lidar point may be generated by extrapolation and/or interpolation in consideration of relative positions of the lidar points located on the line where the side of the target ship and the deck come into contact.

In one embodiment, the specific reference plane includes any first reference plane and a second reference plane, the converted image includes a first converted image and a second converted image, and the first converted image includes the target The ship area may be generated by projecting the first reference plane, and the second converted image may be generated by projecting the target ship area onto the second reference plane.

In one embodiment, the first conversion image and the second conversion image may be aligned based on positions of the plurality of lidar points.

In one embodiment, the feature point of the target ship may include a point corresponding to at least one of a bow and a stern of the target ship.

In one embodiment, the feature point of the target ship may include at least one arbitrary point among lidar points located on a line where the target ship and the sea surface of the converted image come into contact.

In one embodiment, the feature points of the target ship may include lidar points located at both ends of the target ship area of the converted image.

In one embodiment, the artificial neural network may further include a third index indicating that the type of object is a side of a ship and a fourth index indicating that the type of object is a deck of a ship.

According to another aspect of the present invention, as a harbor monitoring method performed by a computing means, obtaining an image taken by a camera, a plurality of obtained by a lidar sensor having a field of view that at least partially overlaps the field of view of the camera Acquiring lidar data including lidar points of, generating a segmentation image from the image, wherein the segmentation image includes a ship area corresponding to a ship, and the ship area corresponds to a side of the ship. including a side area and a deck area corresponding to a deck area of the vessel, projecting the vessel area onto an arbitrary first reference plane and a second reference plane to generate at least one of a first transformation image or a second transformation image; Step of doing - the first conversion image and the second conversion lidar points related to the lidar beam reflected from the ship among the plurality of lidar points, the heights of which are different from each other of the first reference plane and the second reference plane aligning to at least one of the images, interpolating and/or interpolating lidar points associated with a lidar beam reflected from the vessel according to the shape of the vessel area projected in at least one of the first transformed image and the second transformed image; Alternatively, generating an estimated lidar point by extrapolation, determining a feature point of the ship among lidar points related to a lidar beam reflected from the ship and the estimated lidar point, and using the feature point to determine the target A port monitoring method may be provided that includes calculating a distance between a ship and another object.

According to another aspect of the present invention, an artificial neural network learned by using a learning set including a plurality of training images and object information labeled on pixels of the plurality of training images, in a term monitoring method performed by a computing means. The object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, the first index corresponding to an area of the ship in the plurality of training images. labeled pixels, and the second index is labeled pixels corresponding to the area of the sea in the plurality of training images, including a first ship and a second ship whose decks are located at different heights. obtaining a maritime image that corresponds to the first ship and detecting a first ship area corresponding to the first ship and a second ship area corresponding to the second ship from the acquired maritime image using the artificial neural network; Obtaining a location of the first ship and a location of the second ship based on a first height corresponding to a deck height of the first ship and a second height corresponding to a deck height of the second ship, and the first ship A port monitoring method comprising the; calculating a distance between the first vessel and the second vessel based on the position of the second vessel and the position of the second vessel may be provided.

In one embodiment, the obtaining of the position of the first vessel and the position of the second vessel comprises: generating a first transformed image by projecting the first vessel area onto a reference plane at the first height; Obtaining a position of the first ship using a first converted image, generating a second converted image by projecting the second ship area onto a reference plane at the second height, and generating a second converted image It may include obtaining the location of the second vessel by using.

In one embodiment, the obtaining of the position of the first ship may include a first position of at least one of the bow and stern of the first ship by using a pixel position of the first ship area projected from the first converted image. And obtaining the position of the second ship, wherein the step of acquiring the position of at least one of the bow and stern of the second ship using the pixel position of the second ship area projected in the second converted image. It may include obtaining a second location.

According to another aspect of the present invention, a computer-readable recording medium on which a program for performing the above-described methods is recorded may be provided.

Hereinafter, an apparatus and method for monitoring ships and ports according to an embodiment will be described.

1 is a diagram of a port monitoring device according to an embodiment.

Referring to FIG. 1 , the monitoring device 10 may include a sensor module 100 , a control module 200 and a communication module 300 .

The sensor module 100 may sense information about a ship or its surroundings and a port. The sensor module 100 may include an automatic identification system (AIS), an image generating unit, a LIDAR sensor, a position measurement unit, a posture measurement unit, and a casing.

The image generating unit may generate an image. The image generating unit may include a camera, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

A lidar sensor is a sensor for detecting a distance and position of an object using a laser. For example, the distance between the lidar sensor and the target object and the position of the target object based on the lidar sensor may be expressed in a 3D coordinate system. For example, the distance between the lidar sensor and the object and the position of the object based on the lidar sensor may be expressed in a Cartesian coordinate system, a spherical coordinate system, a cylindrical coordinate system, and the like. The lidar sensor may have a plurality of channels in a vertical or horizontal direction, and may be, for example, a lidar sensor having 32 or 64 channels.

The lidar sensor may use laser reflected from the target object to determine the distance R from the target object. For example, the lidar sensor may use time of flight (TOF), which is a time difference between an emitted laser and a detected laser, to determine a distance to an object. To this end, the lidar sensor may include a laser output unit for outputting a laser beam and a receiver unit for detecting the reflected laser beam. The lidar sensor checks the time when the laser is output from the laser output unit and the time when the receiver detects the laser reflected from the object, and determines the distance to the object based on the difference between the emitted time and the detected time. can do. Of course, the lidar sensor uses other methods such as triangulation based on the detected position of the detected laser to determine the distance R from the target object, or using phase shift of the detected laser. R) may be determined.

The lidar sensor may determine the position of the target object using the angle of the irradiated laser. For example, when the lidar sensor can know the irradiation angle of one laser irradiated toward the scan area of the lidar sensor, if the laser reflected from the object existing on the scan area is detected by the receiver, the lidar sensor The sensor may determine the location of the object based on an irradiation angle of the irradiated laser.

A lidar sensor may have a scan area including an object in order to detect a position of an object in the vicinity. Here, the scan area represents a detectable area as one screen, and may mean a set of points, lines, and planes forming one screen for one frame. In addition, the scan area may mean the irradiation area of the laser irradiated from the lidar sensor, and the irradiation area may mean a set of points, lines, and planes where the laser irradiated during one frame meets a sphere at the same distance (R). can In addition, a field of view (FOV) means a detectable area, and may be defined as an angular range of a scan area when viewing the LIDAR sensor as an origin.

The position measurement unit may measure the position of components included in the sensor module 100, such as the sensor module 100 or the image generating unit. For example, the location measurement unit may be a Global Positioning System (GPS). In particular, RTK-GPS (Real-Time Kinematic GPS) may be used to improve the accuracy of position measurement.

The location measurement unit may obtain location information at predetermined time intervals. Here, the time interval may vary according to the installation location of the sensor module 100. For example, when the sensor module 100 is installed on a moving object such as a ship, the position measurement unit may acquire position information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a port, the position measurement unit may obtain position information at long time intervals. A time interval for acquiring location information of the location measurement unit may be changed.

The posture measurement unit may measure the posture of components included in the sensor module 100, such as the sensor module 100 or the image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measurement unit may obtain posture information at every predetermined time interval. Here, the time interval may vary depending on the installation location of the sensor module. For example, when the sensor module 100 is installed on a moving object such as a ship, the attitude measurement unit may acquire attitude information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a harbor, the attitude measurement unit may obtain attitude information at every long time interval. A time interval for acquiring attitude information of the attitude measuring unit may be changed.

The casing may protect sensor modules such as an image generating unit, a position measurement unit, and a posture measurement unit.

At least one of an image generating unit, a position measurement unit, and a posture measurement unit may be present inside the casing. The casing can prevent equipment, such as an image generating unit, from being corroded by salt water. Alternatively, the casing may protect the equipment existing therein by preventing or mitigating impact applied thereto.

A cavity may be formed inside the casing to contain an image generating unit or the like therein. For example, the casing may have a hollow rectangular parallelepiped shape, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like may be disposed.

When the image generating unit is disposed inside the casing, an opening may be formed in one region of the casing or a transparent material such as glass may be formed in one region of the casing to secure the field of view of the image generating unit. The image generation unit may capture an image of the ship's periphery and the harbor through the opening or the transparent area.

The casing may be made of a strong material to protect the image generating unit from external impact. Alternatively, the casing may be made of a material such as an alloy for seawater to prevent corrosion due to salt.

The casing may include equipment for removing foreign substances from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove foreign substances. As another example, foreign substances may be removed by applying water or washer fluid through a liquid spray included in the casing, or foreign substances may be physically removed using a wiper after application.

The foreign matter removal equipment may be operated manually, but may also be operated automatically. For example, the foreign material removal equipment may be operated at predetermined time intervals. Alternatively, the foreign material removal equipment may be operated using a sensor that detects whether or not a foreign material is attached to the image generating unit. Alternatively, after determining whether a foreign material is captured in the image using the image captured by the image generating unit, the foreign material removing device may be operated when it is determined that the foreign material exists. Here, whether a foreign substance is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as two or more identical cameras.

The control module 200 may perform image analysis. In addition, an operation of receiving various data through the sensor module 100, an operation of outputting various outputs through an output device, an operation of storing various data in a memory or acquiring various data from a memory, and the like are of the control module 200. It can be done by control. Hereinafter, various operations or steps disclosed as embodiments of this specification may be interpreted as being performed by the control module 200 or under the control of the control module 200 unless otherwise stated.

Examples of the control module 200 include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), a state machine, and an on-demand Semiconductors (Application Specific Integrated Circuits, ASICs), radio-frequency integrated circuits (Radio-Frequency Integrated Circuits, RFICs), and combinations thereof may be used.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or unidirectional communication. For example, the device 10 may transfer information to an external output device through the communication module 300 and output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.

The sensor module 100, the control module 200, and the communication module 300 may include a control unit. The control unit may process and calculate various types of information within the module, and may control other components constituting the module. The controller may be physically provided in the form of an electronic circuit that processes electrical signals. A module may physically include only a single control unit, but may also include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be provided as processors that are mounted on physically separated servers and terminals and collaborate through communication. Examples of the control unit include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, and an application specific semiconductor (Application Specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100, the control module 200, and the communication module 300 may include a communication unit. The modules may transmit and receive information through a communication unit. For example, the sensor module 100 may transmit information acquired from the outside through the communication unit, and the control module 200 may receive the information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100, the control module 200, and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing program processing, or such processing result data. For example, the memory may store data necessary for learning and/or reasoning, learning is in progress or a learned artificial neural network, and the like. Memory is a non-volatile semiconductor memory, hard disk, flash memory, RAM (Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or any other tangible non-volatile recording medium. etc. can be implemented.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100, and each sensor module 100 may further include two cameras.

2 and 3 are diagrams of an embodiment of a monitoring device according to an embodiment.

Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100 and a control module 200. The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may change the viewpoint of the image by performing a viewpoint conversion to be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 and perform image analysis such as location/movement information estimation and image matching, which will be described later, through the control unit 220. In addition, the control module 200 may transmit analysis results such as location/movement information and matched images to the cloud server through the communication unit 210 . The cloud server may transmit the analysis result received from the control module 200 to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

Referring to FIG. 3 , the monitoring device 10 may include a sensor module 100 . The sensor module 100 may generate an image through the camera 130 and transmit the image to a cloud server through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may change the viewpoint of the image by performing a viewpoint conversion to be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as location/movement information estimation and image matching, which will be described later. In addition, the cloud server may transmit an image analysis result to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

The device 10 shown in FIGS. 1 to 3 is only an example, and the configuration of the device 10 is not limited thereto.

As an example, the device 10 may include an output module 400 . The output module 400 may output a result of an operation performed by the control module 200 and the like. For example, the output module 400 may output analysis results. The output module 400 may be, for example, a display, a speaker, a signal output circuit, etc., but is not limited thereto. In this case, information may be output through the output module 400 in addition to outputting the information by transferring the information to an external output device such as a user terminal.

As another example, device 10 may not include a sensor module. In this case, the control module 200 may perform an image-based monitoring operation such as performing image analysis by receiving information from an external sensor device. For example, the control module 200 may perform image analysis by receiving information from an AIS, camera, lidar, radar, or the like already installed in a ship or port.

In addition, the steps performed by each component of FIGS. 1 to 3 are not necessarily performed by the corresponding component, but may be performed by other components. For example, in FIG. 2 above, it is described that the controller 120 of the sensor module 100 performs the viewpoint conversion, but the controller 220 of the control module 200 or the cloud server may perform the viewpoint conversion. .

Hereinafter, the monitoring device 10 and method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100 . For example, an image may be obtained through an image generation unit included in the sensor module 100 . Alternatively, as described above, an image may be obtained from an external sensor device. Images for vessel and harbor monitoring will generally include sea, vessel, buoy, obstacle, terrain, harbor, sky, building, and the like. Hereinafter, a method of analyzing and monitoring an image acquired through a visible ray camera will be mainly described, but is not limited thereto.

Depending on the image generating unit, a field of view (FOV) and a depth of field may vary. 4 is a view related to a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 4 , the field of view (FOV) may mean how much range is included in an image horizontally or vertically, and is generally expressed as an angle (degree). The meaning of having a larger viewing angle may mean generating an image including an area with a wider width horizontally or vertically. The depth of field may mean a distance range recognized as being in focus, and the depth of field may mean that the distance range recognized as being in focus is wide. Referring to FIG. 4 , according to the depth of field (DOF), the image may include an area A1 recognized as being in focus and an area A2 other than that. Hereinafter, the area included in the image is referred to as the imaging area (A1 + A2), and the area recognized as being in focus is referred to as the effective area (A1), and image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on or a part of the imaging area, the area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and shallow depth of field is a wide-angle camera. Examples of a camera with a small field of view and a deep depth of field include a high-magnification camera and a zoom camera.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, crane, ship, etc. in a port, and there is no limitation in the number thereof. However, depending on the type and performance of the sensor module 100, the installation location or number may vary. For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more from the surface of the water for efficient monitoring, or a plurality may be installed to have different imaging areas. In addition, the position and attitude of the sensor module 100 may be manually or automatically adjusted during or after installation.

A field of view of the lidar may at least partially overlap a field of view of an image generating unit (eg, a camera). For example, the field of view (FOV) of the lidar may be smaller than the field of view of the camera. As will be described later, in this case, lidar data outside the field of view of lidar where lidar data is not acquired may be estimated using an image obtained from a camera.

5 and 6 are diagrams of an installation position of a sensor module according to an embodiment. Referring to FIGS. 5 and 6 , the sensor module 100 may be installed in a fixed location such as a port or land, or installed in a moving object such as a ship. Here, when the sensor module 100 is installed on a ship, it can be installed on a ship to be monitored (hereinafter referred to as “target ship”) as shown in FIG. 6, and as shown in FIG. It can also be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition to this, the sensor module 100 may be installed in a drone or the like to monitor a target ship.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or in a separate location.

As described above, image analysis for image-based monitoring may include obtaining object characteristics. Examples of objects may include ships, harbors, buoys, seas, terrain, sky, buildings, people, animals, fire, and smoke. Examples of object properties may include the type of object, the position of the object, the distance to the object, the absolute and relative speed and speed of the object, and the like.

Image analysis for image-based monitoring may include recognizing/determining the surrounding situation. For example, image analysis may determine that a fire situation has occurred from an image of a fire in a port, or that an intruder has entered from a captured image of a person entering a port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each module 100 and 200 .

The device 10 can recognize objects. For example, the device 10 may recognize an object included in an image. For example, the device 10 may determine whether an object such as a ship, a tugboat, the sea, or a harbor is included in the image. Here, object recognition may be determining which position of an image an object exists.

7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.

7 is an image taken by a camera, and an object can be recognized as shown in FIG. 8 or 9 through an object recognizing step.

Specifically, FIG. 8 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation. In this case, the object recognition step may mean the segmentation step. Through segmentation, a characteristic corresponding to a pixel on the image may be allocated or calculated from the image. One might say that a pixel is assigned or labeled with a property. Referring to FIGS. 7 and 8 , a segmented image as shown in FIG. 8 may be obtained by performing segmentation based on an image captured by the camera of FIG. 7 . In FIG. 8 , the first pixel area P1 is an area on the image of a pixel corresponding to a ship, the second pixel area P2 is the sea, the third pixel area P3 is the quay wall of the harbor, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area P5 is an area on the image of a pixel corresponding to the sky.

Although FIG. 8 shows that information about the type of object corresponding to each pixel on an image is calculated by performing segmentation, information that can be obtained through segmentation is not limited thereto. For example, characteristics such as location, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, different characteristics may be expressed independently or may be reflected and expressed at the same time.

Table 1 is a table related to labeling that simultaneously reflects information on the type and distance of an object according to an embodiment. Referring to Table 1, classes may be set in consideration of object type and distance information, and an identification value may be assigned to each class. For example, identification value 2 may be assigned by considering both topography, which is information about the type of object, and short distance, which is information about distance. Table 1 is an example of a case in which information on type and information on distance are considered together. In addition, other information such as direction information, moving direction of obstacles, speed, and navigational aids may also be considered. In addition, all identification values do not have to include a plurality of pieces of information, nor do they have to include the same type of information. For example, a specific identification value includes only type information (eg, identification value 1 does not include information about distance), and other identification values include information about type and distance. It can be expressed in various ways. Other classes, such as tugboat, rope, ship's side and deck, for example, may be added or modified to other classes.

identification value class 0 sky and other One ocean 2 terrain + close range 3 Terrain + Medium 4 Terrain + Far 5 fixed obstacles + short range 6 fixed obstacles + medium distance 7 fixed obstacles + far 8 Dynamic Obstacles + Short Range 9 Dynamic Obstacles + Medium Range 10 Dynamic Obstacles + Far

FIG. 9 shows where an object exists in an image with a bounding box, which is also referred to as detection. In this case, the object recognition step may mean the detection step. Compared to segmentation, detection can be seen as detecting a position in which an object is included in a box shape rather than calculating characteristics for each pixel of an image. Referring to FIGS. 7 and 9 , a detection image as shown in FIG. 9 may be obtained by performing detection based on an image captured by the camera of FIG. 7 . In FIG. 9 , it can be seen that the ship is detected on the image and the location of the ship is expressed as a rectangular bounding box (BB). Although only one object is illustrated in FIG. 9 as being detected, two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks and combine the results to calculate a final result.

An artificial neural network is a type of algorithm modeled after the structure of a human neural network, and may include one or more layers including one or more nodes or neurons, and each node may be connected through a synapse. Data (input data) input to the artificial neural network may be output (output data) through nodes through synapses, through which information may be obtained.

Types of artificial neural networks include a convolution neural network (CNN) that extracts features using a filter and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back to an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), and relational networks (RN) can be applied, and there are limitations. It is not.

Before using an artificial neural network, a training step is required. Alternatively, it may be trained using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and the step of using it as an inference step.

The artificial neural network may be trained through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.

10 is an embodiment of a learning step of an artificial neural network, in which an untrained artificial neural network receives learning data or training data, outputs output data, and compares the output data with labeling data. The artificial neural network can be trained through backpropagation of the error. Training data, output data, and labeling data may be images. Labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or program.

11 is an example of an artificial neural network inference step, in which the trained artificial neural network receives input data and outputs output data. Information that can be inferred in the inference step may vary depending on the information of the learning data in the learning step. In addition, the accuracy of output data may vary according to the degree of learning of the artificial neural network.

Recognition of an object is not limited to the above description and may be implemented in other ways. For example, although it has been described that an identification value is used to recognize an object for convenience of description, the identification value is only used as one type of index. For example, an index in the form of a vector rather than a value may be used to recognize an object, and training data, output data, and labeling data of an artificial neural network may be data in the form of a vector.

According to one embodiment, the device 10 may provide berthing guide information for a ship. The berthing guide information may be used for berthing of a ship and may refer to information for assisting or guiding a user such as a pilot or a captain in berthing. Eyepiece guide information may include information on distance/velocity. Examples of information on distance/velocity include, but are not limited to, an absolute position such as coordinates, a relative position from a specific reference point, a distance from an arbitrary point, a distance range, a direction, an absolute speed, a relative speed, and a speed.

Information on distance/velocity may be estimated based on a predetermined area or point. For example, the distance between the ship and the quay may be estimated by calculating the distance between one point of the ship and one point of the quay, or may be estimated by calculating the shortest distance between one point of the ship and the quay. As another example, the distance between ships may be estimated by calculating the distance between one point of the first ship and one point of the second ship. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

Information on distance/speed may be expressed as a predetermined distance value, direction value, and speed value. For example, distance information may be expressed as 1m, 2m, etc., direction information may be expressed as 10°, 20°, etc., and speed information may be expressed as 5cm/s, 10cm/s, etc.

Information on distance/speed may be expressed as an index corresponding to a plurality of categories having a certain range. For example, distance information may be expressed as short distance, medium distance, and long distance, direction information may be expressed as left direction, front direction, and right direction, and speed information may be expressed as low speed, medium speed, and high speed. there is. It will also be possible to combine these and express them as left near distance, right far distance, and the like.

12 is a diagram for explaining eyepiece guide information according to an embodiment. Referring to (a) of FIG. 12, the berthing guide information is berthing guide information between the ship (OBJ1) and the quay (OBJ2) (f1, f2) and the berthing guide between the ship (OBJ1) and other ships (OBJ3, OBJ4) Information f3 and f4 may be included. Specifically, the berthing guide information (f1, f2) between the ship (OBJ1) and the quay wall (OBJ2) may include information on the distance/velocity of the ship (OBJ1) to the quay wall (OBJ2), and the ship (OBJ1) The berthing guide information f3 and f4 between the ship OBJ3 and the other ships OBJ3 and OBJ4 may include information about the distance/velocity of the ship OBJ1 to the other ships OBJ3 and OBJ4. Referring to (b) of FIG. 12, berthing guide information (f1, f2) between the ship OBJ1 and the quay wall OBJ2 is the boundary area corresponding to the area in contact with the sea surface of the ship OBJ1 and the quay wall OBJ2. ) may be information between. The boundary area may include not only a line or a point where the ship and the sea surface meet, but also a certain area near the line or point. In addition, the berthing guide information between the ship (OBJ1) and the other ships (OBJ3, OBJ4) may be information between a predetermined area corresponding to the bow / stern of the vessel (OBJ1, OBJ3, OBJ4), where the predetermined area may mean not only the bow/stern points of the ships OBJ1, OBJ3, and OBJ4, but also a certain area near them.

The berthing guide information f1 and f2 between the ship OBJ1 and the quay wall OBJ2 may include two berthing guide information as shown in FIG. 12, but is not limited thereto, and one berthing guide information or three or more It may include berthing guide information. In addition, the berthing guide information f3 and f4 between the ship OBJ1 and the other ships OBJ3 and OBJ4 may include one berthing guide information between the two ships as shown in FIG. 12, but is not limited thereto Instead, two or more berthing guide information may be included between the two ships. The above-described information may be referred to as unloading guide information when used for unloading of a ship.

13 is a diagram for explaining berthing guide information between a ship and a quay according to an embodiment.

The berthing guide information may be sea level height/sea level information. For example, the berthing guide information f5 may include information about the distance/speed between the ship OBJ1 and the quay wall OBJ2 at sea level.

The eyepiece guide information may be information at ground level (or quay wall level). For example, the berthing guide information f6 may include information about the distance/velocity between the ship OBJ1 and the quay wall OBJ2 at the height of the quay wall OBJ2.

The eyepiece guide information may be information at a predetermined height. Here, the predetermined height may be between the height of the sea level and the height of the quay, as shown in the berthing guide information f7 shown in FIG. 13, but is not limited thereto.

The berthing guide information between the ship and the quay may be information f8 between the fender (OBJ6) installed on the ship (OBJ1) and the quay wall (OBJ2). Since the ship (OBJ1) collides with the insect repellent (OBJ6) when berthing or comes into contact with the insect repellent (OBJ6) when anchored, it is important to acquire information (f8) on the distance/speed between the ship (OBJ1) and the insect repellent (OBJ6). can be advantageous In FIG. 13, although the insect repellent OBJ6 is shown as being installed at a position higher than the sea level, at least a portion of the insect repellent OBJ6 may be installed so as to be submerged in the sea OBJ5. In this case, the berthing guide information is at sea level. It may include information on the distance/velocity between the vessel and the insect repellent.

The berthing guide information between the ship and the other ship may include information about the distance/velocity between the ship and the other ship at sea level. The berthing guide information between the ship and the other ship may include information about the distance/velocity between the ship and the other ship at a predetermined height. Here, the predetermined height may be determined in consideration of the shape of the hull. For example, the predetermined height may be the height of a region where a ship protrudes in a direction of another ship. In this case, it may be advantageous to the berthing guide by more accurately grasping the possibility of collision between ships.

According to one embodiment, the device 10 may calculate eyepiece guide information based on lidar data. For example, the device 10 may calculate the distance/velocity to the object using 3D coordinates of lidar points included in the obtained lidar data. Examples of objects may include ships, sea and land, ports, quay walls, buoys, terrain, sky, buildings, people, and animals.

According to one embodiment, eyepiece guide information may be calculated based on images. Here, the image may be an image generated by an image generating unit such as a camera or an image processed through image segmentation therefrom. For example, information on the distance/speed of a ship is calculated based on an image including a ship, sea, and land as objects, or information on the distance/speed of a ship based on a segmentation image generated through segmentation from the image. can be calculated. Examples of objects may include ships, seas, and lands as well as harbors, quay walls, buoys, terrain, sky, buildings, people, and animals.

Hereinafter, an object for estimating eyepiece guide information is referred to as a target object. For example, in the above example, a ship may be a target object. Also, the number of target objects may be plural. For example, when estimating the distance or speed of each of a plurality of ships included in the image, the plurality of ships may be the target object.

According to one embodiment, eyepiece guide information may be calculated based on image pixels. As described above, when eyepiece guide information is calculated based on a point, a point on an image may correspond to a pixel. Thus, eyepiece guide information can be calculated based on the spacing between image pixels.

Information on the distance between points may be calculated based on the spacing between pixels. For example, a predetermined distance may be assigned to each pixel interval and a distance between points may be calculated in proportion to the interval between pixels. As another example, distances between pixels may be calculated based on coordinate values of pixels on an image, and distances between points may be calculated based on the distances between pixels.

Information about the speed between points may be calculated based on changes in information about the distance between points. In this case, movement information may be calculated based on a plurality of images or video frames. For example, information on speed between points may be calculated based on a distance between points in a previous frame, a distance between points in a current frame, and a time interval between frames.

14 is a diagram for explaining a method of acquiring eyepiece guide information according to an embodiment. Obtaining eyepiece guide information is a step of acquiring an image generated by an image generating unit such as a camera (FIG. 14(a)), performing image segmentation on the image to obtain a segmented image (or an image obtained by visualizing the segmented image) Generating (Fig. 14 (b)), finding a point for calculating eyepiece guide information based on the segmentation image (Fig. 14 (c)), and calculating eyepiece guide information corresponding to the point can include Although a method for calculating eyepiece guide information from a segmentation image has been described in FIG. 14, this is only an embodiment, and it is possible to find a point based on an image generated by an image generating unit and calculate eyepiece guide information without generating a segmentation image. There will be.

Obtaining eyepiece guide information may include converting a viewpoint of an image. For example, the obtaining of the eyepiece guide information includes obtaining an image generated by an image generating unit, performing viewpoint transformation on the image, and segmenting the viewpoint-converted image to obtain a segmented image. Eyepiece guide information may be calculated for the segmentation image converted from the viewpoint through the step of generating the segmentation image or performing viewpoint conversion on the segmentation image after the step of generating the segmentation image. Hereinafter, viewpoint conversion will be described.

In general, an image generated by an image generating unit such as a camera may appear as a perspective view. Converting this to a top view (planar viewpoint), a side view, or another perspective viewpoint may be referred to as viewpoint conversion. Of course, a top view or side view image may be converted to another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, the view conversion may not need to be performed.

15 is a diagram related to viewpoint conversion according to an exemplary embodiment. Referring to (a) of FIG. 15 , another near perspective image may be acquired through conversion of a perspective view image. Here, the viewpoint conversion may be performed so that the pier OBJ2 is positioned along the horizontal direction (left-right direction on the image) on the image. Referring to (b) of FIG. 15 , a top view image may be acquired through conversion of a perspective view image of a perspective view image. Here, the top view image may be a view looking down at the sea level from a direction perpendicular to the sea level. Also, as in (a) of FIG. 15 , viewpoint conversion may be performed so that the quay wall OBJ2 is located along the horizontal direction on the image.

Convenience, convenience, and accuracy can be improved when calculating eyepiece guide information through viewpoint conversion. For example, in the case of pixel-based distance calculation, if a top-view image is used, a distance corresponding to an interval between pixels may be the same for the entire image or at least a partial area.

As an example of viewpoint transformation, inverse projective mapping (IPM) may be performed. A 2D image is generated when light reflected from a subject in a 3D space is incident on an image sensor through a lens of a camera, and the relationship between 2D and 3D depends on the image sensor and lens. For example, Equation 1 and can be expressed as

Here, the matrix on the left side denotes 2D image coordinates, the first matrix on the right side denotes intrinsic parameters, the second matrix denotes extrinsic parameters, and the third matrix denotes 3D coordinates. Specifically, fx and fy denote focal lengths, cx and cy denote principal points, and r and t denote rotation and translation parameters, respectively.

A viewpoint can be changed by projecting a 2D image onto an arbitrary 3D plane through back-projection transformation. For example, a perspective view image may be converted into a top view image through back projection transformation, or may be converted into another perspective view image.

Internal parameters may be required for viewpoint conversion. As an example of a method for obtaining internal parameters, the Zhang method may be used. The Zhang method is a kind of polynomial model, which acquires internal parameters by taking pictures of a grid having a known grid size from various angles and distances.

Information on the position and/or attitude of the image generating unit/sensor module that captured the image may be required for viewpoint conversion. This information may be obtained from the position measurement unit and the attitude measurement unit.

Alternatively, information on the position and/or posture may be obtained based on the position of the fixed body included in the image. For example, the image generating unit may generate a first image including a target fixture, which is a fixed object such as a terrain or a building, disposed in a first position and/or a first posture at a first viewpoint. Then, at a second viewpoint, the image generation unit may generate a second image including the target fixture. The position of the target fixture on the first image and the position of the target fixture on the second image are compared to calculate the second position and/or the second posture, which is the position and/or posture of the image generating unit at the second viewpoint. can do.

Acquisition of information on a position and/or posture for viewpoint conversion may be performed at predetermined time intervals. Here, the time interval may depend on the installation location of the image generating unit/sensor module. For example, when the image generating unit/sensor module is installed on a mobile body such as a ship, there may be a need to acquire position and/or attitude information at short time intervals. On the other hand, when the image generating unit/sensor module is installed in a fixed body such as a port, information on the position and/or attitude may be obtained at a relatively long time interval or initially obtained only once. When movement and fixation are repeated, such as in a crane, it may be implemented in a manner in which information on position and/or attitude is obtained only after movement. In addition, the time interval for acquiring information on the location and/or posture may be changed.

The device 10 may convert the viewpoint of the image based on the reference plane. For example, the device 10 may transform the viewpoint of an image using a plane parallel to the sea level and a reference plane where the quay wall is located. Here, the reference plane may depend on the calculated sea level height. Of course, the device 10 is not limited to the above description, and it is also possible to convert the viewpoint of an image to a reference plane other than the sea level or a part of the ship other than the plane where the quay is located (for example, a reference plane at deck height).

The device 10 may convert the viewpoint of the image in consideration of the sea level height. For example, the device 10 may calculate the height of the sea level using data obtained from a lidar sensor or a camera, and convert the viewpoint of the image by considering the calculated height of the sea level.

The above-described viewpoint conversion method is only an example, and viewpoint conversion may be performed in a different way, and the viewpoint conversion information is used for viewpoint conversion, such as the matrix of Equation 1, parameters, coordinates, position and/or posture information, etc. Include the necessary information.

Device 10 may calculate distances to other vessels and/or quays for vessel and/or harbor monitoring. For example, the device 10 may generate feature points of a ship for distance calculation from sensor data, and calculate distances to other ships and/or quay walls based on the generated feature points.

A feature point of a ship is a concept including a specific point in an image, a specific lidar point of lidar data, etc. The feature point is, for example, a point corresponding to one end of the ship, a point corresponding to the bow of the ship, and the stern of the ship. A point corresponding to , a point corresponding to a part where the ship and the sea surface come into contact, and the like may be included.

16 is a flowchart of a method for calculating a distance based on feature points of a vessel according to an embodiment.

Referring to FIG. 16 , a distance calculation method based on ship feature points according to an embodiment includes acquiring sensor data (S1000), generating ship feature points (S2000), and calculating a distance based on the feature points. It may include a step (S2000) of doing.

The device 10 may acquire sensor data (S1000).

According to one embodiment, device 10 may obtain an image from a camera. For example, the device 10 may obtain an image from a camera installed on a berth facing the sea. For example, the device 10 may acquire an image of the sea and, if there is a ship, may also acquire an image of the ship.

According to one embodiment, the device 10 may obtain lidar data from a lidar sensor. For example, the device 10 may obtain lidar data from a lidar sensor installed on a berth facing the sea. For example, the device 10 may acquire lidar data for the sea, and when a ship enters a berth, may also acquire lidar data for the ship.

According to an embodiment, the lidar sensor may obtain lidar data for an area corresponding to an area captured by a camera acquiring an image. For example, a lidar sensor may have a field of view that at least partially overlaps the field of view of a camera acquiring an image.

17 is an example of acquisition of sensor data according to an embodiment. Referring to FIG. 17 , the device 10 according to an embodiment may obtain sensor data for the same area from lidar and cameras installed around a berth. For example, lidar and a camera may be installed on a pier toward or looking at a berth to acquire data about an area where the berth is located. Here, the viewing angle of the lidar may overlap at least partially with the viewing angle of the camera.

Referring to FIG. 17(a), the device 10 may acquire an image of the sea and, if there is a ship, may also acquire an image of the ship. Device 10 may use the acquired camera images to calculate the distance between a ship and another ship and/or between a ship and a quay wall.

Referring to FIG. 17 (b), the device 10 may acquire lidar data for the sea, and may also acquire lidar data for the ship when the ship enters the berth. The lidar data may include a plurality of lidar points captured by lidar sensors. For example, lidar data may include a plurality of lidar points for each vertical or horizontal channel. Device 10 may use the obtained lidar data to calculate the distance between a ship and another vessel and/or between a ship and a quay wall.

Return to FIG. 16 and explain.

The device 10 may generate ship feature points (S2000).

The apparatus 10 may generate feature points of the ship used for calculating the distance between the ship and other objects using the acquired sensor data. For example, the apparatus 10 may generate feature points of the ship by using at least one of sensor data obtained from a camera installed on a berth facing the sea and a lidar sensor. For example, the apparatus 10 may detect a ship area from a camera image, and generate feature points of the ship based on the detected ship area. For example, the apparatus 10 may detect lidar points related to a lidar beam reflected from a ship from lidar data, and generate feature points of the vessel based on the detected lidar points. The device 10 is not limited to the above description, and may generate feature points of the ship in other ways, such as fusing camera images and LIDAR data. Various embodiments of generation of ship feature points will be described later.

The apparatus 10 may calculate a distance between the ship and another object based on the characteristic points of the ship (S3000).

Device 10 may calculate a distance between a vessel and another vessel and/or a distance between a vessel and a quay wall based on the characteristic points of the vessel. For example, the device 10 may calculate the distance between the ship and another ship based on the location of a point corresponding to at least one of the bow and stern of the ship. For another example, the device 10 may calculate the distance between the ship and the quay wall based on the location of a point corresponding to a portion where the ship and the sea level come into contact. For example, distance calculation using a camera image may be calculated based on an interval between image pixels, and distance calculation using lidar data may be calculated based on coordinate values of lidar data. Since it can be applied, details are omitted.

According to an embodiment, the device 10 may calculate the distance between the ship and other objects using feature points of the ship acquired from the camera image. For example, the apparatus 10 may calculate a distance between a ship and another ship or a quay wall using feature points of a ship obtained from a camera image.

According to an embodiment, the device 10 may generate feature points of a ship using an image of which a viewpoint is converted. For example, the device 10 may generate feature points of the ship using a converted image obtained by projecting a young ship in the image onto a reference plane.

18 is a flowchart of a method of generating feature points of a ship using an image in which a viewpoint is converted according to an embodiment. Referring to FIG. 18, a method for generating feature points of a ship using an image in which a viewpoint is converted according to an embodiment includes detecting a ship area from a camera image (S2010), projecting the ship area on a reference plane to generate a converted image, It may include a step (S2020) and a step (S2030) of acquiring feature points of the ship in the converted image.

The device 10 may detect a ship area from the camera image (S2010).

For example, the device 10 may detect a region corresponding to a ship in an image using an artificial neural network. For example, the device 10 may generate a segmentation image from an image using an artificial neural network, and detect an area where a pixel labeled with object information indicating a ship as an object type is located as an area corresponding to the ship. The device 10 is not limited to the above description, and may detect a region corresponding to a ship in an image in another manner, such as determining a region corresponding to a ship through image detection.

The device 10 may generate a converted image by projecting the ship area onto the reference plane (S2020).

For example, the apparatus 10 may generate a transformed image in which an area of the ship is projected onto a reference plane at deck height of the ship. For example, the apparatus 10 generates a segmentation image from the image using an artificial neural network, and based on regions where pixels labeled with object information representing a deck of a ship or a side of the ship as an object type are located, the side and side of the ship The position of the part where the deck is in contact can be obtained, and the area of the ship can be projected with the acquired position of the deck as a reference plane. As another example, the apparatus 10 may acquire the position of the deck of the ship in the image using LIDAR data matched to the camera image, and project the area of the ship to the reference plane using the position of the deck obtained. .

For another example, the device 10 may generate a converted image in which the area of the ship is projected using the sea level as a reference plane. For example, the device 10 generates a segmentation image from the image using an artificial neural network, and the location of a portion where the ship and the sea surface meet based on areas where pixels labeled with object information representing a ship or sea as an object type are located. can be obtained and the area of the ship can be projected with the acquired position of the sea level as a reference plane.

The apparatus 10 is not limited to the description described above, and may generate a converted image in other ways, such as obtaining a position of a deck through image detection or projecting a ship area with a quay wall as a reference plane.

The device 10 may obtain ship feature points from the converted image (S2030).

For example, the apparatus 10 may determine ship feature points for calculating a distance between ships in the converted image. For example, the device 10 obtains a bow and/or a point corresponding to the bow of the vessel in a converted image in which the area of the vessel is projected onto a reference plane at the height of the deck of the vessel, and uses this to determine the distance between the vessels. can be calculated

For another example, the device 10 may determine feature points of the ship for calculating a distance between the ship and the quay wall in the converted image. For example, the apparatus 10 may obtain a bow of the vessel and/or a point corresponding to the bow in a converted image in which the area of the vessel is projected on the sea level as a reference plane, and calculate the distance between the vessel and the quay by using the obtained points. .

The apparatus 10 is not limited to the description described above, and may generate a converted image in other ways, such as obtaining a position of a deck through image detection or projecting a ship area with a quay wall as a reference plane.

19 is an example of a converted image according to an embodiment. In the converted image, distortion may occur due to the projection of the ship area onto the reference plane, and the distortion may become more severe as the distance from the reference plane increases, and the distortion may not occur on the reference plane.

Referring to FIG. 19(a) , the apparatus 10 according to an embodiment generates a converted image in which the area of the ship is projected onto a reference plane 401 at sea level, and uses this to generate a feature point of the ship for distance calculation. can decide

For example, the device 10 may determine an arbitrary point on a line where the ship and the sea level in the converted image come into contact with each other as a feature point of the ship. As an example, the device 10 may calculate the distance between the bow of the ship and the quay or another vessel using an arbitrary point 403 on the bow side of the line where the ship and the sea surface meet in the converted image. As another example, the device 10 may calculate the distance between the stern of the ship and the quay or another vessel using an arbitrary point 402 on the stern side of the line where the ship and sea level meet in the converted image.

For another example, the device 10 may determine points located at the ends of the bow and stern of the ship in the converted image as feature points of the ship. As an example, the device 10 may calculate the distance between the ship and the quay or another vessel using the point 405 of the bow end of the vessel in the converted image. As another example, the device 10 may use the point 406 of the aft end of the vessel in the converted image to calculate the distance between the vessel and the quay or another vessel.

The device 10 is not limited to the foregoing description, and calculates the distance between a ship and another ship or quay wall using a feature point of the ship extracted by polygonizing the area of the ship in the converted image. It is free to calculate

Referring to FIG. 19 (b), the device 10 according to an embodiment generates a converted image in which the area of the ship is projected onto a reference plane 411 at the height of the deck of the ship, and uses the transformed image for calculating the ship distance. It is possible to determine the characteristic points of

For example, the apparatus 10 may determine an arbitrary point of a line where the side of the ship and the deck area of the ship in the converted image come into contact with each other as a feature point of the ship. For example, the device 10 may calculate the distance between the bow of the ship and the quay or another vessel using an arbitrary point 413 on the bow side of a line where the side of the ship and the deck area of the ship in the transformed image are tangent. there is. In another example, the device 10 may calculate the distance between the stern of the ship and the quay or another vessel using an arbitrary point 414 on the stern side of the line where the side of the ship and the deck area of the ship in the transformed image are tangent. can

For another example, the device 10 may determine points located at the ends of the bow and stern of the ship in the converted image as feature points of the ship. For example, the device 10 may calculate the distance between the ship and the quay or another ship using the point 415 of the bow end of the ship in the converted image. As another example, the device 10 may use the point 416 at the stern end of the ship in the converted image to calculate the distance between the ship and the quay or another ship.

The device 10 is not limited to the foregoing description, and calculates the distance between a ship and another ship or quay wall using a feature point of the ship extracted by polygonizing the area of the ship in the converted image. It is free to calculate

20 is a diagram for explaining distance calculation between ships according to an embodiment. Referring to FIG. 20 , the device 10 may calculate the distance between ships by projecting the area of each ship in the image onto different reference planes.

The device 10 may detect a first vessel area 421 and a second vessel area 422 from the camera image. For example, the device 10 may detect the first vessel area 421 and the second vessel area 422 using an artificial neural network. Since the above description may be applied to this, a detailed description thereof will be omitted.

The apparatus 10 may generate a converted image by projecting the detected first ship area 421 and the second ship area 422 onto each reference plane at the deck height of each ship. For example, the device 10 may generate a first transformed image 423 by projecting the detected first ship area 421 onto a first reference plane 425 at a deck level of the first ship, The apparatus 10 may generate a second transformed image 424 by projecting the detected second vessel area 422 onto a second reference plane 426 at a deck height of the second vessel. Since the above description may be applied to this, a detailed description thereof will be omitted.

The device 10 may obtain feature points of each ship using each of the converted images.

The device 10 may determine a feature point of a position corresponding to the bow and/or stern of the first ship in order to calculate the distance between the first ship and the second ship in the first converted image 423 . For example, the device 10 may acquire one of points at both ends of the first ship area in the first transformed image 423 as a feature point. Here, among the points at both ends of the first ship area, a point 427 located closer to the second ship may be determined as a feature point of the ship.

The apparatus 10 may determine a point at a location corresponding to the bow and/or stern of the second ship as a feature point in order to calculate the distance between the second ship and the first ship in the second converted image 424 . For example, the device 10 may acquire one of points at both ends of the second ship area in the second converted image 424 as a feature point. Here, among points at both ends of the second ship area, a point 428 located closer to the first ship may be determined as a feature point of the ship. Since the above description may be applied to this, a detailed description thereof will be omitted.

The device 10 may calculate the distance between the first ship and the second ship using feature points of the first ship and the second ship obtained from each of the converted images 423 and 424 . For example, the device 10 may calculate the distance between the first and second ships based on the number of pixels between feature points 427 and 428 of the first and second ships. For another example, the device 10 may calculate a distance between the first and second vessels based on lidar data matched to feature points 427 and 428 of the first and second vessels. Since the above description may be applied to this, a detailed description thereof will be omitted.

According to an embodiment, the device 10 may calculate the distance between the ship and other objects using feature points of the ship acquired from lidar data. For example, the apparatus 10 may calculate a distance between a ship and another ship or a quay by using feature points of the ship obtained from LIDAR data. For example, the apparatus 10 may calculate a distance between a ship and another ship using lidar points associated with lidar beams reflected from both ends of the ship. As another example, the apparatus 10 may calculate the distance between the ship and the quay wall using a lidar point related to a lidar beam reflected from a portion where the vessel and the sea surface meet.

However, if the lidar sensor has low performance or there is a restriction in a place where the lidar sensor is installed, insufficient lidar data may be obtained to monitor a ship or port. Accordingly, it may be necessary to supplement lidar data that is insufficient for monitoring ships or ports.

To this end, according to an embodiment, the device 10 may estimate new lidar data from the acquired lidar data. For example, the apparatus 10 may newly estimate a lidar point related to a ship using the acquired lidar data. For example, the apparatus 10 may estimate a lidar point associated with a vessel by interpolating or extrapolating lidar points associated with a lidar beam reflected from the vessel.

21 is a flowchart of a lidar point estimation method according to an embodiment. Referring to FIG. 21 , the lidar point estimation method according to an embodiment includes matching a camera image and lidar data (S1010) and estimating a lidar point using the matched camera image and lidar data (S1020). ) may be included.

The device 10 may match the camera image and lidar data (S1010).

According to an embodiment, the device 10 may match camera images and LIDAR data using information for matching. For example, the device 10 may match the camera image and lidar data by matching the coordinate system of the camera image with the coordinate system of lidar data. That is, the coordinate systems of the camera image and LIDAR data may be mutually convertible.

The device 10 may match the camera image and lidar data by considering the camera installation position, the camera installation angle, the lidar sensor installation position, the lidar sensor installation angle, and the like. Here, the device 10 may re-synthesize the camera image and LIDAR data by reflecting the calculated sea level. For example, when matching lidar data and camera images, the device 10 may re-match the camera image and lidar data by using the calculated sea level height as a variable of data conversion.

22 is an example of matching camera images and lidar data according to an embodiment. Referring to FIG. 22 (a), it can be seen that the lidar data obtained from the lidar sensor scanning the same area as the image obtained from the camera installed on the berth is matched.

According to one embodiment, device 10 may match the segmented image with lidar data. For example, the device 10 may generate a segmentation image from an image obtained from a camera using an artificial neural network and match the generated segmentation image with LIDAR data. Referring to FIG. 22 (b), it can be seen that the lidar data obtained from the lidar sensor scanning the same area as the image obtained from the camera installed on the berth is matched.

Matching of camera images and lidar data is not limited to the above description, and may be implemented in other ways, such as matching images detected using an artificial neural network and lidar data.

Returning to FIG. 21 again, description will be made.

The apparatus 10 may estimate a lidar point using the matched camera image and lidar data (S1020).

According to an embodiment, the apparatus 10 may generate an estimated lidar point using an image matched with lidar data among a plurality of lidar points of lidar data. For example, the apparatus 10 may newly generate an estimated lidar point related to the sea by using a lidar point matched to an area corresponding to the sea in the image. For another example, the apparatus 10 may newly generate an estimated lidar point related to the ship by using a lidar point matched to a region of the ship in the image.

23 is a flowchart of a method of generating feature points of a vessel considering estimated lidar points according to an embodiment. Referring to FIG. 23 , a method for generating feature points of a ship considering an estimated lidar point according to an embodiment includes detecting a ship area from a camera image (S2110), and a lidar beam reflected from the ship based on the detected ship area (S2110). Selecting lidar points related to (S2120), estimating lidar points related to the ship using the selected lidar points (S2130), and generating feature points of the vessel in consideration of the estimated lidar points (S2130) S2140) may be included.

The device 10 may detect a ship area from the camera image (S2110).

Camera images and LIDAR data may correspond to each other and may be matched. The above description may be applied to this, and a detailed description thereof will be omitted.

The apparatus 10 may select lidar points related to the lidar beam reflected from the vessel based on the detected vessel area (S2120).

The device 10 may select lidar points related to the lidar beam reflected from the vessel using an area corresponding to the vessel in the detected image. For example, the apparatus 10 may select lidar points associated with a lidar beam reflected from a vessel in consideration of pixel positions of pixels included in an area corresponding to a vessel in the image.

24 is an example of lidar points related to a lidar beam reflected from a ship according to an embodiment. Referring to FIG. 24(a), the apparatus 10 may select lidar points 431 related to the lidar beam reflected from the ship among lidar points matched with the image. For example, the apparatus 10 sets lidar points matched to pixels included in a region corresponding to a ship in an image among a plurality of lidar points, and sets lidar points 431 associated with a lidar beam reflected from a ship. ) can be selected.

Referring to FIG. 24(b) , the apparatus 10 may select lidar points 432 related to lidar beams reflected from a ship among lidar points matched with the segmentation image.

Also, as will be described later, the apparatus 10 may select lidar points related to lidar beams reflected from the vessel among lidar points matched with a converted image obtained by projecting a vessel area on a reference plane.

24, the selection of lidar points related to the lidar beam reflected from the ship, such as using a detected image, is not limited to the above description and may be implemented in other ways. According to one embodiment, device 10 may select lidar points associated with a lidar beam reflected from a ship using only lidar data. For example, the apparatus 10 may select lidar points related to a lidar beam reflected from a ship in consideration of the distribution, number, and the like of lidar data.

Returning to FIG. 23 again, description will be made.

The apparatus 10 may estimate a lidar point related to the ship using the selected lidar points (S2130).

For example, the device 10 may estimate a new lidar point by interpolating or extrapolating lidar points associated with lidar beams reflected from the ship. For example, the apparatus 10 may estimate a new lidar point by interpolating or extrapolating each lidar point in consideration of a relative position using coordinates of lidar points related to a lidar beam reflected from a ship.

According to an embodiment, the apparatus 10 may estimate a new lidar point using a converted image obtained by projecting a ship area onto a reference plane. For example, the device 10 may estimate a new lidar point using a converted image obtained by projecting a ship area on the sea level as a reference plane. For another example, the apparatus 10 may estimate a new lidar point using a converted image obtained by projecting an area at the height of the deck of the vessel to a reference plane.

25 and 26 are diagrams for explaining generation of estimated lidar points using lidar points matched to a ship area projected on a reference plane according to an embodiment.

Referring to FIG. 25 , a transformed image obtained by projecting a ship area on the sea level 441 as a reference plane and a LIDAR point may be matched with each other. Since the lidar points 442 located in the area where the ship and the sea surface meet each other have the same height value, the lidar points 442 located in the same area are interpolated and/or extrapolated to estimate the lidar points 443 , 444, 445) can be generated. For example, the apparatus 10 may generate an estimated lidar point 443 by interpolating lidar points 442 located in an area 441 where the ship and the sea level come into contact. For another example, the apparatus 10 may generate estimated lidar points 444 and 445 by extrapolating lidar points 442 located in an area 441 where the ship and the sea surface come into contact. Here, the device 10 may estimate lidar points by considering the shape of the ship in the image.

Referring to FIG. 26 , a transformed image obtained by projecting an area 451 at the height of a ship's deck to a reference plane and a lidar point may be matched with each other. Since the lidar points 452 located in the deck height region of the ship have the same height value, the estimated lidar points 453, 454, 455) can be created. For example, the apparatus 10 may generate an estimated lidar point 453 by interpolating lidar points located in the deck height region 451 of the ship. For another example, the apparatus 10 may generate estimated lidar points 454 and 455 by extrapolating lidar points located in the deck height region 451 of the ship. Here, the device 10 may estimate lidar points by considering the shape of the ship in the image.

Of course, the apparatus 10 is not limited to the above description, and may generate estimated lidar points in other ways, such as estimating lidar points located in areas that do not have the same height value.

Returning to FIG. 23 again, description will be made.

The device 10 may generate feature points of the ship in consideration of the estimated lidar points (S2140).

According to an embodiment, the apparatus 10 may select a characteristic point of a ship for calculating a distance between ships from among existing lidar points related to a lidar beam reflected from the ship and generated estimated lidar points. there is. That is, the device 10 is an estimated lidar point generated by extrapolating and/or interpolating the existing lidar points matched with a transformed image in which the area of the ship is projected onto a reference plane at the deck height of the vessel, and the existing lidar points. Among the points, a player of the ship and/or a lidar point corresponding to the player may be obtained, and the distance between the ships may be calculated using this.

According to an embodiment, the device 10 determines a feature point of the ship for calculating the distance between the ship and the quay among the existing lidar points and the generated estimated lidar points related to the lidar beam reflected from the ship. can That is, the apparatus 10 selects a ship among existing lidar points matched with a transformed image obtained by projecting the area of the ship onto the sea level as a reference plane and estimated lidar points generated by extrapolating and/or interpolating the existing lidar points. Points corresponding to the player and/or the player may be obtained, and the distance between the ship and the quay may be calculated using this.

For example, the device 10 may determine an arbitrary LIDAR point matched to a line where the ship and the sea level touch in the converted image as a feature point of the ship. As an example, the device 10 may calculate the distance between the bow of the vessel and the quay or another vessel using an arbitrary lidar point matched to the bow side of the line where the vessel and the sea surface meet in the converted image. As another example, the device 10 may calculate the distance between the stern of the ship and the quay or another vessel using an arbitrary lidar point matched to the stern side of the line where the ship and sea level meet in the converted image. Here, the lidar points may include existing lidar points and estimated lidar points.

For another example, the device 10 may determine lidar points matched to the ends of the bow and stern of the ship in the converted image as the characteristic points of the ship. For example, the device 10 may calculate the distance between the ship and the quay or another ship using lidar points matched to the tip of the bow of the ship in the converted image. As another example, the device 10 may calculate the distance between the ship and the quay or another ship using lidar points matched to the stern end of the ship in the converted image. Here, the lidar points may include existing lidar points and estimated lidar points.

Apparatus 10 is not limited to the above description, but polygonizes the area of the ship in the converted image, and calculates the distance between the ship and another ship or quay wall using lidar points matched thereto in other ways. It's okay to calculate the distance.

The device 10 may obtain the height (deck height) of the ship using lidar data, and use the acquired deck height of the ship to generate a feature point of the ship. However, there may be cases where the deck height of the ship cannot be accurately obtained due to lack of LIDAR data, and in this case, the device 10 cannot determine which plane to use as the reference plane for the camera image, and the wrong plane is the reference plane. In this case, severe distortion may occur in the converted image. Accordingly, when the deck height of the ship is not accurately obtained, the apparatus 10 according to an embodiment may acquire feature points of the ship using converted images obtained by projecting the region of the ship onto an arbitrary reference plane.

According to an embodiment, the device 10 may acquire feature points of the ship using transformed images obtained by projecting the ship's area onto an arbitrary reference plane. For example, the apparatus 10 may align converted images projected on arbitrary reference planes having different areas of the ship in the image using LiDAR data, and acquire feature points of the ship using the LIDAR data. Here, the device 10 may use a segmented image.

27 is a flowchart of a method for acquiring feature points of a ship using images in which a ship area is projected onto a plurality of reference planes according to an embodiment. Referring to FIG. 27 , a method of acquiring feature points of a ship using images in which a ship area is projected onto a plurality of reference planes according to an embodiment includes detecting a ship area from a camera image (S2210), and selecting a ship area different from each other. Projecting on a reference plane (S2220), aligning images projected on different arbitrary reference planes (S2230), and generating feature points of a ship using the aligned images and lidar points (S2240). ) may be included.

The device 10 may detect a ship area from the camera image (S2210).

For example, the device 10 may generate a segmented image by segmenting a camera image using an artificial neural network. Specifically, the device 10 may generate a segmentation image for detecting the deck area and the side area of the vessel. For example, the device 10 may generate a segmented image using panoptic segmentation.

The device 10 may project the vessel area onto different arbitrary reference planes (S2220).

For example, the apparatus 10 may project segmentation images for detecting a deck area and a side area of a ship to different reference planes using an artificial neural network. That is, the device 10 may generate at least one of the first image and the second image by projecting the segmentation image onto first and second reference planes that are different from each other.

The device 10 may align images projected on different arbitrary reference planes (S2230).

According to one embodiment, the device 10 may align images projected on different arbitrary reference planes using LIDAR data. For example, the device 10 may align at least one of the projected images to any different reference plane based on the position of the lidar points relative to the lidar beam reflected from the ship. Here, the projected images may be images in which a segmentation image for detecting a deck area and a side area of a ship using an artificial neural network is projected.

28 and 29 are diagrams for explaining alignment of images projected on different arbitrary reference planes according to an exemplary embodiment.

Referring to FIG. 28 , the apparatus 10 may align lidar points with images projected on different reference planes while changing reference planes. As the reference plane changes, the relative position of the lidar points matched with the projected images may also change. The apparatus 10 may align the images and lidar points so that the lidar points are located in a specific region of images projected on arbitrary reference planes different from each other.

Referring to FIG. 28(a) , a first image obtained by projecting a segmentation image for detecting a deck area 461 and a side area 462 of a ship onto an arbitrary first reference plane can be seen. The device 10 may align the lidar points and the first image so that lidar points related to the lidar beam reflected from the ship are located in a specific area in the first image. For example, when the device 10 aligns the lidar points associated with the lidar beam reflected from the ship and the first image, the lidar points associated with the lidar beam reflected from the ship in the first image are segmented image It is possible to align the lidar points and the first image to be located in the projected area corresponding to the side area 462 of the ship of . For example, the apparatus 10 may align the lidar points and the first image so that some of the lidar points related to the lidar beam reflected from the ship are located in an area where the ship and the sea surface meet in the first image. .

Referring to FIG. 28(b), a second image projected on a second reference plane different from the first reference plane may be viewed by a segmentation image for detecting the deck area 463 and the side area 464 of the ship. there is. The apparatus 10 may align the lidar points and the second image so that lidar points related to the lidar beam reflected from the ship are located in a specific area in the second image. For example, when the device 10 aligns the lidar points related to the lidar beam reflected from the ship and the second image, the lidar points related to the lidar beam reflected from the ship in the second image are the segmented image It is possible to align the lidar points and the second image to be located in the projected area corresponding to the deck area 463 of the vessel of the ship. For example, the device 10 may align the lidar points and the second image such that some of the lidar points related to the lidar beam reflected from the ship are located in an area where the deck of the ship starts in the second image. there is.

Referring to FIG. 29 , it can be seen that images projected on different arbitrary reference planes aligned based on the positions of lidar points are fused. Device 10 may align the lidar points with the first image and the second image based on the relative positions of the lidar points in the first image and the second image.

For example, the apparatus 10 may determine that lidar points associated with a lidar beam reflected from a ship in the first image are located below a projected area corresponding to the side area 462 of the ship in the segmentation image, and the second image The lidar points and the first image and the second image may be aligned so that the lidar points related to the lidar beam reflected from the ship are located in the projected area corresponding to the deck area 463 of the ship in the segmentation image. .

Return to FIG. 27 and explain.

The device 10 may generate ship feature points using the aligned images and lidar points (S2240).

For example, the device 10 uses the aligned lidar points and the projected first and second images to determine the characteristics of the vessel used to calculate the distance to another object (eg, another vessel, quay wall, etc.) points can be earned.

According to an embodiment, the apparatus 10 may obtain feature points of a ship by considering a lidar point estimated using the aligned lidar points and the projected first and second images.

For example, the apparatus 10 generates estimated lidar points by extrapolating and/or interpolating lidar points aligned with the projected first and second images, and generating estimated lidar points and existing lidar points. The ship's feature points can be obtained using Referring to FIG. 29 , the apparatus 10 may generate an estimated lidar point 465 by interpolating lidar points, and may generate estimated lidar points 466 and 467 by extrapolating lidar points. there is. Since the above description can be applied to the estimation of the lidar point or the acquisition of the feature point of the ship, the details will be omitted.

Device 10 may obtain the deck height of the vessel in a variety of ways.

In one embodiment, device 10 may use the sensor data to obtain the deck height of the vessel. For example, the device 10 may obtain the deck height of the ship using at least one of image and LIDAR data. For example, the device 10 may obtain the deck height of the ship by using coordinates of the lidar point corresponding to a part where the distance value of the lidar points related to the lidar beam reflected from the side of the ship rapidly changes. As another example, the apparatus 10 detects the side and deck areas of the ship using an artificial neural network capable of detecting the side and deck areas of the ship, and the lidar corresponding to the part where the side area and the deck area of the ship come into contact. The deck height of the ship can be obtained using the coordinates of the points.

In one embodiment, the device 10 may obtain the deck height of the vessel through the communication module 300 . For example, the device 10 may receive AIS information of the ship through the communication module 300 and obtain the deck height of the ship using the received AIS information. For another example, the device 10 may acquire the deck height of the ship by receiving the deck height of the ship inputted from the external device through the communication module 300 .

Of course, the acquisition of the deck height of the ship is not limited to the above description and may be implemented in other ways, such as obtained in other ways.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication department
220: control unit
300: communication module

Claims (1)

Preparing an artificial neural network to be learned using a learning set including a plurality of training images and object information labeled on pixels of the plurality of training images - the object information includes a first index indicating that the type of object is a ship and has a second index indicating that the type of object is a sea, the first index labels pixels corresponding to an area of the ship in the plurality of training images, and the second index is the sea in the plurality of training images Labeled pixels corresponding to the area of -;
acquiring a marine image including a first ship and a second ship whose decks are located at different heights;
detecting a first ship area corresponding to the first ship and a second ship area corresponding to the second ship from the acquired marine image using the artificial neural network;
obtaining a position of the first vessel and a position of the second vessel based on a first height corresponding to a deck height of the first vessel and a second height corresponding to a deck height of the second vessel; and
Calculating a distance between the first vessel and the second vessel based on the position of the first vessel and the position of the second vessel; comprising
Port monitoring methods.

Images