WO2021141338A1 - Dispositif et procédé de surveillance de navire et de port - Google Patents

Dispositif et procédé de surveillance de navire et de port Download PDF

Info

Publication number
WO2021141338A1
WO2021141338A1 PCT/KR2021/000036 KR2021000036W WO2021141338A1 WO 2021141338 A1 WO2021141338 A1 WO 2021141338A1 KR 2021000036 W KR2021000036 W KR 2021000036W WO 2021141338 A1 WO2021141338 A1 WO 2021141338A1
Authority
WO
WIPO (PCT)
Prior art keywords
vessel
image
points
distance
ship
Prior art date
Application number
PCT/KR2021/000036
Other languages
English (en)
Korean (ko)
Inventor
김한근
김동훈
박별터
구정모
Original Assignee
씨드로닉스 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020200003190A external-priority patent/KR102265980B1/ko
Priority claimed from KR1020200139727A external-priority patent/KR102535115B1/ko
Application filed by 씨드로닉스 주식회사 filed Critical 씨드로닉스 주식회사
Publication of WO2021141338A1 publication Critical patent/WO2021141338A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G3/00—Traffic control systems for marine craft
    • G08G3/02—Anti-collision systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the present invention relates to a vessel and port monitoring device and method, and more particularly, to an apparatus and method for performing vessel and port monitoring based on an image.
  • lidar used in a port, it has a problem in that it is difficult to obtain the minimum lidar data necessary for monitoring because it has low performance due to cost problems and the size of the vessel to be monitored is large.
  • One problem to be solved by the present application is to provide an apparatus and method for monitoring a ship and a port for monitoring the vicinity of a ship and a port.
  • Another object to be solved by the present application is to assist the berthing of the vessel based on the image, and to provide a vessel and port monitoring apparatus and method for preventing the collision of the vessel.
  • Another object to be solved by the present application is to provide a vessel and port monitoring apparatus and method using image segmentation.
  • One problem to be solved by the present application is to provide an apparatus and method for monitoring a ship and a port that efficiently fuse data of a camera and a lidar.
  • An object of the present application is to provide a vessel and port monitoring apparatus and method for estimating lidar data related to a vessel using a camera image.
  • a port monitoring method performed by a computing means, acquiring an image of a port including the sea and a vessel using a camera installed in the port to capture an image, the vessel from the port image Extracting a pair of first points corresponding to both ends of the bottom surface in contact with the sea level and a pair of second points corresponding to the bow and stern of the ship, based on the pair of first points Acquiring berthing-related information of the vessel including a bow distance that is a distance between the bow of the vessel and a quay wall and a stern distance that is a distance between the stern and a quay wall of the vessel and the pair of second points and obtaining collision-related information of the vessel including a distance of the vessel to another vessel, wherein the obtaining of the collision-related information includes the pair of second vessels when the other vessel is included in the port image.
  • a port monitoring method may be provided, comprising obtaining a distance between the vessel and the other vessel based on the second one of the points that is closer
  • a method of monitoring a port of a vessel performed by a computing means, using a camera installed in a port to capture an image, acquiring an image of a port including the sea and a vessel, an input image and using an artificial neural network learned using a learning set in which class values indicating the sea, vessel, and feature, respectively, are labeled in pixels corresponding to objects including the sea, ship, and feature included in the input image.
  • the berthing of the ship including a bow distance that is a distance between a bow of the ship and a pier and a stern distance that is a distance between the stern and a quay wall of the ship based on the step of extracting and the pair of first points acquiring relevant information, wherein acquiring the berthing-related information of the vessel includes: acquiring the bow distance of the vessel based on a first point of one of the pair of first points;
  • a port monitoring method may be provided, comprising obtaining the stern distance of the vessel based on a first point of the other of the first points of a pair.
  • a camera installed in a harbor to capture an image a harbor image including the sea and a vessel captured by the camera is obtained, and both ends of the bottom surface of the vessel in contact with the sea level from the harbor image
  • a pair of first points corresponding to and a pair of second points corresponding to the bow and stern of the ship are extracted, and based on the pair of first points, the bow and pier of the ship Acquire berthing-related information of the vessel including a bow distance that is a distance between the vessel and a stern distance that is a distance between the stern and a quay wall of the vessel, and based on the pair of second points, the distance between the vessel and another vessel Acquire the collision related information of the vessel including the distance -
  • the acquisition of the collision related information is the second one of the pair of second points closer to the other vessel when the other vessel is included in the port image
  • a port monitoring device comprising: a control module comprising obtaining a distance between the vessel and the other vessel based on
  • an artificial neural network trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images.
  • preparing - the object information has a first index indicating that the type of object is a vessel and a second index indicating that the type of object is a sea, wherein the first index corresponds to an area of the vessel in the plurality of training images and the second index is labeled to pixels corresponding to the region of the sea in the plurality of training images, obtaining an image captured by a camera, the field of view of the camera and at least partially Acquiring lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle overlapping by , detecting a target vessel area corresponding to the target vessel from the acquired image using the artificial neural network Step, generating a transformed image by projecting the target vessel region onto a specific reference plane, taking into consideration pixel positions of pixels included in the projected target
  • a port monitoring method performed by a computing means, acquiring an image captured by a camera, obtained by a lidar sensor having a viewing angle that at least partially overlaps with a viewing angle of the camera obtaining lidar data comprising a plurality of lidar points, generating a segmentation image from the image, wherein the segmentation image includes a vessel region corresponding to a vessel, the vessel region corresponding to a side of the vessel at least one of a first transformed image or a second transformed image by projecting the vessel region onto an arbitrary first reference plane and a second reference plane; generating - the first transformed image and the second LIDAR points related to the LIDAR beam reflected from the vessel among the plurality of LIDAR points, the heights of the first reference plane and the second reference plane being different from each other.
  • a port monitoring method including calculating a distance between a target vessel and another object may be provided.
  • a port monitoring method performed by a computing means, artificial learning using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images preparing a neural network - the object information has a first index indicating that the type of object is a vessel and a second index indicating that the type of object is a sea, wherein the first index is located in a region of the vessel in the plurality of training images.
  • the second index being labeled with pixels corresponding to the region of the sea in the plurality of training images - a first vessel and a second vessel, the decks of which are located at different heights from each other obtaining a sea image including; detecting a first vessel region corresponding to the first vessel and a second vessel region corresponding to the second vessel from the acquired sea image using the artificial neural network; obtaining the position of the first vessel and the position of the second vessel based on a first height corresponding to the deck height of the first vessel and a second height corresponding to the deck height of the second vessel, and the first A port monitoring method comprising a; calculating a distance between the first vessel and the second vessel based on the position of the vessel and the position of the second vessel may be provided.
  • ship and port monitoring can be performed by accurately calculating information about a ship using image segmentation.
  • FIG. 1 is a diagram of image-based monitoring according to an embodiment.
  • FIG. 2 is a view of a port monitoring apparatus according to an embodiment.
  • 3 and 4 are diagrams related to an embodiment of a port monitoring apparatus according to an embodiment.
  • FIG. 5 is a diagram illustrating a viewing angle and a depth of field according to an exemplary embodiment.
  • FIG. 6 and 7 are views of an installation position of a sensor module according to an embodiment.
  • FIG. 8 is a diagram related to image analysis according to an exemplary embodiment.
  • 9 to 11 are diagrams of an object recognition step according to an exemplary embodiment.
  • 12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
  • FIGS. 14 and 15 are diagrams for estimating position/movement information of an object according to an embodiment.
  • 16 is a flowchart of port monitoring according to an embodiment.
  • 17 is a flowchart related to berthing monitoring and collision monitoring according to an embodiment.
  • FIG. 18 is a diagram related to an example of point extraction for obtaining eyepiece-related information according to an embodiment.
  • FIG. 19 is a diagram related to another example of point extraction for obtaining eyepiece-related information according to an embodiment.
  • 20 is a diagram illustrating an example of point extraction for obtaining collision-related information according to an embodiment.
  • 21 is a diagram illustrating another example of point extraction for obtaining collision-related information according to an embodiment.
  • 22 is a diagram related to an example of extraction of points for port monitoring according to an embodiment.
  • FIG. 23 is a diagram related to an example of obtaining eyepiece-related information according to an embodiment.
  • 24 is a diagram illustrating an example of obtaining collision-related information according to an embodiment.
  • 25 is a diagram illustrating another example of obtaining collision-related information according to an embodiment.
  • 26 and 27 are views related to the distinction between the bow / stern of the ship according to an embodiment.
  • 28 and 29 are diagrams for view transformation according to an embodiment.
  • FIG. 30 is a diagram of image-based monitoring based on a plurality of images according to an embodiment.
  • 31 is a diagram of image fusion according to an embodiment.
  • FIG. 32 is a diagram for correcting position/movement information of an RGB image according to an exemplary embodiment.
  • 33 is a flowchart for correcting position/movement information based on a lidar image according to an exemplary embodiment.
  • 34 is a flowchart of a method for calculating a distance based on a feature point of a ship according to an exemplary embodiment.
  • 35 is an example of acquisition of sensor data according to an embodiment.
  • 36 is a flowchart of a method for generating feature points of a ship using an image in which a viewpoint is converted according to an embodiment.
  • 38 is a diagram for describing calculation of a distance between ships according to an exemplary embodiment.
  • 39 is a flowchart of a method for estimating a lidar point according to an embodiment.
  • 40 is an example of matching a camera image and lidar data according to an embodiment.
  • 41 is a flowchart of a method for generating feature points of a ship in consideration of an estimated lidar point according to an embodiment.
  • lidar points associated with a lidar beam reflected from a vessel according to one embodiment.
  • 43 and 44 are diagrams for explaining generation of an estimated lidar point using a lidar point matched to a vessel area projected on a reference plane according to an embodiment.
  • 45 is a flowchart of a method of acquiring feature points of a ship using images in which a ship area is projected on a plurality of reference planes according to an embodiment.
  • 46 and 47 are diagrams for explaining the alignment of images projected on mutually different arbitrary reference planes according to an embodiment.
  • 'port image' may be understood as an image related to a port, for example, a port image includes an image captured by a camera installed in a port, an image including at least a part of the port, and the like. can do.
  • a method of monitoring a port performed by a computing means, comprising: acquiring an image of a port including the sea and a ship using a camera installed in a port to capture an image; extracting a pair of first points corresponding to both ends of the bottom surface of the ship in contact with the sea level and a pair of second points corresponding to the bow and stern of the ship from the harbor image; Based on the pair of first points, obtaining berthing-related information of the vessel including a bow distance that is a distance between the bow of the vessel and a pier and a stern distance that is a distance between the stern and a quay wall of the vessel to do; and obtaining collision related information of the vessel including a distance of the vessel to another vessel based on the pair of second points; Including, wherein the obtaining of the collision-related information comprises: when another vessel is included in the port image, the vessel based on the one second point that is closer to the other vessel among the pair of second points
  • a port monitoring method comprising
  • the extracting includes: obtaining an area corresponding to the ship from the harbor image; extracting a line in which the area corresponding to the ship is in contact with the sea level; and both ends of the line in contact with the sea level are a pair of It may include determining the first points.
  • the step of extracting is obtaining a region corresponding to the vessel from the harbor image, and extracting a bow vessel corresponding to the bow side of the vessel and a stern vessel corresponding to the stern side in the region corresponding to the vessel. and determining one end of the bow line and one end of the stern line as a pair of second points.
  • the extracting includes determining a boundary polygon indicating the boundary of the vessel and extracting the pair of first points and the pair of second points based on the determined boundary polygon. can do.
  • the pair of first points may be determined as two points among a plurality of points corresponding to the lower portion of the vessel of the boundary polygon.
  • the pair of first points may be determined as two points closest to each of the two points corresponding to the front end and the stern end of the vessel of the boundary polygon.
  • the step of obtaining the eyepiece-related information includes obtaining the bow distance based on a first point of one of the pair of first points and a first of the other one of the pair of first points. It may include obtaining the stern distance based on the point.
  • the step of obtaining the bow distance includes calculating the bow distance based on the number of pixels of the harbor image between the one first point and the quay wall, and the step of obtaining the stern distance is the It may include calculating the stern distance based on the number of pixels of the harbor image between the other first point and the quay wall.
  • the step of obtaining the berthing-related information is based on the bow distance, obtaining a bow speed, which is the speed at which the bow of the ship approaches the quay wall, and the stern of the ship based on the stern distance to the quay wall. It may include obtaining a stern speed, which is an approaching speed.
  • the step of obtaining the bow speed comprises obtaining the bow speed by comparing a first bow distance of the harbor image and a second bow distance of a subsequent frame image of the harbor image, and obtaining the stern speed
  • the doing may include obtaining the stern speed by comparing the first stern distance of the harbor image and the second stern distance of the subsequent frame image.
  • the step of obtaining the bow speed includes calculating the bow speed based on a time interval between the harbor image and the subsequent frame image and the difference in the number of pixels of the first and second bow distances,
  • Obtaining the stern velocity may include calculating the stern velocity based on a difference between the time interval and the number of pixels of the first and second stern distances.
  • the pair of second points are one of a plurality of bow points corresponding to the bow of the vessel of the boundary polygon and one point of a plurality of stern points corresponding to the stern of the vessel of the boundary polygon. can be decided.
  • the pair of second points may be determined as one point corresponding to the fore end of the bow points and one point corresponding to the aft end among the stern points.
  • the obtaining of the collision-related information may include calculating a distance between the vessel and the other vessel based on the number of pixels in the port image between the one second point and the other vessel. .
  • the acquiring of the collision-related information may include acquiring a relative speed between the vessel and the other vessel based on a distance between the vessel and the other vessel.
  • the step of obtaining the relative speed may include comparing a first distance between the vessel and the other vessel in the harbor image and a second distance between the vessel and the other vessel in a subsequent frame image of the harbor image to determine the relative speed. It may include obtaining the speed.
  • the step of obtaining the relative speed may include calculating the relative speed based on a time interval between the harbor image and the subsequent frame image and a difference between the number of pixels at the first distance and the second distance. have.
  • the step of obtaining the collision-related information may include extracting a third point at a position where the one second point is in contact with the bottom surface where the ship is in contact with the sea level, and between the third point and the other ship.
  • the method may include calculating a distance between the vessel and the other vessel based on the number of image pixels.
  • the pair of first points and the pair of second points may be determined based on a part of the boundary polygon.
  • the step of extracting the pair of first points and the pair of second points includes generating a viewpoint transformed harbor image by converting the viewpoint of the harbor image and the pair of viewpoint transformation harbor images based on the image. It may include extracting the first points of and a pair of second points.
  • the pixels corresponding to the input image and the objects including the sea, the ship, and the features included in the input image are sea, respectively.
  • generating segmentation images for the objects from the harbor image using an artificial neural network trained using a running set that labels class values indicating ships and features, and the pair of second images from the segmentation image It may include extracting 1 points and the pair of second points.
  • the harbor monitoring method includes: acquiring a lidar image corresponding to the harbor image; and correcting the eyepiece-related information and/or the collision-related information based on the acquired lidar image.
  • the harbor image may include a panoramic image in which a plurality of harbor images are registered.
  • the port monitoring method comprises: outputting the berthing-related information and/or the collision-related information together with the port image; may further include.
  • the port monitoring method comprises: determining the bow and the stern of the vessel based on the harbor image; may further include.
  • a port monitoring method of a vessel performed by a computing means, comprising: acquiring an image of a port including the sea and a vessel using a camera installed in the port to capture an image; An artificial neural network learned using a learning set in which the input image and the pixels corresponding to the objects including the sea, the ship, and the feature included in the input image are labeled with class values indicating the sea, the ship, and the feature, respectively.
  • berthing-related information of the vessel including a bow distance, which is a distance between the bow of the vessel and a pier, and a stern distance, which is a distance between the stern and a quay wall, of the vessel obtaining; and acquiring the berthing-related information of the vessel includes: acquiring the bow distance of the vessel based on a first point of one of the pair of first points; and the pair of first points
  • a port monitoring method comprising the step of obtaining the stern distance of the vessel based on the first point of the other.
  • the port monitoring method comprises the steps of extracting a pair of second points corresponding to the bow and stern of the ship; and obtaining collision-related information of the vessel including a distance between the vessel and another vessel based on the pair of second points;
  • the method further includes, wherein the obtaining of the collision-related information comprises: when another vessel is included in the port image, the vessel based on the one second point that is closer to the other vessel among the pair of second points and obtaining a distance between the other vessel.
  • a recording medium in which a program for executing the above-described method is recorded may be provided.
  • a camera is installed in the harbor to take an image;
  • a port image including the sea and the vessel captured by the camera is acquired, and from the port image, the vessel corresponds to a pair of first points corresponding to both ends of the bottom surface in contact with the sea level and the bow and stern of the vessel extracting a pair of second points, and based on the pair of first points, a bow distance that is a distance between the bow of the ship and a pier and a stern distance that is a distance between the stern and a quay wall of the ship
  • Acquire berthing-related information of the vessel including, and obtain collision-related information of the vessel including the distance between the vessel and another vessel based on the pair of second points -
  • Acquisition of the collision-related information includes obtaining a distance between the vessel and the other vessel based on the one second point that is closer to the other vessel among the pair of second points when another vessel is included in the port image - which control module; and a communication module for transmitting the berthing
  • an artificial neural network trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images preparing - the object information has a first index indicating that the type of object is a ship and a second index indicating that the type of object is a sea, wherein the first index corresponds to an area of the vessel in the plurality of training images labeled pixels, the second index being labeled to pixels corresponding to the region of the sea in the plurality of training images, obtaining an image captured by a camera, at least in part with an angle of view of the camera and Acquiring lidar data including a plurality of lidar points obtained by lidar sensors having overlapping viewing angles; detecting a target vessel area corresponding to a target vessel from the acquired image using the artificial neural network; , generating a transformed image by projecting the target vessel region onto a specific reference plane, taking into account pixel positions of pixels included in the projected target vessel region
  • the specific reference plane may include a reference plane at sea level.
  • the estimated lidar point may be generated from lidar points located in a line where the target vessel and the sea level meet in the converted image among the selected lidar points.
  • the estimated lidar point may be generated by extrapolating and/or interpolating in consideration of the relative positions of the lidar points located on the line where the target vessel and the sea level are in contact.
  • the specific reference plane may comprise a reference plane at the deck level of the target vessel.
  • the estimated lidar point may be generated from lidar points located on a line in which the side surface of the target vessel and the deck contact the converted image among the selected lidar points.
  • the estimated lidar point may be generated by extrapolating and/or interpolating in consideration of the relative positions of the lidar points located on the line where the side surface of the target vessel and the deck are in contact.
  • the specific reference plane comprises any first and second reference planes
  • the transformed image comprises a first transformed image and a second transformed image
  • the first transformed image comprises the target
  • the vessel region may be generated by projecting the first reference plane
  • the second converted image may be generated by projecting the target vessel region onto the second reference plane.
  • the first transformed image and the second transformed image may be aligned based on positions of the plurality of lidar points.
  • the feature point of the target vessel may include a point corresponding to at least one of a bow and a stern of the target vessel.
  • the feature point of the target vessel may include at least one arbitrary point among LiDAR points located on a line in which the target vessel and the sea level of the converted image are in contact.
  • the feature point of the target vessel may include LiDAR points located at both ends of the area of the target vessel of the converted image.
  • the artificial neural network may further include a third index indicating that the type of the object is the side of the vessel, and a fourth index indicating that the type of the object is the deck of the vessel.
  • a method of monitoring a port performed by a computing means comprising: acquiring an image captured by a camera; a plurality of obtained by a lidar sensor having a viewing angle that at least partially overlaps with a viewing angle of the camera acquiring lidar data including lidar points of , generating a segmentation image from the image, wherein the segmentation image includes a vessel region corresponding to a vessel, wherein the vessel region corresponds to a side of the vessel.
  • first and second reference planes comprising a side area and a deck area corresponding to a deck area of the vessel - projecting the vessel area onto any first and second reference planes to produce at least one of a first transformed image or a second transformed image the first transform image and the second transform of LiDAR points related to the LIDAR beam reflected from the vessel among the plurality of LIDAR points, the heights of the first reference plane and the second reference plane being different from each other.
  • a port monitoring method may be provided that includes calculating a distance between a vessel and another object.
  • an artificial neural network is trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images.
  • preparing - the object information has a first index indicating that the type of object is a vessel and a second index indicating that the type of object is a sea, wherein the first index corresponds to an area of the vessel in the plurality of training images and the second index is labeled with pixels corresponding to the region of the sea in the plurality of training images, comprising a first vessel and a second vessel, the decks of which are located at different heights from each other.
  • obtaining a sea image detecting a first vessel region corresponding to the first vessel and a second vessel region corresponding to the second vessel from the acquired sea image using the artificial neural network; obtaining the position of the first vessel and the position of the second vessel based on a first height corresponding to the deck height of the first vessel and a second height corresponding to the deck height of the second vessel, and the first vessel
  • a port monitoring method comprising a; calculating a distance between the first vessel and the second vessel based on the position of the second vessel and the position of the second vessel may be provided.
  • acquiring the position of the first vessel and the position of the second vessel comprises generating a first transformed image by projecting the region of the first vessel to a reference plane at the first height; obtaining the position of the first vessel by using a first transformed image; projecting the second vessel region onto a reference plane at the second height to generate a second transformed image; and the second transformed image It may include the step of obtaining the position of the second vessel by using.
  • the acquiring of the position of the first vessel includes a first position of at least one of a bow and a stern of the first vessel by using a pixel position of the region of the first vessel projected from the first transformed image. and acquiring the position of the second vessel, wherein the acquiring of the position of at least one of the bow and stern of the second vessel using the pixel position of the region of the second vessel projected from the second transformed image. acquiring the second location.
  • a computer-readable recording medium in which a program for performing the above-described methods is recorded may be provided.
  • sea level height should be broadly interpreted to include not only the absolute sea level height, but also the relative sea level height, the relative sea level height compared with the average sea level height of a specific sea area, and the like.
  • sea level height is the distance between sea level and a quay wall, the distance between sea level and a monitoring device (e.g., an image generating unit), and the length of objects exposed above sea level.
  • monitoring refers to grasping or recognizing the surrounding situation, detecting a detection target such as a certain area or a specific object using various sensors and providing the detection result to the user, as well as through calculation based on the detection result. It should be construed broadly to include, for example, providing additional information.
  • Image-based monitoring may mean identifying or recognizing a surrounding situation based on an image.
  • monitoring may mean acquiring information for calculating berthing guide information during berthing or berthing of a ship or recognizing other ships or obstacles therefrom by acquiring images around the ship when the ship is operating.
  • Berthing guide information refers to information about the environment, such as recognizing other ships or obstacles, understanding port conditions, whether the berth is accessible, distance and speed from the quay wall, distance and speed from other ships, and identification of obstacles in the navigation route. can mean In this specification, although mainly described for monitoring when berthing is performed in ships and ports, the present specification is not limited thereto and may also be applied to the case of driving of a vehicle, operation of an aircraft, and the like.
  • image-based monitoring may include an image acquisition step S10 and an image analysis step S20 .
  • the image acquisition step S10 may mean a step in which the device 10 acquires an image.
  • the type of image may be various, such as an RGB image, an IR image, a depth image, a lidar image, and a radar image, and there is no limitation.
  • an RGB image an RGB image
  • an IR image an IR image
  • a depth image a lidar image
  • a radar image there is no limitation.
  • the image analysis step S20 may mean obtaining an analysis result based on the image.
  • the image analysis step (S20) may include calculating the eyepiece guide information through the image.
  • the image analysis step ( S20 ) may refer to a step of analyzing characteristics of an object included in the image.
  • the image analysis step ( S20 ) may include determining a situation indicated by the image. or,
  • monitoring information information acquired through the image acquisition step ( S10 ) or the image analysis step ( S20 ) is referred to as monitoring information.
  • FIG. 2 is a view of a port monitoring apparatus according to an embodiment.
  • the monitoring device 10 may include a sensor module 100 , a control module 200 , and a communication module 300 .
  • the sensor module 100 may sense information about a ship or a ship's vicinity and a port.
  • the sensor module 100 may include an automatic identification system (AIS), an image generating unit, a position measuring unit, an attitude measuring unit, a casing, and the like.
  • AIS automatic identification system
  • the image generating unit may generate an image.
  • the image generating unit may include a camera, lidar, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.
  • the lidar sensor is a sensor for detecting a distance and a position of an object using a laser.
  • the distance between the lidar sensor and the object and the position of the object with respect to the lidar sensor may be represented by a three-dimensional coordinate system.
  • the distance between the lidar sensor and the object and the position of the object with respect to the lidar sensor may be expressed in a rectangular coordinate system, a spherical coordinate system, a cylindrical coordinate system, or the like.
  • the lidar sensor may have a plurality of channels in a vertical or horizontal direction, and may be, for example, a lidar sensor having 32 or 64 channels.
  • the lidar sensor may use a laser reflected from the object to determine the distance R from the object.
  • the lidar sensor may use a time of flight (TOF), which is a time difference between an emitted laser and a detected laser, to determine the distance to the object.
  • the lidar sensor may include a laser output unit for outputting a laser and a receiving unit for detecting the reflected laser.
  • the lidar sensor determines the time the laser is output from the laser output unit, checks the time the receiver detects the laser reflected from the object, and determines the distance to the object based on the difference between the emitted time and the sensed time can do.
  • the lidar sensor uses other methods such as triangulation based on the sensed position of the detected laser to determine the distance (R) from the object, such as using a phase shift of the detected laser, etc. R) may be determined.
  • the lidar sensor may determine the position of the object by using the angle of the irradiated laser. For example, when the irradiation angle of one laser irradiated from the lidar sensor toward the scan area of the lidar sensor is known, if the laser reflected from the object existing on the scan area is detected by the receiver, the lidar The sensor may determine the position of the object based on the irradiation angle of the irradiated laser.
  • the lidar sensor may have a scan area including the object in order to detect the position of an arbitrary object in the vicinity.
  • the scan area represents a detectable area as one screen, and may mean a set of points, lines, and planes that form one screen during one frame.
  • the scan area may mean the irradiation area of the laser irradiated from the lidar sensor, and the irradiation area may mean a set of points, lines, and surfaces where the laser irradiated during one frame meets a sphere at the same distance (R).
  • a field of view (FOV) means a detectable area, and may be defined as an angular range of a scan area when the lidar sensor is viewed as the origin.
  • the position measuring unit may measure the position of a component included in the sensor module, such as a sensor module or an image generating unit.
  • the location measurement unit may be a Global Positioning System (GPS).
  • GPS Global Positioning System
  • RTK-GPS Real-Time Kinematic GPS
  • the position measuring unit may acquire position information at predetermined time intervals.
  • the time interval may vary depending on the installation position of the sensor module. For example, when the sensor module is installed in a moving object such as a ship, the position measurement unit may acquire position information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the position measurement unit may acquire position information at long time intervals. The time interval for obtaining the location information of the location measuring unit may be changed.
  • the posture measuring unit may measure the posture of components included in the sensor module, such as a sensor module or an image generating unit.
  • the posture measurement unit may be an inertial measurement unit (IMU).
  • the posture measuring unit may acquire posture information at predetermined time intervals.
  • the time interval may vary depending on the installation position of the sensor module. For example, when the sensor module is installed in a moving object such as a ship, the posture measuring unit may acquire posture information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the posture measuring unit may acquire posture information at long time intervals. The time interval for acquiring the posture information of the posture measuring unit may be changed.
  • the casing may protect a sensor module such as an image generating unit, a position measuring unit, and a posture measuring unit.
  • a sensor module such as an image generating unit, a position measuring unit, and a posture measuring unit.
  • At least one of an image generating unit, a position measuring unit, and a posture measuring unit may be present inside the casing.
  • the casing may prevent equipment such as an image generating unit existing therein from being corroded by salt water. Alternatively, the casing may protect it by preventing or mitigating the impact applied to the equipment existing therein.
  • a cavity may be formed in the interior of the casing to contain an image generating unit or the like therein.
  • the casing may have a rectangular parallelepiped shape with an empty interior, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like can be disposed.
  • an opening may be formed in one area of the casing or a transparent material such as glass may be formed in one area of the casing to secure a view of the image generating unit.
  • the image generating unit may image the periphery of the vessel and the harbor through the opening or the transparent area.
  • the casing may be provided with a strong material to protect the image generating unit and the like from external impact.
  • the casing may be provided with a material such as an alloy for seawater in order to prevent corrosion due to salt.
  • the casing may include equipment for removing foreign substances from the image generating unit.
  • foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing.
  • the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove the foreign material.
  • foreign substances may be removed by applying water or washer liquid through a liquid spray included in the casing, or the foreign substances may be physically removed using a wiper after application.
  • the debris removal equipment can be operated manually, but can also be operated automatically.
  • the foreign material removal equipment may operate at a predetermined time interval.
  • the foreign material removal equipment may be operated using a sensor that detects whether a foreign material is attached to the image generating unit.
  • the foreign material removal equipment may be operated when it is determined that the foreign material is present.
  • whether a foreign material is captured in the image may be determined through an artificial neural network.
  • One sensor module 100 may include a plurality of identical equipment, such as including two or more identical cameras.
  • the control module 200 may perform image analysis. In addition, the operation of receiving various data through the sensor module 100, the operation of outputting various outputs through the output device, the operation of storing various data in the memory or acquiring various data from the memory, etc. It can be done by control.
  • various operations or steps disclosed in the embodiments of the present specification may be interpreted as being performed by the control module 200 or being performed by the control of the control module 200 unless otherwise specified.
  • control module 200 examples include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an on-demand system.
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processing unit
  • state machine a state machine
  • on-demand system There may be a semiconductor (Application Specific Integrated Circuit, ASIC), a Radio-Frequency Integrated Circuit (RFIC), and a combination thereof.
  • ASIC Application Specific Integrated Circuit
  • RFIC Radio-Frequency Integrated Circuit
  • the communication module 300 may transmit information from the device 10 to the outside or receive information from the outside.
  • the communication module 300 may perform wired or wireless communication.
  • the communication module 300 may perform bi-directional or unidirectional communication.
  • the device 10 may transmit information to an external output device through the communication module 300 to output a control result performed by the control module 200 through the external output device.
  • the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.
  • VTS Vessel Traffic Service
  • the sensor module 100 , the control module 200 , and the communication module 300 may include a control unit.
  • the control unit may process and operate various types of information within the module, and may control other components constituting the module.
  • the control unit may be provided in the form of an electronic circuit that physically processes an electrical signal.
  • a module may physically include only a single control unit, or alternatively may include a plurality of control units.
  • the control unit may be one or a plurality of processors mounted on one computing means.
  • the control unit may be mounted on a physically separated server and a terminal and provided as processors that cooperate through communication.
  • control unit examples include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an application specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processing unit
  • ASIC application specific Integrated Circuit
  • RFIC Radio-Frequency Integrated Circuit
  • the sensor module 100 , the control module 200 , and the communication module 300 may include a communication unit.
  • the modules may transmit and receive information through a communication unit.
  • the sensor module 100 may transmit information obtained from the outside through the communication unit, and the control module 200 may receive information transmitted by the sensor module 100 through the communication unit.
  • the communication unit may perform wired or wireless communication.
  • the communication unit may perform bi-directional or unidirectional communication.
  • the sensor module 100 , the control module 200 , and the communication module 300 may include a memory.
  • the memory may store various processing programs, parameters for performing the processing of the programs, or processing result data, and the like.
  • the memory may store data required for learning and/or inference, an artificial neural network in progress or learned, and the like.
  • Memory includes non-volatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or any other tangible non-volatile recording medium. etc. can be implemented.
  • the monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 .
  • one device 10 may include two sensor modules 100 , and each sensor module 100 may include two cameras again.
  • 3 and 4 are diagrams related to an embodiment of a monitoring device according to an embodiment.
  • the monitoring device 10 may include a sensor module 100 and a control module 200 .
  • the sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 .
  • the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later.
  • the control module 200 may receive an image from the sensor module 100 through the communication unit 210 , and perform image analysis such as position/movement information estimation and image registration, which will be described later, through the control unit 220 .
  • the control module 200 may transmit an analysis result such as location/movement information and a matched image to the cloud server through the communication unit 210 .
  • the cloud server may transmit the analysis result received from the control module 200 to a user terminal such as a smart phone, a tablet, a PC, or receive an instruction from the user terminal.
  • the monitoring device 10 may include a sensor module 100 .
  • the sensor module 100 may generate an image through the camera 130 and transmit the image to the cloud server through the communication unit 110 .
  • the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later.
  • the cloud server may receive an image from the sensor module 100 and perform image analysis such as position/movement information estimation and image matching, which will be described later.
  • the cloud server may transmit the image analysis result to a user terminal such as a smartphone, tablet, or PC or receive an instruction from the user terminal.
  • the apparatus 10 shown in FIGS. 2 to 4 is merely an example, and the configuration of the apparatus 10 is not limited thereto.
  • the device 10 may include an output module 400 .
  • the output module 400 may output a result of an operation performed by the control module 200 .
  • the output module 400 may output an analysis result.
  • the output module 400 may be, for example, a display, a speaker, a signal output circuit, and the like, but is not limited thereto.
  • information may be output through the output module 400 in addition to transmitting information to an external output device such as a user terminal and the external output device outputting information.
  • the device 10 may not include a sensor module.
  • the control module 200 may receive information from an external sensor device and perform an image-based monitoring operation, such as performing image analysis.
  • the control module 200 may perform image analysis by receiving information from an AIS, a camera, a lidar, a radar, etc. already installed in a ship or a port.
  • each of the components of FIGS. 2 to 4 are not necessarily performed by the corresponding components and may be performed by other components.
  • the control unit 120 of the sensor module 100 performs viewpoint transformation in FIG. 3 above
  • the control unit 220 of the control module 200 or the cloud server may perform viewpoint transformation. .
  • Image acquisition for image-based monitoring may be performed through the sensor module 100 .
  • an image may be acquired through an image generating unit included in the sensor module 100 .
  • an image may be acquired from an external sensor device.
  • Images for vessel and port monitoring will generally include sea, vessel, buoy, obstacle, terrain, port, sky, building, and the like.
  • the analysis and monitoring of an image acquired through a visible light camera will be mainly described, but the present invention is not limited thereto.
  • a field of view and a depth of field may vary according to an image generating unit.
  • 5 is a diagram illustrating a viewing angle and a depth of field according to an exemplary embodiment.
  • a field of view may mean to what extent an image is included in an image horizontally or vertically, and is generally expressed as an angle (degree, degree).
  • a larger viewing angle may mean generating an image including an area having a larger width left and right or generating an image including an area having a larger width up and down.
  • the depth of field may mean a distance range recognized as being in focus of the image, and a deep depth of field may mean that the distance range recognized as being in focus of the image is wide. Referring to FIG.
  • the image may include an area A1 recognized as being in focus and an area A2 other than that.
  • an area included in the image is referred to as an imaging area (A1 + A2) and an area recognized as being in focus is referred to as an effective area (A1).
  • Image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on , or based on a part of the imaging area, an area used to perform image analysis and monitoring is referred to as a monitoring area.
  • An example of a camera with a large field of view and a shallow depth of field is a wide-angle camera.
  • Examples of cameras with a small field of view and a deep depth of field include high magnification cameras and zoom cameras.
  • the sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, a crane, a ship, etc. in a port, and there is no limitation in the number. However, the installation location or number of the sensor module 100 may vary according to characteristics such as the type and performance of the sensor module 100 . For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more from the water surface for efficient monitoring, or a plurality of sensor modules 100 may be installed to have different imaging areas. In addition, the position and posture of the sensor module 100 may be manually or automatically adjusted during or after installation.
  • the viewing angle of the lidar may at least partially overlap with the viewing angle of the image generating unit (eg, a camera).
  • the field of view (FOV) of the lidar may be smaller than the field of view of the camera.
  • lidar data outside the viewing angle of lidar for which lidar data is not acquired may be estimated using an image acquired from a camera.
  • FIG. 6 and 7 are views of an installation position of a sensor module according to an embodiment.
  • the sensor module 100 may be installed at a fixed location, such as a port or on land, or installed on a moving object, such as a ship.
  • a vessel to be monitored hereinafter referred to as a “target vessel”
  • FIG. 7 a vessel to be monitored
  • FIG. 6 berthing or berthing of the target vessel as shown in FIG. 6 .
  • the sensor module 100 may be installed on a drone or the like to monitor a target vessel.
  • Other components of the monitoring device 10 may be installed together with the sensor module 100 or at a separate location.
  • image analysis for image-based monitoring may include acquiring object characteristics.
  • the object may include a ship, port, buoy, sea, terrain, sky, building, person, animal, fire, smoke, and the like.
  • object characteristics may include a type of object, a location of an object, a distance to the object, absolute and relative speed and speed of the object, and the like.
  • Image analysis for image-based monitoring may include recognizing/judging a surrounding situation.
  • the image analysis may be to determine that a fire situation has occurred from an image of a fire in the port, or to determine that an intruder has entered the port from an image captured by a person who entered the port at an unscheduled time.
  • image analysis may include detecting a fire from an image in which smoke is present.
  • Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each of the modules 100 and 200 .
  • image analysis may include an object recognition step S210 and a position/movement information estimation step S220 .
  • the image analysis may include an object recognition step ( S210 ).
  • the object recognition step S210 may include recognizing an object included in an image. For example, object recognition may be determining whether an object such as a ship, tugboat, sea, or harbor is included in the image. Furthermore, object recognition may be determining at which position in an image the object exists.
  • 9 to 11 are diagrams of an object recognition step according to an exemplary embodiment.
  • FIG. 9 is an image captured by the camera, and an object may be recognized as shown in FIG. 10 or 11 through the object recognition step.
  • FIG. 10 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation.
  • the object recognition step may mean a segmentation step.
  • a characteristic corresponding to a pixel on an image may be assigned or calculated from an image. It could be said that the pixel has been assigned a property or labeled.
  • a segmentation image as shown in FIG. 10 may be obtained by performing segmentation based on an image captured by the camera of FIG. 9 .
  • the first pixel area P1 is the area on the image of the pixel corresponding to the ship
  • the second pixel area P2 is the sea
  • the third pixel area P3 is the quay wall of the harbor
  • the fourth pixel area ( P4) is the terrain
  • the fifth pixel area P5 is the area on the image of the pixel corresponding to the sky.
  • FIG. 10 shows that information on the type of object corresponding to each pixel on the image is calculated by performing segmentation
  • information obtainable through segmentation is not limited thereto.
  • characteristics such as position, coordinates, distance, and direction of an object may also be acquired through segmentation.
  • the different characteristics may be expressed independently or may be expressed by reflecting them at the same time.
  • Table 1 is a table related to labeling in which information on the type and distance of an object is simultaneously reflected, according to an embodiment.
  • a class may be set in consideration of information on the type and distance of an object, and an identification value may be assigned to each class.
  • the second identification value may be assigned in consideration of both topography, which is information about the type of object, and short distance, which is information about distance.
  • Table 1 is an example of a case in which type information and distance information are considered together.
  • other information such as direction information, obstacle movement direction, speed, and route mark may also be considered.
  • not all identification values need to include a plurality of pieces of information, nor do they need to include the same type of information.
  • a specific identification value includes only information about the type (for example, identification value 1 does not include information about distance) and another identification value includes information about type and distance, etc. It can be expressed in various ways. For another example, other classes such as tugs, ropes, sides and decks of ships may be added or modified to other classes.
  • FIG. 11 is a diagram showing a position of an object in an image with a bounding box, which is also referred to as detection.
  • the object recognition step may mean a detection step. Compared with segmentation, detection can be viewed as detecting the position of an object in the form of a box, rather than calculating a characteristic for each pixel of an image.
  • a detection image as shown in FIG. 11 may be obtained by performing detection based on an image captured by the camera of FIG. 9 . 11 , it can be seen that a vessel is detected on the image and the position of the vessel is expressed as a rectangular bounding box BB. Although only one object is detected in FIG. 11 , two or more objects may be detected from one image.
  • Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks, and the final result may be calculated by combining the results.
  • An artificial neural network is a kind of algorithm modeled after the structure of a human neural network, and may include one or more nodes or one or more layers including neurons, and each node may be connected through a synapse. Data input to the artificial neural network (input data) may be output (output data) through a node through a synapse, and information may be obtained through this.
  • Types of artificial neural networks include a convolutional neural network (CNN) that extracts features using a filter, and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back as an input.
  • CNN convolutional neural network
  • RNN recurrent neural network
  • Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), relational network (RN), etc. can be applied and have limitations. it is not
  • the step of learning the artificial neural network will be expressed as a learning step
  • the step of using the artificial neural network will be expressed as an inference step.
  • the artificial neural network may be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.
  • 12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
  • the 12 is an embodiment of the learning step of the artificial neural network, in which an untrained artificial neural network receives learning data or training data and outputs output data, and compares the output data with the labeling data.
  • the artificial neural network can be trained through the backpropagation of the error.
  • the training data, output data, and labeling data may be images.
  • the labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or a program.
  • a learned artificial neural network may receive input data and output output data. According to the information of the learning data in the learning stage, information that can be inferred in the inference stage may vary. Also, the accuracy of the output data may vary according to the learning degree of the artificial neural network.
  • the recognition of the object is not limited to the above description and may be implemented in other ways.
  • the identification value is used for object recognition for convenience of description, the identification value is only used as one type of index.
  • a vector type index may be used for object recognition, and the training data, output data, and labeling data of the artificial neural network may be vector type data.
  • the image analysis may include a position/movement information estimation step ( S220 ).
  • the location/movement information estimation step ( S220 ) may include estimating information on the location and/or movement of at least some of the objects recognized in the object recognition step ( S210 ).
  • the position information may include an absolute position such as coordinates of an object, a relative position from a specific reference, a distance (distance from an arbitrary point, a distance range, etc.), a direction, etc.
  • the movement information includes an absolute velocity, a relative velocity
  • Information about the movement of the object, such as speed, may be included.
  • the position/movement information of the object may be used when the vessel is berthing or berthing. For example, it is possible to assist or guide the safe berthing or berthing of a vessel by using the distance to the berth or quay wall, the approach speed based on them, the distance from other vessels, and the relative speed when berthing or disembarking a vessel.
  • the position/movement information of the object may be used when the vessel is operating. For example, it is possible to assist or guide the safe operation of a ship, such as detecting other ships or obstacles around the ship, warning of collisions using the distance to them, and their moving speed, or creating/recommending routes. . Alternatively, autonomous navigation may be performed based on this information.
  • the position/movement information of the object may be calculated based on the image.
  • location/movement information of a ship may be calculated based on an image including a ship, sea, and land as objects.
  • an object for estimating location/movement information is referred to as a target object.
  • a ship may be a target object.
  • the plurality of ships may be a target object.
  • the location/movement information of the object may be expressed in a plurality of categories having a certain range.
  • the distance information may be expressed in a short distance, a medium distance, and a long distance
  • the direction information may be expressed in a left direction, a front direction, and a right direction. It may be possible to combine these to express the left near field, the right far field, and the like.
  • the movement information may be expressed as a high speed, a low speed, or the like.
  • the position/movement information of the object may be expressed as an actual distance value, a direction value, a velocity value, and the like.
  • distance information may be expressed in units of meters (m)
  • direction information may be expressed in units of degrees
  • movement information may be expressed in units of cm/s.
  • the location/movement information of the object may be estimated based on an area or a point.
  • the distance between the ship and the quay wall may be estimated by calculating the distance between one point of the ship and one point of the quay wall, or may be estimated by calculating the shortest distance between one point of the ship and the quay wall.
  • the distance between the vessels may be estimated by calculating a distance between a point of the first vessel and a point of the second vessel.
  • One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.
  • the position/movement information of the object may be estimated based on image pixels. As described above, when estimating location/movement information based on a point, a point on an image may correspond to a pixel. Accordingly, the position/movement information of the object may be calculated based on the distance between image pixels.
  • Distance information between points may be calculated based on an interval between pixels. For example, a predetermined distance may be allocated to each pixel interval, and the distance between points may be calculated in proportion to the interval between pixels. As another example, the distance between pixels may be calculated based on the coordinate values of the pixels on the image, and the distance between points may be calculated based on this.
  • Movement information between points may be calculated based on a change in distance information between points.
  • movement information may be calculated based on a plurality of images or image frames.
  • movement information between points may be calculated based on a distance between points in a previous frame, a distance between points in a current frame, and a time interval between frames.
  • FIGS. 14 and 15 are diagrams for estimating position/movement information of an object according to an embodiment.
  • the position/movement information estimation step includes position/movement information f1, f2 with the berthing wall OBJ2 or other vessels OBJ3, OBJ4 when the vessel OBJ1 is berthing or berthing. It may include estimating information f3, f4. As shown in FIG. 14 , the position/movement information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 may be calculated for two points of the vessel OBJ1 . In this case, the two points may correspond to a point where the ship OBJ1 is in contact with the sea.
  • the distance between the vessel OBJ1 and the quay wall OBJ2 may be the shortest distance between the two points and the quay wall OBJ2.
  • the position/movement information f3 and f4 between the vessel OBJ1 and the other vessels OBJ3 and OBJ4 may be position/movement information between points corresponding to the bow or stern of the vessels OBJ1, OBJ3, OBJ4. .
  • berthing guide information when the location/movement information is used to assist or guide the berthing or berthing of the vessel, it may be referred to as berthing guide information or berthing guide information.
  • the step of estimating location/movement information may include estimating location/movement information (f5, f6) with another vessel (OBJ6) or an obstacle (OBJ7) such as a buoy when the vessel (OBJ5) is operating. have.
  • Port operation or management may be performed based on the data calculated in the location/movement information estimation step. For example, when a ship collides with a fender, the time to replace the insect repellent may be predicted by calculating the amount of impact from movement information such as the speed of the ship.
  • object recognition and location/movement information estimation may be performed in one step. For example, it is possible to recognize an object by performing segmentation or detection and at the same time estimate position/movement information of the object.
  • 16 is a flowchart of port monitoring according to an embodiment.
  • the port monitoring may include the step of monitoring the berth (S230) and the step of monitoring the collision (S240).
  • the device 10 may monitor the vessel included in the image berthing (S230).
  • Berthing monitoring may mean monitoring the process of the vessel berthing on the quay wall, and berthing monitoring may be to provide information necessary for berthing when the vessel is berthing on the quay wall.
  • berthing monitoring may be to obtain and provide information of the vessel in relation to the quay wall, and conversely, berthing monitoring may be to obtain and provide information of the quay wall in relation to the vessel.
  • berthing monitoring may be to provide berthing-related information, which is information necessary for berthing, when the vessel berths on the quay wall.
  • the berthing-related information may include information related to the berthing situation of the vessel, for example, the distance between the vessel and the quay wall, the speed at which the vessel approaches the quay wall, the condition of the quay wall, water depth, wind speed, strength of the sea current, etc. can do.
  • the berthing monitoring may be obtaining the bow distance and/or the stern distance of the vessel.
  • the bow distance may mean the distance between the bow of the ship and the quay wall
  • the stern distance may mean the distance between the stern and the quay wall of the ship.
  • the berthing monitoring may be acquiring the bow speed and/or the stern speed of the vessel.
  • the bow speed may mean the speed at which the bow of the ship approaches the quay wall
  • the stern speed may mean the speed at which the stern of the ship approaches the quay wall.
  • berthing monitoring may be to extract two points from an image including a vessel, and to acquire berthing-related information based on the extracted two points. For example, the berthing monitoring acquires two points corresponding to the bow and stern of the vessel in the image, obtains berthing-related information related to the bow of the vessel from one of the extracted two points, and It may be to acquire berthing-related information related to the stern of the ship from another one of the points.
  • the berthing-related information related to the bow of the ship may include a bow distance and bow speed of the ship
  • the berthing-related information related to the stern of the ship may include the stern distance and stern speed of the ship.
  • the extracted two points may be determined as two points corresponding to both ends of the bottom surface of the ship in contact with the sea level.
  • the device 10 may collision-monitor the vessel included in the image ( S240 ).
  • Collision monitoring may mean monitoring the vessel and its surroundings in order to prevent the vessel from colliding with an adjacent object, and the collision monitoring provides information necessary to prevent the vessel from colliding with an adjacent object (eg, a vessel).
  • an adjacent object eg, a vessel
  • collision monitoring is described as monitoring to prevent a ship and an adjacent ship from collide, but collision monitoring may be understood as monitoring to prevent a ship and an adjacent object from colliding.
  • the adjacent vessel is a target for calculating the collision-related information of the target vessel, and means a vessel in the vicinity of the target vessel.
  • the adjacent vessel may be a vessel that is less than or equal to a predetermined distance from the target vessel.
  • the adjacent vessel may be a vessel closest to the target vessel among several vessels or a vessel other than the target vessel among vessels included in the image.
  • collision monitoring may be to obtain and provide information on a target vessel to be monitored in relation to an adjacent vessel, and conversely, collision monitoring may be to obtain and provide information on an adjacent vessel in relation to a target vessel.
  • collision monitoring may provide collision-related information that is information related to a collision situation of a ship.
  • Collision-related information may include information for preventing collision of ships.
  • collision-related information may include the existence of an adjacent ship, the distance between the ship and the adjacent ship, and the speed of approaching the ship and the adjacent ship.
  • the collision monitoring may be to obtain the distance between the bow of the vessel and another vessel and/or the distance between the stern of the vessel and the other vessel. Specifically, the collision monitoring may be to obtain a distance between one and the other vessel closer to the other of the bow and stern of the target vessel.
  • a distance between the bow of a ship and another ship or a distance between the stern of a ship and another ship may be referred to as an inter-ship distance for convenience.
  • the collision monitoring is the speed at which the bow of the vessel approaches the other vessel (relative speed between the bow of the vessel and the other vessel) and/or the speed at which the stern of the vessel approaches the other vessel (different from the stern of the vessel). relative speed between ships).
  • the collision monitoring may be to acquire the speed at which one of the bow and stern of the target vessel, which is closer to the other vessel, approaches the other vessel.
  • collision monitoring may be to extract two points from an image including a ship, and to obtain collision-related information based on the extracted two points. For example, collision monitoring acquires two points corresponding to the bow and stern of the ship in the image, obtains collision-related information related to the bow of the ship from one of the extracted two points, and obtains the extracted two points. It may be to acquire collision-related information related to the stern of the ship from another one of the points.
  • the collision-related information related to the bow of the vessel may include a distance between the bow of the vessel and another vessel, a speed at which the bow of the vessel approaches another vessel, and the like, and the collision related information related to the stern of the vessel is the stern of the vessel.
  • the extracted two points may be determined as two points corresponding to the bow and stern of the ship or two points corresponding to the fore and aft ends of the ship.
  • image-based monitoring may be performed based on a total of four points, two points for eyepiece monitoring and two points for collision monitoring, as described above.
  • image-based monitoring is eyepiece monitoring. It may be performed based on a smaller number of points or a larger number of points, such as may be performed based on one point for , and two points for collision monitoring.
  • 17 is a flowchart related to berthing monitoring and collision monitoring according to an embodiment.
  • port monitoring is a step of extracting a pair of first points and a pair of second points of the vessel (S222), berthing-related based on the pair of first points It may include obtaining information (S232) and obtaining collision-related information based on the pair of second points (S242).
  • the berthing-related information and/or the collision-related information of the vessel may be included in the berthing guide information.
  • the device 10 may extract a pair of first points and a pair of second points from an image including a ship ( S222 ).
  • the control module 200 may extract a pair of first points for obtaining the berthing-related information of the vessel.
  • the control module 200 may extract a pair of first points for obtaining the bow distance and the stern distance of the ship.
  • the control module 200 may extract a pair of first points corresponding to both ends of the bottom surface of the vessel in contact with the sea level from the image.
  • the control module 200 may extract in other ways, such as extracting two points from among a plurality of points forming the bottom surface of the ship in contact with the sea level as a pair of first points.
  • control module 200 may extract a pair of first points based on an area corresponding to the vessel.
  • FIG. 18 is a diagram related to an example of point extraction for obtaining eyepiece-related information according to an embodiment.
  • the control module 200 may extract a line 152 in which an area corresponding to the vessel is in contact with the sea level. To this end, the control module 200 may obtain a region corresponding to the vessel from the image. For example, the control module 200 may perform image segmentation to generate a segmented image from an image obtained from a camera, and obtain a region in the segmented image in which a class value of a pixel corresponds to a ship.
  • control module 200 may extract a line 152 in which an area corresponding to the acquired vessel is in contact with the sea level.
  • the control module 200 may extract a line in which a region corresponding to a ship having a class value of a pixel in the segmented image is in contact with the sea level.
  • control module 200 may determine any two points among a plurality of points forming the line 152 in contact with the sea level as a pair of first points. For example, the control module 200 may determine both ends of the line in contact with the sea level as a pair of first points 151-1 and 151-2. However, even if the control module 200 determines the pair of first points in a different way, such as determining the points located at a location separated by a predetermined distance from both ends of the line in contact with the sea level as a pair of first points, free of charge
  • control module 200 may extract a pair of first points 151-1 and 151-2 based on an image such as a captured image obtained from a camera.
  • control module 200 may extract a pair of first points in a manner that polygons the vessel on the image. For example, the control module 200 may determine a boundary polygon indicating the boundary of the vessel on the image, and the control module 200 may extract a pair of first points based on the determined boundary polygon.
  • FIG. 19 is a diagram related to another example of point extraction for obtaining eyepiece-related information according to an embodiment.
  • the control module 200 may determine a boundary polygon 161 indicating the boundary of the vessel by polygonizing the vessel on the image. For example, the control module 200 may perform image segmentation to generate the boundary polygon 161 indicating the boundary of the vessel based on the segmentation image from the image obtained from the camera. Specifically, the control module 200 may determine a boundary polygon 161 that indicates the boundary of the vessel by polygonizing an area in the segmented image where the class value of the pixel corresponds to the vessel.
  • control module 200 may extract a pair of first points 163 for obtaining the berthing-related information of the vessel based on the determined boundary polygon 161 .
  • the control module 200 may determine the pair of first points 163 - 1 and 163 - 2 as any two points among a plurality of points of the boundary polygon 161 .
  • the control module 200 may determine the pair of first points 163-1 and 163-2 as any two points among a plurality of points corresponding to the lower part of the vessel of the boundary polygon 161. have.
  • the control module 200 sets the pair of first points 163-1 and 163-2 to the two points closest to each of the two points corresponding to the front end and the stern end of the vessel of the boundary polygon 161. points can be determined.
  • control module 200 may determine the pair of first points 163-1 and 163-2 in various ways based on the boundary polygon 161, and the control module 200 may determine the pair of first points 163-1 and 163-2 in various ways.
  • the pair of first points 163 - 1 and 163 - 2 may be determined in another method without being limited to the above-described method, such as determining each of the points as two points adjacent to each other.
  • control module 200 may perform the extraction of the pair of first points 163-1 and 163-2 based on a specific area including the boundary polygon 161 rather than the boundary polygon 161 itself.
  • the boundary polygon 161 may have a different number of vertices depending on the shape and size of the vessel, such as a quadrangle, a pentagon, a hexagon, and a heptagon.
  • the control module 200 may extract a pair of first points 163-1 and 163-2 based on an image such as a captured image obtained from a camera.
  • control module 200 may extract a pair of first points of the vessel based on a part of the boundary polygon 161 of the vessel on the image.
  • the control module 200 may determine a preset area of the boundary polygon 161 for extracting the pair of first points.
  • the control module 200 may extract a pair of first points based on the preset area.
  • the control module 200 may determine a lower area of the boundary polygon 161 corresponding to the lower part of the ship, and extract a pair of first points based on the lower area of the boundary polygon 161 .
  • the control module 200 may determine the preset area of the boundary polygon in various ways.
  • the control module 200 may determine a preset area based on the size or shape of the vessel.
  • the control module 200 may determine the preset area of the boundary polygon to increase as the size of the vessel increases.
  • the control module 200 may determine the preset area of the boundary polygon as a circular area according to the shape of the vessel.
  • control module 200 may extract a pair of first points based on a preset area including a part of the boundary polygon, as well as extracting two points instead of obtaining berthing-related information of the vessel It is ok to acquire information related to berthing of a ship based on the outline of the area or the area itself.
  • control module 200 may extract a pair of second points for obtaining collision-related information of a ship.
  • control module 200 may extract a pair of second points for obtaining the intership distance of the vessel.
  • control module 200 may extract a pair of second points corresponding to the bow and stern of the ship from the image.
  • control module 200 may extract a pair of second points based on an area corresponding to the vessel.
  • 20 is a diagram illustrating an example of point extraction for obtaining collision-related information according to an embodiment.
  • a bow 172 corresponding to the bow side of the vessel and a stern 173 corresponding to the stern side may be extracted from the area corresponding to the vessel.
  • the bow side of the ship may be a part of the ship from the end of the bow to the bow end of the bottom of the ship in contact with the sea level
  • the aft side of the ship is from the end of the stern to the stern side of the bottom where the ship is in contact with the sea level
  • the control module 200 may obtain a region corresponding to the vessel from the image.
  • control module 200 may perform image segmentation to generate a segmented image from an image obtained from a camera, and the control module 200 obtains an area in which the class value of a pixel in the segmented image corresponds to a ship. can do.
  • control module 200 may extract the bow 172 corresponding to the bow side of the ship and the stern ship 173 corresponding to the stern side in the area corresponding to the vessel.
  • the control module 200 extracts the bow line 172 corresponding to the bow side of the vessel and the stern line 173 corresponding to the stern side in the region where the class value of the pixel in the segmentation image corresponds to the vessel.
  • control module 200 may determine one point among a plurality of points forming the bow line 172 and one point among a plurality of points forming the stern line 173 as a pair of second points. For example, the control module 200 may determine one end of the bow line 172 and one end of the stern line 173 as a pair of second points 171-1 and 171-2. .
  • one end of the bow line 172 may be an end of the bow side
  • one end of the stern line 173 may be an end of the stern side.
  • control module 200 determines the points at a location separated by a predetermined distance from one end of the bow line 172 and one end of the stern line 173 as a pair of second points, etc.
  • the pair of second points may be determined in this manner.
  • control module 200 may extract a pair of second points 171-1 and 171-2 based on an image such as a captured image obtained from a camera.
  • control module 200 may extract a pair of second points in a manner that polygons the vessel on the image.
  • control module 200 may determine a boundary polygon indicating the boundary of the vessel on the image.
  • the control module 200 may extract a pair of second points based on the determined boundary polygon.
  • 21 is a diagram illustrating another example of point extraction for obtaining collision-related information according to an embodiment.
  • the control module 200 may determine a boundary polygon 181 indicating the boundary of the vessel by polygonizing the vessel on the image.
  • the control module 200 may extract a pair of second points 183 - 1 and 183 - 2 for obtaining collision related information of a ship based on the determined boundary polygon 181 .
  • the control module 200 may extract a pair of second points 183 - 1 and 183 - 2 corresponding to the bow and stern of the ship based on the boundary polygon 181 .
  • the control module 200 may determine the pair of second points 183 - 1 and 183 - 2 as any two points among a plurality of points of the boundary polygon 181 .
  • the control module 200 sets the pair of second points 183 - 1 and 183 - 2 to a plurality of second points corresponding to the first direction part (eg, bow) of the vessel of the boundary polygon 181 .
  • the control module 200 converts the pair of second points 183-1 and 183-2 to one point corresponding to the fore end of the bow points and one point corresponding to the aft end among the stern points. can be decided with
  • the control module 200 determines a pair of second points 183 - 1 and 183 - 2 between each of the second points 183 - 1 and 183 - 2 among the points of the boundary polygon 181 and another object.
  • the two points at which the sum of the distances are the minimum or one point closest to the point corresponding to the fore end of the bow points and one point closest to the point corresponding to the aft end among the stern points are determined as described above, etc.
  • the method is not limited thereto, and the control module 200 may determine the pair of second points 183 - 1 and 183 - 2 based on the boundary polygon 181 in various ways.
  • control module 200 may perform the extraction of the pair of second points 183-1 and 183-2 based on a specific area including the boundary polygon 181 rather than the boundary polygon 181 itself.
  • the boundary polygon 181 may have a different number of vertices depending on the shape and size of the vessel, such as a quadrangle, a pentagon, a hexagon, and a heptagon. This is not necessarily the case, and the control module 200 may extract a pair of second points 183 - 1 and 183 - 2 based on an image such as a captured image obtained from a camera.
  • the control module 200 may extract a pair of second points 183 - 1 and 183 - 2 based on a part of the boundary polygon 181 of the vessel on the image. Specifically, the control module 200 may determine a preset area of the boundary polygon 181 for extraction of the pair of second points 183 - 1 and 183 - 2 . For example, the control module 200 may determine a lower region of the boundary polygon 181 corresponding to the lower portion of the vessel. The control module 200 may extract a pair of second points 183 - 1 and 183 - 2 based on the preset area. For example, the control module 200 may extract a pair of second points 183-1 and 183-2 for acquiring collision-related information of a vessel based on the lower region of the boundary polygon 181 . have.
  • the control module 200 may determine the preset area of the boundary polygon 181 in various ways. For example, the control module 200 may determine the preset area based on the size or shape of the vessel. Specifically, the control module 200 may determine the preset area of the boundary polygon 181 to increase as the size of the vessel increases. As another example, the control module 200 may determine the preset area of the boundary polygon 181 as a circular area according to the shape of the vessel.
  • control module 200 may extract a pair of second points 183 - 1 and 183 - 2 based on a preset area including a part of the boundary polygon 181 .
  • control module 200 may acquire collision-related information of a ship based on the outline of the area or the area itself, instead of obtaining the collision-related information of the vessel by extracting two points.
  • the step of extracting the pair of first points and the pair of second points of the vessel is for the control module 200 to obtain berthing-related information of the vessel using an artificial neural network. It may include extracting a pair of first points and/or a pair of second points for obtaining collision related information of the vessel.
  • the control module 200 may extract a pair of first points for obtaining a bow distance and a stern distance of a ship using an artificial neural network.
  • the control module 200 may extract a pair of second points for obtaining the intership distance of the vessel using the artificial neural network.
  • 22 is a diagram related to an example of extraction of points for port monitoring according to an embodiment.
  • control module 200 may extract a pair of first points 191-1 and 191-2 for acquiring berthing-related information of a ship by image segmentation from an image of the ship. .
  • the control module 200 is a pair of first for acquiring information related to berthing of a vessel by image segmentation using an artificial neural network trained to output feature points of an object included in the input image from an input image.
  • 1 points 191-1 and 191-2 may be extracted.
  • the control module 200 is a class indicating both ends of the sea, the vessel, and the bottom surface in contact with the sea level of the vessel, respectively, to the pixels corresponding to the input image and the objects including the vessel and the sea included in the input image.
  • a pair of first points 191-1 and 191-2 may be extracted by performing image segmentation using an artificial neural network trained using a learning set labeled with values.
  • the control module 200 may generate a segmentation image from the image, and may obtain a pixel whose class value of the segmented image corresponds to both ends of the bottom surface in contact with the sea level of the ship.
  • control module 200 is a pair of first points for acquiring the berthing-related information of the vessel by image detection using an artificial neural network trained to output the feature points of the object included in the input image from the input image ( 191-1, 191-2) may be extracted.
  • control module 200 may perform segmentation/detection through one artificial neural network, and the control module 200 uses a plurality of artificial neural networks to perform segmentation/detection for each artificial neural network to form a pair of The first points 191-1 and 191-2 may be extracted.
  • control module 200 may extract a pair of second points 192-1 and 192-2 for acquiring collision-related information of a ship by image segmentation from the image of the ship.
  • the control module 200 is a pair of first for obtaining collision-related information of a ship by image segmentation using an artificial neural network trained to output feature points of an object included in the input image from an input image.
  • Two points 192-1 and 192-2 may be extracted.
  • the control module 200 indicates one point of the sea, the ship, and the bow and one point of the stern to pixels corresponding to the input image and objects including the ship and the sea included in the input image, respectively.
  • a pair of second points 192-1 and 192-2 may be extracted by performing image segmentation using an artificial neural network learned using a learning set that labels class values.
  • the control module 200 may generate a segmentation image from the image, and may obtain pixels whose class values correspond to the bow and the stern of the segmented image.
  • the class value of the pixel may be a value set at the front end and the stern end.
  • control module 200 provides a pair of second points 192 for acquiring collision-related information of a ship by image detection using an artificial neural network trained to output feature points of an object included in the input image from the input image. -1, 192-2) may be extracted.
  • control module 200 may perform segmentation/detection through one artificial neural network, and the control module 200 uses a plurality of artificial neural networks to perform segmentation/detection for each artificial neural network to form a pair of The second points 192-1 and 192-2 may be extracted.
  • the extraction ( S222 ) of the pair of first points and the pair of second points is not limited to being performed by the above-described method and may be performed in various ways.
  • the device 10 may acquire eyepiece-related information based on the extracted pair of first points (S232).
  • the step of obtaining eyepiece-related information based on the pair of first points may include obtaining, by the control module 200, eyepiece-related information related to each of the pair of first points.
  • the control module 200 may obtain a distance between one first point and the quay wall and a distance between the other first point and the quay wall among the pair of first points. That is, the control module 200 may acquire the bow distance and the stern distance of the ship based on the pair of first points.
  • the control module 200 may acquire the bow distance and/or the stern distance of the vessel based on the image pixel.
  • the control module 200 may acquire the distance between the pair of first points and the quay wall based on the image pixel.
  • the control module 200 may obtain the distance between the first point of the pair of first points and the quay wall as the bow distance based on the image pixel, and the control module 200 may A distance between the first point of the other one of the first points and the quay wall may be obtained as the stern distance based on the image pixel.
  • the pixel may be a pixel on an image on which the control module 200 performs image analysis, but is not limited thereto, such as a pixel on an image displayed on a screen.
  • FIG. 23 is a diagram related to an example of obtaining eyepiece-related information according to an embodiment.
  • control module 200 may obtain a distance between each of the pair of first points 2001 and 2002 and the quay wall based on the image pixel.
  • the control module 200 may acquire the bow distance 2003 between the first point 2001 and the quay wall based on the image pixel. For example, the control module 200 may allocate a certain distance to each image pixel and calculate the bow distance 2003 in proportion to the number of pixels between the first point 2001 and the quay wall. Specifically, the control module 200 may calculate the bow distance 2003 by multiplying the number of pixels between the first point 2001 and the quay wall and the distance allocated to each image pixel. For example, when the distance allocated to each pixel in FIG. 23 is 50 m, the number of pixels between the first point 2001 and the quay wall is 5, so the control module 200 sets the bow distance 2003 for each pixel.
  • control module 200 may increase the distance allocated to the pixel as the pixel is farther from the quay wall on the image by the camera angle.
  • control module 200 may calculate the distance between pixels based on the coordinate values of the pixels on the image, and calculate the distance between points based on this.
  • the control module 200 may obtain the stern distance 2004 between the first point 2002 and the quay wall based on the image pixel. For example, the control module 200 may allocate a predetermined distance to each image pixel and calculate the stern distance 2004 in proportion to the number of pixels between the first point 2002 and the quay wall. Specifically, the control module 200 may calculate the stern distance 2004 by multiplying the number of pixels between the first point 2002 and the quay wall by the distance allocated for each image pixel. For example, when the distance allocated to each pixel in FIG. 23 is 50 m, the number of pixels between the first point 2002 and the quay wall is 6.5, so the control module 200 sets the stern distance 2004 to each pixel.
  • control module 200 may increase the distance allocated to the pixel as the pixel is farther from the quay wall on the image by the camera angle. Also, the control module 200 may calculate the distance between pixels based on the coordinate values of the pixels on the image, and calculate the distance between points based on this.
  • the step of obtaining eyepiece-related information based on the pair of first points may include obtaining the speed at which the control module 200 approaches the quay wall of each of the pair of first points. .
  • the control module 200 may acquire the speed at which the one first point approaches the quay wall based on a distance between the first point of the pair of first points and the quay wall.
  • the control module 200 may acquire a speed at which the other first point approaches the quay wall based on a distance between the quay wall and the other first point among the pair of first points. That is, the control module 200 may acquire the bow speed and the stern speed of the ship based on the pair of first points.
  • the control module 200 may acquire the bow speed and/or the stern speed of the vessel based on the image pixel. For example, the control module 200 may acquire a speed at which each of the pair of first points approaches the quay wall based on the image pixel. Specifically, the control module 200 may acquire the speed at which one of the pair of first points approaches the quay wall as the bow speed based on the image pixel. Also, the control module 200 may acquire, as a stern speed, a speed at which the other first point of the pair of first points approaches the quay wall based on the image pixel.
  • control module 200 may acquire the speed at which each of the pair of first points 2001 and 2002 approaches the quay wall based on the image pixel.
  • the control module 200 may acquire the speed at which the first point 2001 approaches the quay wall based on the image pixel. For example, the control module 200 may acquire the bow speed based on the bow distance 2003 that is the distance between the first point 2001 and the quay wall acquired based on the image pixel. Specifically, the control module 200 may calculate the speed of the bow based on the obtained change in the distance 2003 of the bow. In this case, the control module 200 may calculate the bow speed by comparing the bow distances 2003 in a plurality of images or image frames. For example, the control module 200 may calculate the bow speed based on a difference in the number of pixels between the bow distance 2003 in the current frame and the bow distance in a subsequent frame and a time interval between frames.
  • the control module 200 calculates a value obtained by dividing the time interval between each frame by multiplying the distance allocated to the pixel by the difference between the number of pixels of the bow distance 2003 in the current frame and the bow distance in the subsequent frame. It can be calculated by player speed. For example, in FIG. 23 , when the distance allocated to each pixel is 50 m, the time interval between frames is 1 minute, and the difference in the number of pixels between the bow distance 2003 in the current frame and the bow distance in the subsequent frame is 1 If the dog, the control module 200 may calculate the bow speed as described above as 50 m/min. Here, when the number of pixels of the bow distance in a subsequent frame is greater than the bow distance in the current frame, the control module 200 may determine the direction of the bow speed as the bow moves away from the quay wall.
  • the control module 200 may acquire the stern velocity at which the first point 2002 approaches the quay wall based on the image pixel. For example, the control module 200 may acquire the stern speed based on the stern distance 2004 that is the distance between the first point 2002 and the quay wall acquired based on the image pixel. Specifically, the control module 200 may calculate the stern speed based on the obtained change in the stern distance 2004 . In this case, the control module 200 may calculate the stern speed by comparing the stern distance 2004 in a plurality of images or image frames. For example, the control module 200 may calculate the stern speed based on the difference in the number of pixels between the stern distance 2004 in the current frame and the stern distance in the subsequent frame and the time interval between frames.
  • the control module 200 calculates a value obtained by dividing the time interval between each frame by multiplying the distance allocated to the pixel by the difference between the number of pixels of the stern distance 2004 in the current frame and the stern distance in the subsequent frame. It can be calculated at stern speed. For example, if the distance allocated to each pixel in FIG. 23 is 50 m, the time interval between frames is 1 minute, and the difference in the number of pixels between the stern distance 2004 in the current frame and the stern distance in the subsequent frame is 2 If the dog, the control module 200 may be calculated as a stern speed of 100 m/min as described above. Here, when the number of pixels of the stern distance in a subsequent frame is greater than the stern distance in the current frame, the control module 200 may determine the direction of the stern speed as the stern moves away from the quay wall.
  • control module 200 may increase the distance allocated to the pixel as the pixel is farther from the quay wall in the image due to the camera angle. Also, the control module 200 may calculate the distance between pixels based on the coordinate values of the pixels on the image, and calculate the relative speed between points based on the distance between the pixels.
  • the device 10 may acquire collision-related information based on the extracted pair of second points ( S242 ).
  • the step of obtaining collision-related information based on the pair of second points includes obtaining, by the control module 200 , collision-related information related to each of the pair of second points. can do.
  • the control module 200 may acquire the intership distance of the vessel based on the pair of second points.
  • the control module 200 may acquire the intership distance of the vessel based on one second point among the pair of second points.
  • the control module 200 may acquire the intership distance of the vessel based on one second point closer to the neighboring vessel.
  • the control module 200 may obtain a distance between each of the pair of second points and an adjacent vessel, and determine a lower distance value among the obtained distances as the intership distance.
  • the control module 200 may obtain the distance between the vessels of the vessel based on the image pixel. For example, the control module 200 may obtain a distance between each of the pair of second points and another vessel based on the image pixel. Here, the control module 200 may acquire the distance between the second point closer to the adjacent ship and the adjacent ship based on the image pixel.
  • 24 is a diagram illustrating an example of obtaining collision-related information according to an embodiment.
  • control module 200 may obtain a distance between a pair of second points 2101 and 2102 and another vessel 2105 based on an image pixel.
  • the control module 200 may acquire the vessel-to-ship distance 2103 of the vessel based on the image pixel. For example, the control module 200 obtains a distance between a second point 2101 closer to another ship 2105 among a pair of second points 2101 and 2102 of the ship based on the image pixel. can do.
  • the control module 200 allocates a certain distance to each image pixel, and the inter-vessel distance 2103 is proportional to the number of pixels between the second point 2101 closer to the other vessel 2105 and the other vessel 2105 . ) can be calculated.
  • the control module 200 may calculate the intership distance 2103 by multiplying the number of pixels between the second point 2101 and the other vessel 2105 by the distance allocated for each image pixel.
  • the control module 200 determines the inter-vessel distance 2103 ) can be calculated as 350 m, which is a value obtained by multiplying the distance 50 m allocated for each pixel and the number of pixels between the second point 2101 and the other vessel 2105 multiplied by seven.
  • the control module 200 may increase the distance allocated to the pixel as the pixel is farther from the quay wall on the image by the camera angle.
  • the control module 200 may calculate the distance between pixels based on the coordinate values of the pixels on the image, and calculate the distance between points based on this.
  • the control module 200 when the control module 200 has an adjacent ship, it is adjacent to the ship based on the pair of second points. It may include obtaining the relative speed between the vessels. For example, when there is an adjacent ship, the control module 200 may acquire the relative speed between the ship and the adjacent ship based on one of the pair of second points. Here, the control module 200 may acquire the relative speed between the ship and the adjacent ship based on one second point closer to the adjacent ship among the pair of second points. Specifically, the control module 200 may acquire the speed at which the one second point approaches the other vessel based on the distance between the second point of the pair of second points and the other vessel.
  • the control module 200 may obtain a relative speed between the vessel and another vessel based on the image pixel. For example, the control module 200 may acquire the speed at which the bow of the vessel approaches another vessel and/or the speed at which the stern of the vessel approaches the other vessel based on the image pixel. For example, the control module 200 may acquire the speed at which each of the pair of second points approaches the other vessel. Specifically, the control module 200 may acquire the speed at which one second point closer to another ship from among the pair of second points approaches the other ship based on the image pixel.
  • control module 200 may acquire the speed at which each of the pair of second points 2101 and 2102 approaches the other vessel 2105 based on the image pixel.
  • the control module 200 may acquire the speed at which the second point 2101 approaches another vessel 2105 based on the image pixel. For example, the control module 200 may control the relative speed between the vessel and the other vessel 2105 based on the vessel-to-vessel distance, which is the distance between the second point 2101 and the other vessel 2105 obtained based on the image pixel. can be obtained. Specifically, the control module 200 may calculate the relative speed between the vessel and the other vessel 2105 based on the obtained change in the intership distance. In this case, the control module 200 may calculate the relative speed between the ship and the other ship 2105 by comparing the distances between ships in a plurality of images or image frames.
  • control module 200 may control the relative speed between the vessel and the other vessel 2105 based on the difference in the number of pixels of the inter-vessel distance in the current frame and the inter-vessel distance in the subsequent frame and the time interval between frames. can be calculated. Specifically, the control module 200 calculates a value obtained by dividing the time interval between each frame from a value obtained by multiplying the distance allocated to a pixel by the difference between the number of pixels of the distance between ships in the current frame and the distance between ships in the subsequent frame. It can be calculated as the relative speed between and another vessel (2105). For example, if the distance allocated to each pixel in FIG.
  • the rear surface control module 200 may calculate the relative speed between the ship and the other ship 2105 as 150 m/min as described above. Here, the control module 200 determines that the vessel moves away from the other vessel 2105 when the number of pixels of the intership distance in the subsequent frame is greater than the intership distance in the current frame, and the relative speed between the vessel and the other vessel 2105 direction can be determined.
  • Objects that exist in the image are treated as if they exist on a horizontal plane (sea level). That is, the upper part of the vessel is treated as being farther away than the lower part. Therefore, when calculating the distance/relative speed between the ship and other objects, an error may occur when the calculation is based on the upper part of the ship, and in particular, when the ship is inclined, the error may become larger.
  • the control module 200 determines that the pair of second points and the ship are at sea level and The method may include acquiring collision-related information between a ship and another ship based on a pair of third points at a position perpendicular to the contacting bottom surface.
  • the control module 200 may be configured to determine the inter-ship distance of the vessel and/or the vessel based on the pair of second points and the pair of third points at a position in which the vessel is in contact with the bottom surface in contact with the sea level and/or vertically. It is possible to obtain the relative speed between adjacent ships.
  • the control module 200 may obtain the intership distance of the vessel and/or the relative speed between the vessel and the neighboring vessel based on one third point closer to the neighboring vessel among the pair of third points. .
  • 25 is a diagram illustrating another example of obtaining collision-related information according to an embodiment.
  • the control module 200 performs a third pair of points for obtaining collision-related information of a ship based on a pair of second points 2201-1 and 2201-2 extracted through step S222.
  • Points 2203-1 and 2203-2 may be extracted.
  • the control module 200 sets the pair of third points 2203-1 and 2203-2 to the pair of second points 2201-1 and 2201-2 and the bottom surface of the vessel in contact with the sea level. It can be determined by two points perpendicular to and tangent to.
  • the control module 200 acquires collision-related information of a ship based on a third point 2203 that is closer to another ship 2204 among a pair of third points 2203-1 and 2203-2.
  • the control module 200 may be configured to connect the third point 2203 - 2 closer to the other vessel 2204 among the pair of third points 2203 - 1 and 2203 - 2 and the other vessel 2204 . distance can be obtained.
  • the control module 200 converts the third points 2203-1 and 2203-2 to an arbitrary point between the pair of second points 2201-1 and 2201-2 and the bottom surface of the vessel in contact with the sea level.
  • the pair of third points 2203-1 and 2203-2 may be variously determined based on the pair of second points 2201-1 and 2201-2 without being limited to the above-described method. have.
  • the control module 200 may extract one or two or more points instead of extracting the two third points 2203-1 and 2203-2 for obtaining the collision-related information of the ship.
  • the control module 200 may obtain the distance between the vessels of the vessel based on the image pixel. For example, the control module 200 may obtain a distance between each of the pair of third points and another vessel based on the image pixel. Here, the control module 200 may acquire the distance between the third point closer to the other vessel and the other vessel based on the image pixel.
  • control module 200 may obtain a distance between a pair of third points 2203 - 1 and 2203 - 2 and another vessel 2204 based on an image pixel.
  • the control module 200 may acquire the ship-to-ship distance 2205 of the ship based on the image pixel. For example, the control module 200 may control one third point 2203 - 2 closer to the other ship 2204 among the pair of third points 2203 - 1 and 2203 - 2 of the ship and the other ship. The distance between 2204 may be obtained based on image pixels. Here, the control module 200 allocates a certain distance to each image pixel, and the distance between ships is proportional to the number of pixels between the third point 2203 - 2 closer to the other ship 2204 and the other ship 2204 . (2205) can be calculated.
  • the control module 200 may calculate the intership distance 2205 by multiplying the number of pixels between the third point 2203 - 2 and the other vessel 2204 by the distance allocated to each image pixel. For example, when the distance allocated to each pixel in FIG. 25 is 50 m, the number of pixels between the third point 2203-2 and the other vessel 2204 is 11, so the control module 200 determines the inter-vessel distance (2205) can be calculated as 550m, which is a value obtained by multiplying a distance of 50m allocated to each pixel by 11 pixels between the third point 2203-2 and another vessel.
  • the control module 200 may increase the distance allocated to the pixel as the pixel is farther from the quay wall on the image by the camera angle.
  • the control module 200 may calculate the distance between pixels based on the coordinate values of the pixels on the image, and calculate the distance between points based on this.
  • the control module 200 may acquire the relative speed between the vessel and the adjacent vessel based on the image pixel. For example, the control module 200 may acquire the speed at which each of the pair of third points approaches the other vessel. Specifically, the control module 200 may acquire, based on the image pixel, the speed at which one third point closer to another ship approaches the other ship among the pair of third points.
  • the control module 200 may acquire the speed at which each of the pair of third points 2203 - 1 and 2203 - 2 approaches the other vessel 2204 based on the image pixel. For example, the control module 200 may determine that one third point 2203-2 closer to the other ship 2204 among the pair of third points 2203-1 and 2203-2 of the ship is the other ship. The speed of approaching 2204 may be obtained based on image pixels.
  • the control module 200 determines the distance between the vessel and the other vessel 2204 based on the vessel-to-vessel distance 2205 between the third point 2203-2 obtained based on the image pixel and the other vessel 2204. Relative speed can be obtained. For example, the control module 200 may calculate the relative speed between the vessel and the other vessel 2204 based on the obtained change in the intership distance 2205 . In this case, the control module 200 may calculate the relative speed between the ship and the other ship 2204 by comparing the distances 2205 between ships in a plurality of images or image frames.
  • control module 200 may control the distance between the vessel and the other vessel 2204 based on the difference in the number of pixels between the inter-vessel distance 2205 in the current frame and the inter-vessel distance in the subsequent frame and the time interval between frames.
  • a value can be calculated as the relative speed between the vessel and the other vessel 2204 . For example, if the distance allocated to each pixel in FIG.
  • the control module 200 may calculate the relative speed between the vessel and the other vessel 2204 as 150 m/min as described above. Here, the control module 200 determines that the vessel moves away from the other vessel 2204 when the number of pixels of the intership distance in the subsequent frame is greater than the intership distance in the current frame, and the relative speed between the vessel and the other vessel 2204 direction can be determined.
  • a point corresponding to the other vessel may be determined in various ways.
  • the point corresponding to the other vessel may be one of the second points of the vessel extracted according to step S222.
  • a point corresponding to another object (ship) may be a third point in which the second point extracted according to step S222 is a point perpendicular to the bottom surface of the ship in contact with the sea level.
  • the port monitoring may further include the step of distinguishing the bow/stern of the vessel. Distinguishing the bow/stern of a ship may mean determining the front and rear of the ship.
  • the shape of the vessel is various, general characteristics for distinguishing the bow and the stern of the vessel may appear on the image.
  • the bow of a ship may be generally higher than the stern to prevent seawater from entering over the bow when the ship is sailing forward.
  • the ship is formed in a streamlined shape to reduce resistance during propulsion, so that the bow can have a sharper structure than the stern.
  • the control module 200 may determine the bow/stern of the vessel based on the image. For example, the control module 200 may determine the bow/stern of the ship based on an area corresponding to the ship. In more detail, the control module 200 may determine the bow/stern of the ship based on an area in which the class value of the pixel in the segmented image corresponds to the ship. As another example, the control module 200 may generate a boundary polygon of the vessel by polygonizing the vessel of the image, and determine the bow/stern of the vessel based on the boundary polygon of the vessel.
  • control module 200 is based on the extracted pair of first points 2301-1 and 2301-2 and/or the pair of second points 2302-1 and 2302-2. Thus, the bow/stern of the ship can be determined.
  • 26 and 27 are views related to the distinction between the bow / stern of the ship according to an embodiment.
  • the control module 200 extracts a pair of first points 2301-1 and 2301-2 and/or a pair of second points 2302-1 and 2302-2. ) based on the ship's bow/stern can be determined.
  • the control module 200 may determine the bow/stern of the ship based on the inclination between the extracted first point and the second point.
  • the control module 200 may include a first inclination of one first point 2301-1 and one second point 2302-1 located on one side of the vessel and another located on the other side of the vessel.
  • the bow/stern of the ship may be determined based on the second inclination difference between the first point 2301 - 2 and the second point 2302 - 2 .
  • the control module 200 is the angle formed by the first vertical line starting from the first point 2301-1 and the first line connecting the first point 2301-1 and the second point 2302-1.
  • the control module 200 may determine a portion in which the first point 2301 - 2 and the second point 2302 - 2 are located as the stern of the ship.
  • the control module 200 may determine the bow/stern of the ship based on the extracted positions between the first and second points. For example, the control module 200 is located at a first height between one first point 2301-1 and one second point 2302-1 located on one side of the vessel and the other side of the vessel. The bow/stern of the vessel may be determined based on a second height difference between the other first point 2301 - 2 and the other second point 2302 - 2 .
  • the control module 200 determines that the distance 2305 in the vertical direction between the first point 2301-1 and the second point 2302-1 is the first point 2301-2 and the second point ( When the distance 2306 in the vertical direction between 2302-2) is greater, the portion in which the first point 2301-1 and the second point 2302-1 are located may be determined as the bow of the ship.
  • the control module 200 may determine a portion in which the first point 2301 - 2 and the second point 2302 - 2 are located as the stern of the ship.
  • the control module 200 may determine the bow/stern of the ship using an artificial neural network to output the bow/stern of the ship included in the input image from the input image. For example, the control module 200 may generate class values indicating the sea, the vessel, the bow of the vessel, and the stern of the vessel in pixels corresponding to the input image and objects including the vessel and the sea included in the input image, respectively.
  • the fore/stern of the ship can be determined by performing image segmentation using the artificial neural network learned using the labeled running set. Specifically, the control module 200 may generate a segmentation image from the image, and may obtain pixels whose class values correspond to the bow and stern of the ship in the segmentation image.
  • control module 200 may determine the bow/stern of the ship by image detection using an artificial neural network learned to output the bow/stern of the ship included in the input image from the input image.
  • control module 200 may perform segmentation/detection through a single artificial neural network, and the control module 200 uses a plurality of artificial neural networks to perform segmentation/detection of each artificial neural network, so that the bow of the ship / You can also decide the stern.
  • the determination of the bow and stern of the ship is not necessarily limited to being performed by the above-described method, and may be performed in various ways.
  • the image-based monitoring may further include a viewpoint conversion step.
  • an image generated by an image generating unit such as a camera may be displayed as a perspective view. Converting this to a top view (planar view), a side view, another perspective view, etc. may be referred to as view transformation.
  • a top view or a side view image may be converted into another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, it may not be necessary to perform view point conversion.
  • 28 and 29 are diagrams for view transformation according to an embodiment.
  • FIG. 28 another perspective view image may be acquired through viewpoint transformation of the perspective view image.
  • viewpoint conversion may be performed so that the quay wall OBJ8 is positioned along a horizontal direction (left and right direction on the image) on the image.
  • a top view image may be acquired through viewpoint transformation of the perspective viewpoint image.
  • the top view image may be a view looking down at the sea level in a direction perpendicular to the sea level.
  • viewpoint conversion may be performed so that the quay wall OBJ9 is positioned along the horizontal direction on the image.
  • image analysis can be performed to obtain information necessary for the berthing guide of the vessel.
  • the acquisition of the eyepiece guide information may be performed based on the segmented image in which the viewpoint is converted. Specifically, the viewpoint is converted from the segmentation image of the perspective viewpoint into the top-view segmentation image, and the position/movement information of the vessel may be obtained based on the region in which the class value of the pixel in the top-view segmented image corresponds to the vessel.
  • a pair of first points for obtaining berthing-related information of the vessel and a pair of second points for obtaining collision-related information of the vessel are extracted And, based on the extracted pair of first points and the pair of second points, the distance/velocity between the ship and the quay wall and the distance/speed between the ship and another object (ship) may be acquired.
  • the pair of first points correspond to both ends of the bottom surface of the ship in contact with the sea level
  • the bow distance/speed and the stern distance/speed can be obtained based on the pair of first points, and the pair When the second points of ?
  • the distance/velocity between the ships may be obtained based on the pair of second points.
  • the viewpoint-converted segmentation image may be a segmentation image in which the viewpoint is converted to various viewpoints, such as a side viewpoint, rather than a top view.
  • the location/movement information of the vessel may be acquired based on the area itself.
  • monitoring information such as displaying an image to the user may be output after the viewpoint is changed after acquiring the image.
  • information on the surrounding situation can be more easily provided to the user through viewpoint conversion.
  • the viewpoint transformation of the image may be performed in various ways.
  • inverse projection transformation may be performed.
  • a two-dimensional image is generated when light reflected from a subject in a three-dimensional space is incident on an image sensor through the lens of the camera, and the relationship between two dimensions and three dimensions depends on the image sensor and lens, for example, Equation 1 and can be expressed together.
  • the matrix on the left side indicates two-dimensional image coordinates
  • the first matrix on the right side indicates an intrinsic parameter
  • the second matrix indicates an external parameter
  • the third matrix indicates three-dimensional coordinates.
  • fx and fy denote focal lengths
  • cx and cy denote principal points
  • r and t denote rotation and translation transformation parameters, respectively.
  • a perspective view image may be converted into a top view image through inverse projection transformation, or may be converted into another perspective view image.
  • the Zhang method is a type of polynomial model, and is a method of acquiring internal parameters by photographing a grid with a known size at various angles and distances.
  • Information on the position and/or posture of the image generating unit/sensor module capturing the image may be required for viewpoint conversion. Such information may be obtained from the position measuring unit and the posture measuring unit.
  • information on the position and/or posture may be acquired based on the position of the fixture included in the image.
  • the image generating unit may generate a first image including a target fixture that is disposed at a first position and/or a first posture and is a fixed object such as a terrain or a building.
  • the image generating unit may generate a second image including the target fixture. Comparing the position of the target fixture on the first image and the position of the target fixture on the second image to calculate a second position and/or a second posture that is the position and/or posture of the image generating unit at the second time point can do.
  • the accuracy of image analysis may vary depending on the selection of the reference plane when changing the viewpoint. For example, when a perspective view image is converted into a top view image, the accuracy of image analysis based on the top view image may vary according to the height of the reference plane. In order to accurately calculate the distance between objects on the sea level, it may be preferable that the reference plane be the sea level when changing the viewpoint. Since the height of the sea level may change with time, it may be desirable to perform viewpoint conversion in consideration of the height of the sea level to improve the accuracy of image analysis.
  • Acquisition of information on a position and/or posture for viewpoint transformation may be performed at predetermined time intervals.
  • the time interval may depend on the installation position of the image generating unit/sensor module.
  • the image generating unit/sensor module is installed in a moving object such as a ship, there may be a need to obtain information on the position and/or attitude at short time intervals.
  • information on the position and/or posture may be acquired at relatively long time intervals, or may be initially acquired only once.
  • moving and fixing are repeated like a crane, it may be implemented in a way of acquiring information about a position and/or posture only after moving.
  • the time interval for obtaining information on such a position and/or posture may be changed.
  • the device 10 may transform the image based on the reference plane.
  • the device 10 may view-convert an image from a plane in which a quay wall is located and parallel to the sea level to a reference plane.
  • the reference plane may depend on the calculated sea level height.
  • the device 10 is not limited to the above description, and the view of the image may be converted into a reference plane on another plane such as sea level, a part of the ship other than the plane where the quay wall is located (for example, the reference plane of the deck height).
  • the device 10 may convert the image to a viewpoint in consideration of the sea level.
  • the device 10 may calculate a sea level height using data obtained from a lidar sensor or a camera, and convert an image into a viewpoint in consideration of the calculated sea level height.
  • viewpoint transformation information includes viewpoint transformation such as information on the matrix, parameters, coordinates, position and/or posture of Equation 1 above. contains the necessary information for
  • Image-based monitoring for monitoring a vessel, etc. may include outputting monitoring information (eg, berthing guide information).
  • the information output in the monitoring information output step is not limited as long as it is information related to image-based monitoring, such as images of the vicinity of a ship, the ocean, or a port, and the type and distance/speed of objects included in the image.
  • Monitoring information may be output visually.
  • the monitoring information may be output through an output module such as a display.
  • the monitoring information output step may include displaying the image acquired in the image obtaining step using the image generating unit. In addition to this, it may include displaying various images related to image-based monitoring, such as an image that has undergone a pre-processing step, an image after segmentation or detection, and an image after viewpoint conversion, which will be described later.
  • the monitoring information output step may include displaying the position/movement information (eyepiece guide information) estimated in the image analysis step.
  • the image and eyepiece guide information may be displayed together.
  • the specifically displayed eyepiece guide information may include a bow distance, a bow speed, a stern distance, and a stern speed of the target vessel.
  • the step of outputting monitoring information may include providing information to the user in other ways such as outputting sound or vibration in addition to a visual display.
  • a warning sound may be output when the target vessel is in danger of colliding with a quay wall, other vessel, obstacle, etc., when the speed of approaching to the quay wall is higher than the reference speed when berthing, or when the vessel deviates from the route.
  • Image-based monitoring may include surveillance.
  • the surveillance may mean to provide the user with information about the occurrence of an emergency situation, such as a fire, and security-related information such as monitoring an intruder or monitoring an access of an unregistered vessel to a port.
  • Intruder monitoring can be performed based on whether or not a person is included in the image and when the image was captured. For example, if a person is included in the image of the port taken at a time when the operation is not in progress in the port, it may be determined that an intruder exists.
  • Vessel monitoring may be performed based on whether a vessel is included in the image. For example, when a vessel not registered with the AIS is detected, information about this may be provided to the user.
  • Surveillance can be performed by detecting people or ships based on images through segmentation or detection.
  • Image-based monitoring may include a pre-processing step.
  • Preprocessing refers to all kinds of processing performed on images, including image normalization, image equalization, histogram equalization, image resizing, and upscaling to change the resolution/size of the image. and downscaling, cropping, noise removal, and the like.
  • the noise may include fog, rain, water droplets, sea clutter, fine dust, direct sunlight, salt, and combinations thereof.
  • normalization may mean obtaining an average of RGB values of all pixels of an RGB image and subtracting it from the RGB image.
  • defogging may mean converting an image of a foggy area to look like an image of a clear area through preprocessing.
  • 13 is a view related to fog removal according to an embodiment. Referring to FIG. 13 , an image of a foggy area as shown in FIG. 13 ( a ) may be converted into an image in which fog is removed as shown in FIG. 13 ( b ) through fog removal.
  • water drop removal may mean converting water droplets on the front of the camera so that the water droplets appear to have been removed through pre-processing in the captured image.
  • the image analysis step ( S20 ) may be performed through a pre-processing step.
  • image analysis may be performed after performing pre-processing on an image obtained by using the image generating unit.
  • Image preprocessing may facilitate image analysis or improve accuracy.
  • Image preprocessing may be performed through an artificial neural network. For example, you can input an image of a foggy area into an artificial neural network to convert a clear area to look like a photographed image, etc. You can input an image containing noise into an artificial neural network to obtain an image with noise removed.
  • the artificial neural network include, but are not limited to, a GAN.
  • image preprocessing may be performed using an image mask.
  • an image mask For example, by applying an image mask to an image of a foggy area, you can transform a clear area to look like a photographed image.
  • examples of the image mask include a deconvolution filter, a sharpen filter, and the like, and an image mask may be generated through an artificial neural network such as a GAN, but is not limited thereto.
  • the case of performing image analysis after image preprocessing has been discussed.
  • the image analysis step includes segmentation or detection, it may be implemented so that a result of performing segmentation or detection of an image including noise is equivalent to a result of performing segmentation or detection of an image without noise.
  • eyepiece monitoring can be performed based on a plurality of images.
  • image analysis is performed based on a plurality of images, the total monitoring area of the device 10 may increase or monitoring accuracy may be improved.
  • FIG. 30 is a diagram of image-based monitoring based on a plurality of images according to an embodiment.
  • the image acquisition step may include a first image acquisition step S11 and a second image acquisition step S12 , and the image analysis step S20 is obtained in the first image acquisition step S11 .
  • Image analysis may be performed based on the first image and the second image acquired in the second image acquiring step ( S12 ).
  • control module 200 may generate a panoramic image in which a plurality of images are matched, and perform image analysis of the panoramic image.
  • control module 200 may generate a registered image or a fusion image by registering or fusing the first image and the second image, and may perform image analysis based on the generated registered image or the fusion image.
  • a final analysis result may be calculated based on a result of performing image analysis based on each of the plurality of images. For example, after image analysis is performed from the first image to obtain first monitoring information, and image analysis is performed to obtain second monitoring information from the second image, the final monitoring information is based on the first monitoring information and the second monitoring information. Monitoring information can be obtained.
  • a method of acquiring the final monitoring information from the plurality of monitoring information there may be a method of calculating the final monitoring information by considering the plurality of monitoring information for each weight.
  • the final monitoring information may be calculated based on whether a plurality of pieces of monitoring information do not match each other or whether the difference is equal to or greater than a threshold equal to a specific value (hereinafter referred to as “error occurrence”). For example, based on the occurrence of an error, the final monitoring information is calculated by considering a plurality of monitoring information by weight, or the final monitoring information is calculated by giving priority to specific monitoring information among a plurality of monitoring information, or the specific monitoring information is used for other monitoring There may be a method of compensating with information or ignoring the corresponding monitoring information, but is not limited thereto.
  • the plurality of images may be images of the same type.
  • the first image and the second image are of the same type can be
  • Monitoring areas of the plurality of images may be different from each other.
  • the first image may monitor a short distance from the image generating unit, and the second image may monitor a long distance.
  • the first image may monitor the left side from the image generating unit, and the second image may monitor the right side.
  • 31 is a diagram of image fusion according to an embodiment.
  • the control module 200 may perform image analysis ( S20 ) by fusing the RGB image IMG1 and the lidar image IMG2 .
  • the control module 200 may acquire an RGB image and a lidar image representing the same area through a camera and a lidar, and map both acquired images to each other.
  • the distance per pixel used for distance/velocity calculation in the RGB image may be determined based on the distance in the mapped lidar image.
  • the control module 200 fuses images to perform image analysis, as described above, the control module 200 fuses the first image and the second image to generate one fusion image, then performs image analysis to Acquire monitoring information (for example, by fusing an RGB image (IMG1) and a lidar image (IMG2) to generate one fused image and perform image analysis (S20) based on it), or image analysis from the first image to obtain the first monitoring information, perform image analysis from the second image to obtain the second monitoring information, and then fuse the first monitoring information and the second monitoring information to obtain the final monitoring information (e.g., RGB
  • the first monitoring information may be obtained through image analysis based on the image IMG1 and the second monitoring information may be obtained through image analysis based on the lidar image IMG2 and then fused to obtain final monitoring information).
  • FIG. 32 is a diagram for correcting position/movement information of an RGB image according to an exemplary embodiment.
  • the device 10 receives the position/movement information of the vessel obtained from the RGB image through other data acquired through the sensor module 100 . can be corrected.
  • the control module 200 may correct the distance/speed of the vessel acquired on the RGB image based on the lidar image acquired through the sensor module 100 .
  • the device 10 may obtain the corrected vessel distance/velocity based on the vessel distance/velocity acquired on the RGB image and the vessel distance/velocity acquired on the lidar image.
  • the device 10 may obtain the corrected distance/speed of the vessel by comparing the distance/speed of the vessel acquired on the RGB image and the distance/speed of the vessel acquired on the lidar image.
  • the correction of the position/movement information of the monitored vessel may be performed using not only the lidar image but also various types of sensor data such as conventionally used radar and ultrasonic detectors.
  • 33 is a flowchart for correcting position/movement information based on a lidar image according to an exemplary embodiment.
  • the image-based monitoring may include performing, by the control module 200, obtaining a lidar image (S252) and correcting position/movement information based on the lidar image (S254). .
  • the device 10 may obtain a lidar image from the image generating unit (S252).
  • control module 200 may obtain a LiDAR image synchronized with the RGB image in real time from the sensor module 100 .
  • the lidar image may be a three-dimensional image as well as a two-dimensional image.
  • the device 10 may correct the position/movement information of the vessel obtained from the RGB image based on the obtained lidar image (S254).
  • the step of correcting the position/movement information based on the lidar image may include obtaining, by the control module 200 , the position/movement information of the vessel from the lidar image.
  • the control module 200 may acquire the distance/speed between the vessel and the quay wall from the lidar image or the distance/speed between the vessel and another vessel.
  • the control module 200 may acquire the bow distance/speed of the ship, the stern distance/speed of the ship, the distance between ships, the relative speed between the ship and the adjacent ship, and the like from the lidar image.
  • the control module 200 may acquire the position/movement information of the vessel based on the RGB image and the LiDAR image synchronized in real time.
  • the control module 200 may acquire berthing-related information and/or collision-related information of the vessel based on the RGB image and the LiDAR image synchronized in real time.
  • the control module 200 samples the points that are perpendicular to the berth plane at regular intervals in the RGB image and the LiDAR image synchronized in real time to obtain the position/movement information of the vessel based on the estimated side surface of the vessel. can be obtained
  • the control module 200 may determine both ends of the side surface of the ship estimated through sampling from the lidar image as the bow/stern of the ship.
  • the control module 200 may acquire the distance/speed between the ship and the quay wall based on the bow/stern determined from the lidar image. For example, the control module 200 may acquire the bow distance, the stern distance, the bow speed, the stern speed, etc. of the ship based on the bow/stern determined from the lidar image.
  • control module 200 may acquire the distance/velocity between the vessel and the other vessel based on the bow/stern determined from the lidar image. For example, the control module 200 may acquire a ship-to-ship distance of a ship, a relative speed between a ship and an adjacent ship, etc. based on the bow/stern determined from the lidar image.
  • control module 200 may determine the fore/stern of the ship in a position other than both ends of the side of the ship in the lidar image, and the control module 200 selects one point in the lidar image. Therefore, instead of acquiring the position/movement information of the vessel, it is okay to select the area itself to acquire the position/movement information of the vessel. Also, this is not necessarily the case, and the acquisition of the position/movement information of the vessel in the lidar image may be performed in various ways.
  • the control module 200 uses the position/movement information obtained from the RGB image and the position/movement information of the vessel obtained from the lidar image. This may include correction.
  • the control module 200 may correct the berthing-related information and/or collision-related information obtained from the RGB image by using the vessel's berthing-related information and/or collision-related information obtained from the lidar image.
  • the RGB image and the lidar image may be images obtained by capturing the same area and may be images mapped to each other.
  • the RGB image and LiDAR image can be synchronized in real time.
  • the control module 200 controls the distance/speed of the vessel in the corresponding frame.
  • the velocity can be determined based on distance/velocity obtained from the lidar image. For example, when the difference between the bow and stern distances obtained from the RGB image and the bow and stern distances obtained from the lidar image is equal to or greater than a preset value, the control module 200 may control the vessel's bow and stern distances. can be determined as the bow distance and the stern distance obtained from the lidar image.
  • the control module 200 calculates the distance between ships from the LiDAR image. It can be determined by the obtained intership distance. Of course, it is also possible for the control module 200 to determine the speed of the vessel as the speed obtained from the lidar image rather than the speed obtained from the RGB image.
  • control module 200 may determine the distance/speed of the ship as a value in which the distance/speed obtained from the RGB image in the corresponding frame and the distance/speed obtained from the lidar image are reflected at a certain ratio. have.
  • control module 200 may determine the bow distance and the stern distance of the ship as values in which the bow and stern distances obtained from the RGB image and the bow and stern distances obtained from the lidar image are reflected at a certain ratio. .
  • control module 200 may determine the intership distance of the vessel as a value in which the intership distance obtained from the RGB image and the intership distance obtained from the lidar image are reflected at a certain ratio.
  • control module 200 may determine the speed of the vessel as a value in which the speed obtained from the RGB image and the speed obtained from the lidar image are reflected in a certain ratio.
  • the position/movement information correction based on the lidar image does not need to be limited to being performed by the above-described method, and the control module 200 determines the position/movement information of the vessel obtained from the RGB image based on the lidar image. It can be corrected in a number of ways.
  • Device 10 may calculate distances to other vessels and/or quay walls for vessel and/or harbor monitoring. For example, the device 10 may generate a feature point of a vessel for distance calculation from sensor data, and calculate a distance to another vessel and/or a quay wall based on the generated feature point.
  • a characteristic point of a ship is a concept including a specific point in an image, a specific lidar point of lidar data, and the like. It may include a point corresponding to , a point corresponding to a portion where the ship and the sea level are in contact.
  • 34 is a flowchart of a method for calculating a distance based on a feature point of a ship according to an exemplary embodiment.
  • a method for calculating a distance based on a feature point of a ship includes the steps of acquiring sensor data ( S1000 ), generating the feature point of the ship ( S2000 ), and calculating a distance based on the feature point. It may include a step (S2000) of doing.
  • the device 10 may acquire sensor data (S1000).
  • the device 10 may obtain an image from a camera.
  • the device 10 may acquire an image from a camera installed on a berth toward the sea.
  • the device 10 may acquire an image of the sea, and if there is a ship, it may also acquire an image of the ship.
  • the device 10 may obtain lidar data from a lidar sensor.
  • the device 10 may acquire lidar data from a lidar sensor installed on a berth toward the sea.
  • the device 10 may acquire lidar data for the sea, and when the vessel enters the berth, it may also acquire lidar data for the vessel.
  • the lidar sensor may acquire lidar data for an area corresponding to an area captured by a camera that acquires an image.
  • the lidar sensor may have a field of view that at least partially overlaps with a field of view of a camera acquiring the image.
  • the device 10 may acquire sensor data for the same area from the lidar and the camera installed around the berth.
  • the lidar and the camera may be installed on the pier toward or looking at the berth to acquire data on an area where the berth is located.
  • the viewing angle of the lidar may at least partially overlap with the viewing angle of the camera.
  • the device 10 may acquire an image of the sea, and if there is a ship, it may also acquire an image of the ship.
  • the device 10 may use the acquired camera images to calculate the distance between the vessel and another vessel and/or the distance between the vessel and the quay wall.
  • the device 10 may acquire lidar data for the sea, and when the vessel enters the berth, it may also acquire lidar data for the vessel.
  • the lidar data may include a plurality of lidar points captured by lidar sensors.
  • the lidar data may include a plurality of lidar points for each vertical or horizontal channel.
  • the device 10 may use the obtained lidar data to calculate a distance between a vessel and another vessel and/or a distance between a vessel and a quay wall.
  • the device 10 may generate a feature point of the vessel ( S2000 ).
  • the device 10 may use the acquired sensor data to generate a feature point of a ship used to calculate a distance between the ship and another object.
  • the device 10 may generate a feature point of the vessel by using at least one of a camera installed on a berth toward the sea and sensor data obtained from a lidar sensor.
  • the device 10 may detect an area of the vessel from the camera image and generate a feature point of the vessel based on the detected area of the vessel.
  • the device 10 may detect lidar points related to a lidar beam reflected from a vessel from lidar data, and generate a characteristic point of the vessel based on the detected lidar points.
  • the device 10 is not limited to the above description, and may generate the feature points of the vessel in other ways, such as fusing camera images and lidar data. Various embodiments of the generation of the feature point of the vessel will be described later.
  • the device 10 may calculate a distance between the ship and another object based on the feature points of the ship ( S3000 ).
  • the device 10 may calculate a distance between a vessel and another vessel and/or a distance between a vessel and a quay wall based on the vessel's characteristic points. For example, the device 10 may calculate a distance between a vessel and another vessel based on a position of a point corresponding to at least one of a bow and a stern of the vessel. For another example, the device 10 may calculate the distance between the ship and the quay wall based on the location of the point corresponding to the portion where the ship and the sea level come into contact. For example, the distance calculation using the camera image may be calculated based on the distance between image pixels, and the distance calculation using the lidar data may be calculated based on the coordinate values of the lidar data. It can be applied, so the details will be omitted.
  • the device 10 may calculate a distance between the ship and another object by using the feature points of the ship obtained from the camera image. For example, the device 10 may calculate the distance between the ship and another ship or a quay wall using the feature points of the ship obtained from the camera image.
  • the apparatus 10 may generate a feature point of a ship using an image in which a viewpoint is converted.
  • the device 10 may generate feature points of the vessel using a transformed image in which the vessel zero in the image is projected onto a reference plane.
  • the method for generating feature points of a ship using an image in which a viewpoint is converted according to an embodiment includes detecting a ship area from a camera image (S2010), and generating a converted image by projecting the ship area on a reference plane It may include the step (S2020) and the step (S2030) of obtaining the feature point of the vessel from the converted image.
  • the device 10 may detect the vessel area from the camera image (S2010).
  • the device 10 may detect a region corresponding to a vessel in the image by using an artificial neural network.
  • the device 10 may generate a segmented image from an image using an artificial neural network, and detect an area in which a pixel labeled with object information indicating a ship of an object type is located as an area corresponding to the ship.
  • the apparatus 10 is not limited to the above description, and may detect an area corresponding to the vessel in the image in another method, such as determining an area corresponding to the vessel through image detection.
  • the apparatus 10 may generate a converted image by projecting the vessel area on the reference plane ( S2020 ).
  • the device 10 may generate a transformed image in which the area of the vessel is projected as a reference plane at the deck level of the vessel.
  • the device 10 generates a segmentation image from the image using an artificial neural network, and based on the area in which the pixel is located, the object type is the deck of the ship or the object information representing the side of the ship. It is possible to obtain the position of the part where the deck is in contact, and project the area of the ship to the obtained position of the deck as a reference plane.
  • the device 10 may acquire the position of the deck of the vessel in the image using lidar data matched to the camera image, and may project the area of the vessel to the reference plane based on the acquired position of the deck. .
  • the device 10 may generate a converted image in which the sea level is a reference plane and a region of a ship is projected.
  • the device 10 generates a segmented image from the image using an artificial neural network, and based on the regions in which pixels labeled with object information indicating a ship or sea of an object type are located, the position of the portion where the ship and the sea level come into contact can be obtained, and the area of the vessel can be projected on the reference plane based on the obtained sea level position.
  • the device 10 is not limited to the above description, and may generate the converted image in other ways, such as acquiring the position of the deck through image detection or projecting the vessel area using the quay wall as a reference plane.
  • the device 10 may acquire the feature point of the vessel from the converted image (S2030).
  • the device 10 may determine a feature point of a vessel for calculating the distance between the vessels in the transformed image. For example, the device 10 obtains a bow and/or a point corresponding to the bow of the ship from the converted image in which the area of the ship is projected as a reference plane at the deck height of the ship, and uses this to determine the distance between the ships can be calculated
  • the device 10 may determine a feature point of the vessel for calculating the distance between the vessel and the quay wall in the transformed image.
  • the device 10 may obtain a bow and/or a point corresponding to the bow of the ship from the converted image in which the area of the ship is projected with the sea level as a reference plane, and calculate the distance between the ship and the quay wall using this. .
  • the device 10 is not limited to the above description, and may generate the converted image in other ways, such as acquiring the position of the deck through image detection or projecting the vessel area using the quay wall as a reference plane.
  • a distortion phenomenon may occur due to the projection of the vessel area to the reference plane, the distortion may become more severe as the distance from the reference plane increases, and distortion may not occur in the reference plane.
  • the apparatus 10 generates a converted image in which the area of the ship is projected to the reference plane 401 at sea level, and uses this to generate a feature point of the ship for calculating the distance. can be decided
  • the device 10 may determine an arbitrary point of a line where the ship and the sea level in the converted image meet as a feature point of the ship.
  • the device 10 may calculate the distance between the bow of the vessel and the quay wall or another vessel by using an arbitrary point 403 on the bow side of the line where the vessel and the sea level meet in the converted image.
  • the device 10 may calculate the distance between the stern of the vessel and the quay wall or another vessel using an arbitrary point 402 on the stern side of the line where the vessel and the sea level meet in the converted image.
  • the device 10 may determine points located at the ends of the bow and stern of the ship in the converted image as the feature points of the ship. As an example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the point 405 of the bow end of the vessel in the transformed image. As another example, the device 10 may use the point 406 of the stern end of the vessel in the transformed image to calculate the distance between the vessel and the quay wall or other vessel.
  • the apparatus 10 is not limited to the above description, and the distance between the ship and another ship or quay wall is calculated by using the feature point of the ship extracted by polygonalizing the area of the ship in the converted image, such as calculating the distance in other ways. It is free to calculate
  • the apparatus 10 generates a converted image in which the area of the ship is projected to the reference plane 411 at the deck height of the ship, and uses the converted image to calculate the distance. It is possible to determine the characteristic points of
  • the device 10 may determine an arbitrary point of a line where the side of the ship and the deck area of the ship in the converted image are in contact with the feature point of the ship.
  • the device 10 may calculate the distance between the bow of the vessel and the quay wall or another vessel using an arbitrary point 413 on the bow side of the line where the side of the vessel and the deck area of the vessel in the converted image meet. have.
  • the device 10 calculates the distance between the stern of the vessel and the quay wall or another vessel using an arbitrary point 414 on the stern side of the line where the side of the vessel and the deck area of the vessel in the converted image meet. can
  • the device 10 may determine points located at the ends of the bow and stern of the ship in the converted image as the feature points of the ship. As an example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the point 415 of the bow end of the vessel in the transformed image. As another example, the device 10 may use the point 416 of the stern end of the vessel in the transformed image to calculate the distance between the vessel and the quay wall or other vessel.
  • the apparatus 10 is not limited to the above description, and the distance between the ship and another ship or quay wall is calculated by using the feature point of the ship extracted by polygonalizing the area of the ship in the converted image, such as calculating the distance in other ways. It is free to calculate
  • the device 10 may calculate the distance between the ships by projecting the area of each ship in the image onto different reference planes.
  • the device 10 may detect the first vessel region 421 and the second vessel region 422 from the camera image.
  • the device 10 may detect the first vessel region 421 and the second vessel region 422 using an artificial neural network. For this, since the above description may be applied, a detailed description thereof will be omitted.
  • the device 10 may project the detected first vessel region 421 and the second vessel region 422 onto each reference plane at the deck height of each vessel to generate a transformed image.
  • the device 10 may project the detected first vessel area 421 onto a first reference plane 425 at the deck level of the first vessel to generate a first transformed image 423
  • the device 10 may project the detected second vessel region 422 onto a second reference plane 426 at the deck height of the second vessel to generate a second transformed image 424 .
  • a detailed description thereof will be omitted.
  • the device 10 may obtain a feature point of each vessel by using each of the transformed images.
  • the device 10 may determine a feature point by using a point at a position corresponding to the bow and/or stern of the first ship in order to calculate the distance between the first ship and the second ship in the first converted image 423 .
  • the device 10 may acquire one of the points at both ends of the first vessel region in the first transformed image 423 as a feature point.
  • a point 427 located closer to the second vessel may be determined as a characteristic point of the vessel.
  • the device 10 may determine a feature point by using a point at a position corresponding to the bow and/or stern of the second vessel in order to calculate the distance between the second vessel and the first vessel in the second transformed image 424 .
  • the device 10 may acquire one of the points at both ends of the second vessel region in the second transformed image 424 as a feature point.
  • a point 428 located near the first ship among points at both ends of the second ship area may be determined as a feature point of the ship. For this, since the above description may be applied, a detailed description thereof will be omitted.
  • the device 10 may calculate a distance between the first vessel and the second vessel using feature points of the first vessel and the second vessel obtained from each of the transformed images 423 and 424 . For example, the device 10 may calculate a distance between the first vessel and the second vessel based on the number of pixels between the feature points 427 , 428 of the first vessel and the second vessel. For another example, the device 10 may calculate a distance between the first vessel and the second vessel based on lidar data matched to the feature points 427 and 428 of the first vessel and the second vessel. For this, since the above description may be applied, a detailed description thereof will be omitted.
  • the device 10 may calculate a distance between the ship and another object by using the feature points of the ship obtained from the lidar data.
  • the device 10 may calculate the distance between the ship and another ship or a quay wall using the feature points of the ship obtained from the lidar data.
  • the device 10 may calculate the distance between the vessel and another vessel by using a lidar point associated with a lidar beam reflected from both ends of the vessel.
  • the device 10 may calculate the distance between the ship and the quay wall using a lidar point related to a lidar beam reflected from a portion where the vessel and the sea level meet.
  • lidar sensor has low performance or there are restrictions on the place where the lidar sensor is installed, insufficient lidar data may be obtained for monitoring a ship or a port. Therefore, it may be necessary to supplement insufficient lidar data for monitoring of ships or ports.
  • the device 10 may estimate new lidar data from the acquired lidar data. For example, the device 10 may newly estimate a lidar point associated with a ship by using the acquired lidar data. As an example, the apparatus 10 may estimate the lidar point associated with the vessel by interpolating or extrapolating lidar points associated with the lidar beam reflected from the vessel.
  • the lidar point estimation method includes the steps of matching a camera image and lidar data (S1010) and estimating a lidar point using the matched camera image and lidar data (S1020). ) may be included.
  • the device 10 may match the camera image and lidar data (S1010).
  • the device 10 may match the camera image and the lidar data using information for matching.
  • the device 10 may match the camera image and the lidar data by matching the coordinate system on the camera image with the coordinate system on the lidar data. That is, the coordinate system of the camera image and the lidar data may be converted to each other.
  • the device 10 may match the camera image and lidar data in consideration of the installation position of the camera, the installation angle of the camera, the installation position of the lidar sensor, the installation angle of the lidar sensor, and the like.
  • the device 10 may realign the camera image and lidar data by reflecting the calculated sea level height.
  • the device 10 may use the calculated sea level height as a data conversion variable to realign the camera image and the lidar data.
  • FIG. 40 is an example of matching a camera image and lidar data according to an embodiment. Referring to FIG. 40( a ), it can be seen that the lidar data acquired from the lidar sensor scanning the same area as the image acquired from the camera installed on the berth are matched.
  • the device 10 may match the segmented image with the lidar data.
  • the device 10 may generate a segmentation image from an image obtained from a camera using an artificial neural network, and may match the generated segmentation image with lidar data. Referring to FIG. 40(b) , it can be seen that the lidar data acquired from the lidar sensor scanning the same area as the image acquired from the camera installed on the berth are matched.
  • the matching of the camera image and the lidar data is not limited to the above description, and may be implemented in other ways, such as matching the image detected using an artificial neural network with the lidar data.
  • the device 10 may estimate the lidar point using the matched camera image and lidar data (S1020).
  • the device 10 may generate an estimated lidar point by using an image matched with the lidar data among a plurality of lidar points of the lidar data. For example, the device 10 may newly generate an estimated LiDAR point related to the sea by using a lidar point matched to an area corresponding to the sea in the image. As another example, the device 10 may use the lidar point matched to the region of the vessel in the image to newly generate an estimated lidar point associated with the vessel.
  • FIG. 41 is a flowchart of a method for generating feature points of a ship in consideration of an estimated lidar point according to an embodiment.
  • detecting a ship area from a camera image ( S2110 ), and a lidar beam reflected from the ship based on the detected ship area selecting LiDAR points related to (S2120), estimating a LiDAR point related to the vessel using the selected LiDAR points (S2130), and generating a feature point of the vessel in consideration of the estimated LiDAR point (S2130) S2140) may be included.
  • the device 10 may detect the vessel area from the camera image (S2110).
  • the camera image and lidar data may correspond to each other and may be matched.
  • the above-described contents may be applied, and a detailed description thereof will be omitted.
  • the device 10 may select lidar points related to the lidar beam reflected from the vessel based on the detected vessel area ( S2120 ).
  • the device 10 may use an area corresponding to the vessel in the detected image to select lidar points associated with the lidar beam reflected from the vessel. For example, the device 10 may select lidar points associated with a lidar beam reflected from the vessel in consideration of pixel positions of pixels included in an area corresponding to the vessel in the image.
  • the device 10 may select lidar points 431 related to a lidar beam reflected from a ship from among lidar points matched with the image.
  • the apparatus 10 may select LiDAR points matched to pixels included in an area corresponding to the vessel in the image among the plurality of lidar points to the lidar points 431 associated with the lidar beam reflected from the vessel. ) can be selected.
  • the apparatus 10 may select lidar points 432 related to a lidar beam reflected from a ship from among lidar points matched with the segmentation image.
  • the apparatus 10 it is also possible for the apparatus 10 to select LiDAR points related to the LIDAR beam reflected from the vessel among LIDAR points that are matched with the transformed image in which the vessel area is projected on the reference plane.
  • the selection of lidar points related to the lidar beam reflected from the vessel is not limited to the above description and may be implemented in other ways.
  • the device 10 may select lidar points associated with a lidar beam reflected from a vessel using only lidar data.
  • the apparatus 10 may select lidar points related to a lidar beam reflected from a ship in consideration of a distribution, number, and the like of lidar data.
  • the apparatus 10 may estimate a LiDAR point related to a ship using the selected LiDAR points (S2130).
  • device 10 may estimate a new lidar point by interpolating or extrapolating lidar points associated with a lidar beam reflected from a vessel.
  • the apparatus 10 may estimate a new lidar point by interpolating or extrapolating each lidar point in consideration of a relative position using coordinates of lidar points related to a lidar beam reflected from a ship.
  • the device 10 may estimate a new lidar point using a transformed image in which the vessel area is projected as a reference plane.
  • the apparatus 10 may estimate a new LiDAR point using a converted image in which a ship area is projected with the sea level as a reference plane.
  • the device 10 may estimate a new lidar point using a converted image in which the ship area is projected with the area at the deck height of the ship as a reference plane.
  • 43 and 44 are diagrams for explaining generation of an estimated lidar point using a lidar point matched to a vessel area projected on a reference plane according to an embodiment.
  • the converted image in which the ship area is projected with the sea level 441 as the reference plane and the lidar point may be matched with each other. Since the lidar points 442 located in the area where the ship and the sea level contact each other have the same height value, the estimated lidar points 443 are interpolated and/or extrapolated by interpolating and/or extrapolating the lidar points 442 located in the same area. , 444, 445). For example, the apparatus 10 may generate the estimated lidar point 443 by interpolating lidar points 442 located in the region 441 where the ship and the sea level meet.
  • the apparatus 10 may generate the estimated lidar points 444 and 445 by extrapolating the lidar points 442 located in the region 441 where the ship and the sea level meet.
  • the device 10 may estimate the lidar points in consideration of the shape of the vessel in the image.
  • the converted image in which the ship area is projected with respect to the area 451 at the deck height of the ship as a reference plane and the lidar point may be matched with each other. Since the lidar points 452 located in the deck height region of the vessel have the same height value, the estimated lidar points 453, 453, by interpolating and/or extrapolating the lidar points 452 located in the same region. 454, 455).
  • the device 10 may interpolate the lidar points located in the deck-height region 451 of the vessel to generate an estimated lidar point 453 .
  • device 10 may extrapolate lidar points located in the ship's deck-height region 451 to generate estimated lidar points 454 , 455 .
  • the device 10 may estimate the lidar points in consideration of the shape of the vessel in the image.
  • the apparatus 10 is not limited to the above description, and the estimated lidar points may be generated by other methods, such as estimating lidar points located in regions that do not have the same height value.
  • the device 10 may generate a feature point of the vessel in consideration of the estimated lidar point ( S2140 ).
  • the device 10 may select a characteristic point of a vessel for calculating a distance between the vessels from among existing lidar points associated with a lidar beam reflected from the vessel and generated estimated lidar points. have. That is, the device 10 is an estimated lidar generated by extrapolating and/or interpolating the existing lidar points and the existing lidar points matched with the transformed image in which the region of the vessel is projected as a reference plane at the deck height of the vessel. Among the points, the bow of the vessel and/or the lidar point corresponding to the bow may be obtained, and the distance between the vessels may be calculated using the obtained.
  • the device 10 determines the characteristic point of the vessel for calculating the distance between the vessel and the quay wall among the existing lidar points related to the lidar beam reflected from the vessel and the generated estimated lidar points.
  • the device 10 sets the existing lidar points matched with the transformed image in which the region of the vessel is projected with the sea level as the reference plane, and the vessel among the estimated lidar points generated by extrapolating and/or interpolating the existing lidar points. It is possible to obtain points corresponding to the bow and/or bow of , and use them to calculate the distance between the vessel and the quay wall.
  • the device 10 may determine any LiDAR point matched to a line where the ship and the sea level in the converted image meet as the feature point of the ship. For example, the device 10 may calculate the distance between the bow of the ship and the quay wall or another ship by using any lidar points matched to the bow side of the line where the ship and the sea level in the converted image meet. As another example, the device 10 may calculate the distance between the stern of the vessel and the quay wall or another vessel by using any lidar point matched to the stern side of the line where the vessel and the sea level meet in the converted image.
  • the lidar point may include an existing lidar point and an estimated lidar point.
  • the device 10 may determine the lidar points matched to the ends of the bow and stern of the ship in the converted image as the feature points of the ship. As an example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the lidar point matched to the bow end of the vessel in the converted image. As another example, the device 10 may calculate the distance between the vessel and the quay wall or other vessel by using the lidar point matched to the stern end of the vessel in the converted image.
  • the lidar point may include an existing lidar point and an estimated lidar point.
  • the device 10 is not limited to the description described above, and may use other methods, such as polygonalizing an area of the vessel in the converted image, and calculating the distance between the vessel and another vessel or quay wall using the lidar point matched thereto. It is free to calculate the distance.
  • the device 10 may obtain the height (deck height) of the vessel by using the lidar data, and use the acquired deck height of the vessel to generate a feature point of the vessel.
  • the device 10 cannot determine which plane to use the camera image as the reference plane, and sets the wrong plane as the reference plane. If you do, the converted image may be severely distorted. Accordingly, when the device 10 according to an embodiment does not accurately acquire the deck height of the vessel, it may acquire the feature point of the vessel using converted images in which the region of the vessel is projected onto an arbitrary reference plane.
  • the apparatus 10 may acquire the feature point of the ship by using the converted images in which the area of the ship is projected on an arbitrary reference plane. For example, the apparatus 10 may align the transformed images projected on an arbitrary reference plane having different regions of the vessel in the image using the lidar data, and obtain the characteristic points of the vessel using the LIDAR data.
  • the device 10 may use the segmented image.
  • the method for acquiring feature points of a ship using images in which the ship area is projected on a plurality of reference planes includes detecting a ship area from a camera image (S2210), Projecting to a reference plane of (S2220), arranging images projected on arbitrary reference planes different from each other (S2230), and generating feature points of the vessel using the aligned images and lidar points (S2240) ) may be included.
  • the device 10 may detect the vessel area from the camera image (S2210).
  • the device 10 may generate a segmented image by segmenting a camera image using an artificial neural network. Specifically, the device 10 may generate a segmentation image that detects the deck area and the side area of the vessel. For example, the device 10 may generate a segmented image using panoptic segmentation.
  • the device 10 may project the vessel area onto arbitrary reference planes different from each other ( S2220 ).
  • the device 10 may project the segmented image for detecting the deck area and the side area of the ship on different reference planes using an artificial neural network. That is, the apparatus 10 may generate at least one of the first image and the second image by projecting the segmented image onto different first and second reference planes.
  • the apparatus 10 may align the images projected on arbitrary reference planes that are different from each other ( S2230 ).
  • the device 10 may align the images projected on any reference planes different from each other by using the lidar data.
  • the device 10 may align at least one of the images projected onto any different reference planes based on the positions of the lidar points relative to the lidar beam reflected from the vessel.
  • the projected images may be a projected image of a segmentation image for detecting the deck area and the side area of the ship using an artificial neural network.
  • 46 and 47 are diagrams for explaining the alignment of images projected on mutually different arbitrary reference planes according to an embodiment.
  • the device 10 may align the LIDAR point with images projected on different reference planes while changing the reference plane. As the reference plane is changed, the relative position of the lidar point that is matched with the projected images may also change. The device 10 may align the images and the lidar points so that the lidar points are located in a specific area of the images projected on any different reference planes.
  • a first image in which a segmentation image for detecting the deck area 461 and the side area 462 of the ship is projected onto an arbitrary first reference plane can be seen.
  • the device 10 may align the lidar points and the first image such that the lidar points associated with the lidar beam reflected from the vessel are located in a specific area in the first image. For example, when the device 10 aligns the first image with lidar points associated with a lidar beam reflected from the vessel, in the first image the lidar points associated with the lidar beam reflected from the vessel are segmented into the segmented image. It is possible to align the lidar points and the first image to be located in the projected area corresponding to the side area 462 of the vessel of . As an example, the device 10 may align the lidar points and the first image so that some of the lidar points related to the lidar beam reflected from the vessel are located in a region where the vessel and the sea level meet in the first image. .
  • the segmentation image for detecting the deck area 463 and the side area 464 of the ship can see a second image projected on an arbitrary second reference plane different from the first reference plane.
  • the device 10 may align the lidar points and the second image such that the lidar points associated with the lidar beam reflected from the vessel are located in a specific area in the second image. For example, when the device 10 aligns the second image with lidar points associated with a lidar beam reflected from the vessel, in the second image the lidar points associated with the lidar beam reflected from the vessel are segmented into the segmented image. It is possible to align the lidar points and the second image so that they are located in the projected area corresponding to the deck area 463 of the ship. As an example, the device 10 may align the lidar points and the second image such that some of the lidar points associated with the lidar beam reflected from the vessel are located in the region where the deck of the vessel begins in the second image. have.
  • the device 10 may align the lidar points with the first image and the second image based on the relative positions of the lidar points in the first image and the second image.
  • the device 10 may be configured such that in the first image the lidar points associated with the lidar beam reflected from the vessel are located below the projected region corresponding to the side region 462 of the vessel in the segmentation image, and in the second image may align the first image and the second image with the lidar points such that the lidar points associated with the lidar beam reflected from the vessel are located in the projected region corresponding to the deck region 463 of the vessel in the segmentation image.
  • the device 10 may generate a feature point of the vessel using the aligned images and the lidar points ( S2240 ).
  • the device 10 may use the aligned lidar points and the projected first and second images to calculate the distance to another object (eg, another vessel, quay wall, etc.) You can earn points.
  • another object eg, another vessel, quay wall, etc.
  • the device 10 may acquire the feature points of the vessel in consideration of the LIDAR points estimated using the aligned LIDAR points and the projected first and second images.
  • the device 10 extrapolates and/or interpolates the LiDAR points aligned with the projected first image and the second image to generate an estimated LiDAR point, the generated estimated LIDAR point and the existing LIDAR point can be used to acquire the ship's characteristic points.
  • the device 10 may interpolate the lidar points to generate an estimated lidar point 465 , and extrapolate the lidar points to generate estimated lidar points 466 and 467 . have.
  • the above description may be applied to the estimation of the lidar point or the acquisition of the characteristic point of the ship, and thus the detailed description will be omitted.
  • the device 10 may obtain the deck height of a ship in a variety of ways.
  • the device 10 may use the sensor data to obtain the deck height of the vessel.
  • the device 10 may obtain the deck height of the vessel using at least one of an image and lidar data.
  • the apparatus 10 may obtain the deck height of the ship by using the coordinates of the LiDAR points corresponding to the portion in which the distance values of the LiDAR points related to the LIDAR beam reflected from the side of the vessel change rapidly.
  • the device 10 detects the side and the deck area of the ship using an artificial neural network that can detect the side and the deck area of the ship, and a lidar corresponding to the portion where the side area and the deck area of the ship come into contact with each other.
  • the deck height of the ship can be obtained using the coordinates of the points.
  • the device 10 may obtain the deck height of the vessel through the communication module 300 .
  • the device 10 may receive AIS information of the vessel through the communication module 300 , and may obtain the deck height of the vessel by using the received AIS information.
  • the device 10 may obtain the deck height of the ship by receiving the deck height of the ship input from the external device through the communication module 300 .
  • the acquisition of the deck height of the ship is not limited to the above description and may be implemented in other ways, such as being obtained in another way.
  • the method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium.
  • the computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.
  • the program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software.
  • Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks.
  • - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.
  • Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.
  • the hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Abstract

La présente invention concerne un procédé par lequel un moyen informatique surveille un port, et selon un mode de réalisation, un procédé de surveillance d'un port, comprenant les étapes consistant à : acquérir une image de port comprenant la mer et un navire ; extraire, à partir de l'image du port, une paire de premiers points correspondant aux deux parties d'extrémité d'une surface inférieure sur laquelle le navire vient en contact avec le niveau de la mer, et une paire de seconds points correspondant à la proue et à la poupe du navire ; acquérir des informations relatives à l'accostage concernant le navire sur la base de la paire de premiers points ; et acquérir des informations relatives à une collision concernant le navire, y compris la distance entre le navire et un autre navire sur la base de la paire de seconds points.
PCT/KR2021/000036 2020-01-09 2021-01-04 Dispositif et procédé de surveillance de navire et de port WO2021141338A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR1020200003190A KR102265980B1 (ko) 2020-01-09 2020-01-09 선박 및 항만 모니터링 장치 및 방법
KR10-2020-0003190 2020-01-09
KR10-2020-0139727 2020-10-26
KR1020200139727A KR102535115B1 (ko) 2020-10-26 2020-10-26 선박 및 항만 모니터링 장치 및 방법

Publications (1)

Publication Number Publication Date
WO2021141338A1 true WO2021141338A1 (fr) 2021-07-15

Family

ID=76788813

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/000036 WO2021141338A1 (fr) 2020-01-09 2021-01-04 Dispositif et procédé de surveillance de navire et de port

Country Status (1)

Country Link
WO (1) WO2021141338A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113744570A (zh) * 2021-11-03 2021-12-03 武汉理工大学 一种桥区水域船舶防撞预警方法及装置
US20230175832A1 (en) * 2021-12-08 2023-06-08 Ship And Ocean Industries R&D Center Automatic berthing image ranging system for vessels and operation method thereof
CN116634631A (zh) * 2023-06-09 2023-08-22 江苏唐城霓虹数码科技有限公司 一种照明监控的港口高杆灯双升降智能控制方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110059206A (ko) * 2009-11-27 2011-06-02 김영복 터그보트를 이용한 접안선박 유도시스템
KR101452994B1 (ko) * 2013-05-06 2014-10-22 한국생산기술연구원 선박의 안전접안 시스템
KR20140126998A (ko) * 2013-04-24 2014-11-03 주식회사 씨큐로 갑문 충돌방지 시스템
KR20170133010A (ko) * 2016-05-25 2017-12-05 (주)주인정보시스템 카메라 영상을 이용한 선박의 안전접안 시스템 및 그 방법
KR102005559B1 (ko) * 2018-09-04 2019-08-07 씨드로닉스(주) 이미지 세그멘테이션을 이용한 주변 환경 감지 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110059206A (ko) * 2009-11-27 2011-06-02 김영복 터그보트를 이용한 접안선박 유도시스템
KR20140126998A (ko) * 2013-04-24 2014-11-03 주식회사 씨큐로 갑문 충돌방지 시스템
KR101452994B1 (ko) * 2013-05-06 2014-10-22 한국생산기술연구원 선박의 안전접안 시스템
KR20170133010A (ko) * 2016-05-25 2017-12-05 (주)주인정보시스템 카메라 영상을 이용한 선박의 안전접안 시스템 및 그 방법
KR102005559B1 (ko) * 2018-09-04 2019-08-07 씨드로닉스(주) 이미지 세그멘테이션을 이용한 주변 환경 감지 방법

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113744570A (zh) * 2021-11-03 2021-12-03 武汉理工大学 一种桥区水域船舶防撞预警方法及装置
US20230175832A1 (en) * 2021-12-08 2023-06-08 Ship And Ocean Industries R&D Center Automatic berthing image ranging system for vessels and operation method thereof
US11920912B2 (en) * 2021-12-08 2024-03-05 Ship And Ocean Industries R&Dcenter Automatic berthing image ranging system for vessels and operation method thereof
CN116634631A (zh) * 2023-06-09 2023-08-22 江苏唐城霓虹数码科技有限公司 一种照明监控的港口高杆灯双升降智能控制方法及系统
CN116634631B (zh) * 2023-06-09 2024-01-12 江苏唐城霓虹数码科技有限公司 一种照明监控的港口高杆灯双升降智能控制方法及系统

Similar Documents

Publication Publication Date Title
WO2021141338A1 (fr) Dispositif et procédé de surveillance de navire et de port
WO2020071839A1 (fr) Dispositif et procédé de surveillance de port et de navires
WO2021141339A1 (fr) Procédé et dispositif de surveillance de port et de navire en considération du niveau de la mer
WO2020050498A1 (fr) Procédé et dispositif destinés à détecter un milieu environnant à l'aide d'une segmentation d'image
WO2020050499A1 (fr) Procédé d'acquisition d'informations d'objet et appareil pour le mettre en œuvre
WO2017188706A1 (fr) Robot mobile et procédé de commande de robot mobile
WO2020085540A1 (fr) Procédé et appareil de réalité augmentée pour l'aide à la conduite
WO2015152692A1 (fr) Appareil et procédé de génération d'image périphérique de véhicule
WO2017219588A1 (fr) Procédé et système de détection d'environnement basés sur un véhicule aérien sans pilote
WO2015194867A1 (fr) Dispositif de reconnaissance de position de robot mobile utilisant le suivi direct, et son procédé
WO2020141657A1 (fr) Dispositif de mesure corporelle et procédé de commande associé
WO2018062647A1 (fr) Appareil de génération de métadonnées normalisées, appareil de détection d'occlusion d'objet et procédés associés
WO2016182275A1 (fr) Appareil de conduite autonome et véhicule le comprenant
WO2018139879A1 (fr) Procédé et appareil de détermination d'informations multimédia stéréoscopiques
WO2019054636A1 (fr) Appareil d'étalonnage de caméra de véhicule et procédé associé
WO2015194866A1 (fr) Dispositif et procédé permettant de reconnaître un emplacement d'un robot mobile au moyen d'un réajustage basé sur les bords
WO2015088289A1 (fr) Caméra stéréoscopique, dispositif auxiliaire de conduite de véhicule, et véhicule
WO2022050507A1 (fr) Procédé et système de surveillance d'un module de génération d'énergie photovoltaïque
WO2017188800A1 (fr) Robot mobile et son procédé de commande
WO2015194865A1 (fr) Dispositif et procede pour la reconnaissance d'emplacement de robot mobile au moyen d'appariement par correlation a base de recherche
WO2018212608A1 (fr) Système de marquage mobile, procédé de commande de dispositif de marquage mobile, et support d'enregistrement lisible par ordinateur
AU2020244635B2 (en) Mobile robot control method
WO2017007166A1 (fr) Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur
WO2016088960A1 (fr) Procédé et système de détection, dans un environnement nocturne, de danger dû à la présence de piéton, pour système avancé d'aide à la conduite
WO2019199112A1 (fr) Système et procédé de travail autonome et support d'enregistrement lisible par ordinateur

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21738504

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21738504

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21738504

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 25/01/2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21738504

Country of ref document: EP

Kind code of ref document: A1