KR20220145016A - Around-view system for floating objects - Google Patents

Abstract

The present invention relates to an around-view system of a floating object. The around-view system of a floating object according to one embodiment of the present invention is an around-view system of a floating object by using an artificial neural network performed by a computing means, which comprises the following steps of: obtaining a marine image including a navigable area and obstacles by a camera installed in a ship; inputting the marine image to an artificial neural network that performs image segmentation and outputting output data; obtaining second distance information of a second pixel based on a location of the second pixel in the marine image and position information of the camera; and updating the second output value by reflecting final distance information calculated based on the first distance information and the second distance information. The around-view system acquires a surrounding image and information of a floating object and informs an operator.

Description

Around-view system for floating objects

The present invention relates to an around-view system of a water vehicle, and more particularly, to an apparatus and method capable of providing images and information around the surface in real time during operation of a water vehicle and berthing in a port.

Many accidents occur in the operation of ships and berthing and berthing in ports, and it is known that the main cause of the accidents is human negligence. Here, the negligence of navigation is mainly caused by the fact that the situation around the ship or in the port cannot be accurately monitored through the naked eye. Currently, various types of obstacle sensors are used to supplement this problem, but there are still limitations. For example, in the case of ECDIS, there are limitations due to the inaccuracy of GPS, the update cycle of AIS, and non-AIS moving objects, and in the case of radar, there are limitations due to the existence and noise of the non-search area. As a result, in order to accurately detect an obstacle, it is still necessary to visually check the obstacle.

The problem to be solved by the present invention is to acquire the surrounding image and information of the floating body and inform the operator.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, the around-view system of a water-moving object according to an embodiment of the present invention is an around-view system of a water-moving object using an artificial neural network performed by a computing means, and can be operated by a camera installed on a ship. acquiring a maritime image including an area and an obstacle; outputting output data by inputting the resolution image to an artificial neural network that performs image segmentation; obtaining second distance information of the second pixel based on a position of the second pixel on the sea image and posture information of the camera; and updating the second output value by reflecting final distance information calculated based on the first distance information and the second distance information.

The details of other embodiments are included in the detailed description and drawings.

According to the around-view system of the floating body of the present invention, it acquires the surrounding image and information of the floating body and informs the operator to provide the surrounding image when berthing and berthing in the port, so that it is safely operated and berthing without being affected by the weather and the surrounding environment There is an effect that can be done.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram illustrating a configuration of an apparatus for controlling an around-view image according to an embodiment.
Figure 2 is a block diagram showing the sensing means included in the control system of the ship according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Hereinafter, the present invention will be described with reference to the drawings for explaining a method for detecting a ship's surroundings using an image according to embodiments of the present invention.

The around view 　 image is an image showing the surroundings of the ship, and may be called a top view or a bird view. Such 　around view 　 images may be generated based on images obtained from different directions.

When the Around View 　 image control device is a separate device, 　 At least some of the respective components of the Around View 　 video control device are not included in the Around View 　 Image control device, but external components included in the ship or other devices mounted on the ship can be These external components can be understood as constituting the 　around view 　 image control device by transmitting and receiving data through the interface unit of the 　around view 　 image control device.

1 is a block diagram illustrating the configuration of an apparatus for controlling an 'around view' image according to an embodiment.

Referring to FIG. 1 , the 　around view 　 image control apparatus may include a sensing unit and a processor. Also, the apparatus for controlling an 'around view' image according to an embodiment may further include at least one of an input unit, a communication unit, an interface unit, a memory, a monitoring unit, a display unit, an audio output unit, and a power supply unit. However, the components shown in FIG. 1 are not essential for implementing the 　around view 　 image control apparatus, so the 　around view 　 image control apparatus according to the embodiment has more or fewer components than the components listed above. can

When each configuration is described in detail, the input unit may detect a user's input. For example, the user inputs settings for 　around view 　image provided by the image control device through the input unit, or inputs an execution of turning on/off the power of the 　around view 　 image control device. can do.

The input unit includes a gesture input unit (eg, an optical sensor, etc.) for sensing a user's gesture, a touch input unit (eg, a touch sensor) for sensing a touch, a touch key, and a mechanical push button. key) and the like) and a microphone for detecting a voice input, and may detect a user input.

The communication unit may communicate with other ships, mobile terminals, servers, and the like.

In this case, the 　around view 　 image control device may receive at least one of navigation information, other vessel driving information, and traffic information through the communication unit. In addition, the 　around view 　 image control device may transmit information about the own vessel equipped with the 　 around view 　 image control device through the communication unit.

Specifically, the communication unit may receive at least one of location information, weather information, and sea condition information from the mobile terminal and/or the server.

The communication unit may receive the real-time location of the vessel. Specifically, the communication unit may acquire the position of the vessel including a Global Positioning System (GPS) module and/or a Wireless Fidelity (WiFi) module.

In addition, the communication unit may receive the traveling information of the other vessel from the other vessel and transmit the information of the intended vessel to the other vessel to share the traveling information between the vessels with each other. Here, the running information shared with each other may include at least one or more of information on the movement direction of the vessel and information on the location.

Such a communication unit may exchange data with another ship, a mobile terminal, or a server in a wireless manner.

In detail, the communication unit may perform wireless communication using a wireless data communication method. As the wireless data communication method, technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV-DO) (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution (LTE-A) Term Evolution-Advanced), etc.) can be used.

In addition, the communication unit may use wireless Internet technology, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance) as wireless Internet technologies. ), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-) Advanced) can be used.

In addition, the communication unit may use short range communication, for example, Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), infrared communication (Infrared Data Association; IrDA), UWB (Ultra Wideband), ZigBee , Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, and at least one of Wireless Universal Serial Bus (USB) technologies may be used to support short-range communication.

In addition, the 　 Around View 　 image control device may pair with a mobile terminal inside a ship using a short-distance communication method, and wirelessly exchange data with other ships or servers using a long-distance wireless communication module of the mobile terminal.

The interface unit may perform an interface between the inside and outside of the video control device, such as receiving data from an Electronic Control Unit (ECU) that controls the entire vessel or transmitting a signal processed or generated by the processor to the outside.

Specifically, the 　around view 　 image control apparatus may receive at least one of vessel driving information, navigation information, and sensing information through the interface unit.

In addition, the 　Around View 　image control device may transmit a control signal for executing 　Around View or information generated by the 　Around View 　image control device to the ship's ECU through the interface unit.

To this end, the interface unit may perform data communication with at least one of an ECU, an AVN (Audio Video Navigation) device, and a sensing unit inside the ship through wired communication or wireless communication.

Specifically, the interface unit may receive navigation information through data communication with an ECU, an AVN device, and/or a separate navigation device (not shown).

Also, the interface unit may receive sensing information from the ECU or the sensing unit.

2 is a block diagram of a ship sensor system according to an embodiment of the present invention.

Referring to FIG. 2 , it is common to use an obstacle detection sensor such as a radar, a lidar, and an ultrasonic detector and an automatic identification system (AIS) when operating a ship.

In addition, the camera can be used to detect obstacles. Referring to FIG. 1 , examples of the camera include, but are not limited to, a monocular camera, a binocular camera, an infrared camera, and a TOF camera.

Image segmentation may be used as one of the methods of detecting an obstacle using an image obtained from a camera. Hereinafter, the step of detecting the surrounding environment using image segmentation will be described in detail.

Image segmentation may mean segmenting an image region according to attributes. Alternatively, it may be a process of allocating an attribute value to each pixel of an image. For example, the attribute may be the type of object. In this case, image segmentation may mean segmenting an object included in an image for each pixel. Alternatively, it may mean indicating which object the specific pixel corresponds to.

Image segmentation may be performed using an artificial neural network. One artificial neural network may be used, and each artificial neural network may perform image segmentation using a plurality of artificial neural networks, and the surrounding environment may be sensed by combining the results. Various structures such as ENet structure can be applied to the network structure of artificial neural network for image segmentation.

The method for detecting the perimeter of a ship using an image according to an embodiment of the present invention is to detect the perimeter of a ship using an image that receives a sea image performed by a computing means and outputs information on the type and distance of an object included in the sea image. A method comprising the steps of: acquiring a sea image including a sea and an obstacle; obtaining labeling data including a plurality of labeling values generated based on the maritime image; receiving, by an artificial neural network, the sea image and outputting output data including a plurality of output values; calculating an error value using the labeling data and the output data; and updating the artificial neural network based on the error value.

The second labeling value and the second output value are selected from a plurality of identification values determined by a combination of information on a type and distance of an obstacle, and the plurality of identification values include a first distance range, the first distance range It is determined by a combination of information about a distance including a larger second distance range and a third distance range larger than the second distance range, and information about types of obstacles including fixed obstacles and dynamic obstacles.

An around-view system of a water-moving object according to an embodiment of the present invention, in the around-view system of a water-moving object using an artificial neural network performed by a computing means, includes an area operable by a camera installed on a ship and an obstacle including an obstacle acquiring an image; outputting output data by inputting the resolution image to an artificial neural network that performs image segmentation; obtaining second distance information of the second pixel based on a position of the second pixel on the sea image and posture information of the camera; and updating the second output value by reflecting final distance information calculated based on the first distance information and the second distance information.

The first distance information is selected from a plurality of categories each having a distance range, the second distance information is expressed as a distance value, and the updating of the second output value includes: the second distance information is the first distance information determining an error based on whether it is included in a distance range of ; calculating the final distance information based on the error; and updating the second output value by reflecting the final distance information.

In the present invention, the analysis of the image acquired by the image sensor may be performed using an artificial neural network. An artificial neural network is a kind of algorithm modeled after the structure of a human neural network, and may include one or more nodes or one or more layers including neurons, and each node may be connected through a synapse. Data input to the artificial neural network (input data) may be output (output data) through a node through a synapse, and information may be obtained through this.

Types of artificial neural networks include a convolutional neural network (CNN) that extracts features using filters, and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back as an input. In addition, there are various types such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), relational network (RN), etc.

Before using the artificial neural network, it is necessary to train it. Alternatively, it can be trained using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and the step of using the artificial neural network will be expressed as an inference step.

As a learning method of an artificial neural network, there are various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning. Here, reinforcement learning may be expressed as a Markov decision process. Alternatively, reinforcement learning may refer to how an agent behaves in an environment to maximize a reward.

In order to operate a vessel, obstacles and/or operable areas must be identified through sensing of the surrounding environment, and this is also the case for autonomous navigation. Here, the obstacle means any object that may become an obstacle during operation, such as terrain, buildings, ships, buoys, and people. In addition, sensing the surrounding environment is a comprehensive meaning that includes not only sensing an obstacle, but also acquiring information about the situation around the vessel. Detecting an obstacle includes acquiring information about the presence, type, location, and the like of the obstacle.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (2)

In the around-view system of the floating body using an artificial neural network performed by a computing means,
acquiring a sea image including a navigable area and obstacles by a camera installed on a ship;
outputting output data by inputting the resolution image to an artificial neural network that performs image segmentation;
obtaining second distance information of the second pixel based on a position of the second pixel on the sea image and posture information of the camera; and
and updating the second output value by reflecting the final distance information calculated based on the first distance information and the second distance information. The method of claim 1,
The first distance information is selected from a plurality of categories each having a distance range, and the second distance information is expressed as a distance value,
Updating the second output value includes:
determining an error based on whether the second distance information is included in a distance range of the first distance information;
calculating the final distance information based on the error; and
and updating the second output value by reflecting the final distance information.

Images