KR102339465B1 - Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition - Google Patents

Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition Download PDF

Info

Publication number
KR102339465B1
KR102339465B1 KR1020190084762A KR20190084762A KR102339465B1 KR 102339465 B1 KR102339465 B1 KR 102339465B1 KR 1020190084762 A KR1020190084762 A KR 1020190084762A KR 20190084762 A KR20190084762 A KR 20190084762A KR 102339465 B1 KR102339465 B1 KR 102339465B1
Authority
KR
South Korea
Prior art keywords
image
marine
marine waste
image recognition
obstacle
Prior art date
Application number
KR1020190084762A
Other languages
Korean (ko)
Other versions
KR20210007767A (en
Inventor
이성종
Original Assignee
수상에스티(주)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 수상에스티(주) filed Critical 수상에스티(주)
Priority to KR1020190084762A priority Critical patent/KR102339465B1/en
Publication of KR20210007767A publication Critical patent/KR20210007767A/en
Application granted granted Critical
Publication of KR102339465B1 publication Critical patent/KR102339465B1/en

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0088—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B63—SHIPS OR OTHER WATERBORNE VESSELS; RELATED EQUIPMENT
    • B63B—SHIPS OR OTHER WATERBORNE VESSELS; EQUIPMENT FOR SHIPPING 
    • B63B35/00—Vessels or similar floating structures specially adapted for specific purposes and not otherwise provided for
    • B63B35/32—Vessels or similar floating structures specially adapted for specific purposes and not otherwise provided for for collecting pollution from open water
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B63—SHIPS OR OTHER WATERBORNE VESSELS; RELATED EQUIPMENT
    • B63B—SHIPS OR OTHER WATERBORNE VESSELS; EQUIPMENT FOR SHIPPING 
    • B63B43/00—Improving safety of vessels, e.g. damage control, not otherwise provided for
    • B63B43/18—Improving safety of vessels, e.g. damage control, not otherwise provided for preventing collision or grounding; reducing collision damage
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B63—SHIPS OR OTHER WATERBORNE VESSELS; RELATED EQUIPMENT
    • B63H—MARINE PROPULSION OR STEERING
    • B63H21/00—Use of propulsion power plant or units on vessels
    • B63H21/21—Control means for engine or transmission, specially adapted for use on marine vessels
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/0206—Control of position or course in two dimensions specially adapted to water vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B63—SHIPS OR OTHER WATERBORNE VESSELS; RELATED EQUIPMENT
    • B63B—SHIPS OR OTHER WATERBORNE VESSELS; EQUIPMENT FOR SHIPPING 
    • B63B35/00—Vessels or similar floating structures specially adapted for specific purposes and not otherwise provided for
    • B63B2035/006—Unmanned surface vessels, e.g. remotely controlled
    • B63B2035/007—Unmanned surface vessels, e.g. remotely controlled autonomously operating

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Combustion & Propulsion (AREA)
  • Mechanical Engineering (AREA)
  • Ocean & Marine Engineering (AREA)
  • Chemical & Material Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Public Health (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Automation & Control Theory (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Environmental & Geological Engineering (AREA)
  • Game Theory and Decision Science (AREA)
  • Medical Informatics (AREA)
  • Business, Economics & Management (AREA)
  • Image Analysis (AREA)

Abstract

본 발명은 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템을 제공한다. 이와 같은 본 발명에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템은 무인선 또는 최소인원이 탑승한 선박 본체에 구비된 카메라모듈을 통한 영상인식으로 스캐닝되는 비닐, 스티로폼, 플라스틱 등의 해상폐기물을 자율운항으로 이동하는 선박 본체의 부유물 수집장치로 제거하도록 함으로써 해양환경의 개선효율 증대 및 개선작업비용 절감이 도모될 수 있고, 자율운항 및 해향환경 개선을 위한 선박/해상 구조물 및 해상폐기물의 영상인식이 실사영상과 3D 가상모델로부터 대량으로 생성되는 양태이미지에 대한 기계학습을 통해 수행되도록 함으로써 해상폐기물의 인식률 증대 및 인식 정밀도/정확도 향상이 도모될 수 있는 한편 자율운항 안정성 증대가 도모될 수 있는 기술적 특징을 갖는다.The present invention provides an autonomous navigation ship system for removing marine waste based on deep learning-image recognition. The deep learning-image recognition-based autonomous navigation ship system for marine waste removal according to the present invention as described above is an unmanned ship or a camera module provided in the body of a ship with a minimum number of people on board. By removing marine wastes with the floating object collection device of the ship's body moving autonomously, it is possible to increase the efficiency of improvement of the marine environment and reduce the cost of improvement work, and to improve the autonomous navigation and maritime environment of ships/offshore structures and marine wastes By making the image recognition of image recognition performed through machine learning for aspect images generated in large quantities from live-action images and 3D virtual models, it is possible to increase the recognition rate and improve recognition precision/accuracy of marine waste, while increasing the stability of autonomous navigation. capable technical features.

Description

딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템{Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition}Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition}

본 발명은 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템에 관한 것으로, 좀더 구체적으로는 무인선 또는 최소인원이 탑승한 선박 본체에 구비된 카메라모듈을 통한 영상인식으로 스캐닝되는 비닐, 스티로폼, 플라스틱 등의 해상폐기물을 자율운항으로 이동하는 선박 본체의 부유물 수집장치로 제거하도록 함으로써 해양환경의 개선효율 증대 및 개선작업비용 절감이 도모될 수 있고, 자율운항 및 해향환경 개선을 위한 선박/해상 구조물 및 해상폐기물의 영상인식이 실사영상과 3D 가상모델로부터 대량으로 생성되는 양태이미지에 대한 기계학습을 통해 수행되도록 함으로써 해상폐기물의 인식률 증대 및 인식 정밀도/정확도 향상이 도모될 수 있는 한편 자율운항 안정성 증대가 도모될 수 있는 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템에 관한 것이다.The present invention relates to a deep learning-image recognition-based autonomous navigation ship system for removing marine waste, and more specifically, vinyl, styrofoam scanned by image recognition through a camera module provided in an unmanned ship or a ship body with a minimum number of people on board. By removing marine wastes such as plastics, plastics, etc. with the floating object collection device of the ship body moving autonomously, it is possible to increase the improvement efficiency of the marine environment and reduce the improvement work cost, and to improve the ship/sea environment for autonomous navigation and navigation. By allowing the image recognition of structures and marine wastes to be performed through machine learning on the aspect images generated in large quantities from live-action images and 3D virtual models, it is possible to increase the recognition rate and improve recognition precision/accuracy of marine wastes, while also improving the stability of autonomous navigation. It relates to an autonomous ship system for removing marine waste based on deep learning-image recognition that can be increased.

바다에는 비닐, 스티로폼, 플라스틱, 고무제품, 목재품 등의 해상폐기물들이 다량으로 부유하고 있는데, 대부분의 해상폐기물들은 여기저기 이동하면서 해상환경을 악화시키는 한편 분해에 장시간이 소요되면서 지속적으로 해양생태계에 악영향을 미치고 있다.A large amount of marine wastes such as vinyl, Styrofoam, plastics, rubber products, and wood products are floating in the sea. is adversely affecting

이에 따라 해상폐기물은 수시로 제거될 필요가 있는데, 종래에는 잠수부와 작업선을 이용하여 인력으로 작업을 하였다. 그러나 인력에 의한 해상폐기물 제거작업은 작업능률의 저하, 수거시간 증대 등 많은 한계가 있으므로, 선박이나 구조물을 통해 해상폐기물을 제거하는 기술이 개발되어 사용되고 있다.Accordingly, marine wastes need to be removed from time to time. Conventionally, divers and work boats were used to work by manpower. However, since the removal of marine waste by manpower has many limitations, such as a decrease in work efficiency and an increase in collection time, a technology for removing marine waste through a ship or structure has been developed and used.

이와 관련한 기술로는 대한민국 등록특허공보 등록번호 제10-1803364호 "수상부유물 제거장치", 등록실용신안공보 등록번호 제20-0324102호 "댐 내부 및 해상의 부유물질 제거를 위한 수거장비" 등이 안출되어 있는데, 종래의 해상폐기물 제거장치의 경우 다양한 종류의 해상폐기물을 능동적으로 인식하여 제거하는데 어려움이 있었으며, 해상폐기물 제거효율도 떨어지는 문제점이 있었다.As related technologies, the Republic of Korea Patent Publication No. 10-1803364 "Floating material removal device", Utility Model Gazette Registration No. 20-0324102 "Collection equipment for removing suspended solids in dams and in the sea", etc. Although it has been devised, in the case of the conventional marine waste removal apparatus, it was difficult to actively recognize and remove various types of marine waste, and there was a problem in that the efficiency of removing the marine waste was also reduced.

대한민국 등록특허공보 등록번호 제10-1803364호 "수상부유물 제거장치"Republic of Korea Patent Publication Registration No. 10-1803364 "A device for removing floating matter" 대한민국 등록실용신안공보 등록번호 제20-0324102호 "댐 내부 및 해상의 부유물질 제거를 위한 수거장비"Republic of Korea Utility Model Registration No. 20-0324102 "Collection equipment for removing suspended solids in dams and in the sea"

따라서 본 발명은 이와 같은 종래 기술의 문제점을 개선하여, 무인선 또는 최소인원이 탑승한 선박 본체에 카메라모듈과 대상체 이격거리 측정장치가 설치되고, 카메라모듈의 360°전방향 영상에 대한 CNN 기반 딥러닝 영상인식 및 레이저 측위를 통해 회피대상 장애물(선박/해상구조물)과 해상폐기물에 대한 인식과 위치정보 검출이 수행되면서 선박 본체의 자율운항과 해상폐기물 수집이 유도됨으로써 해양환경의 개선효율 증대 및 개선작업비용 절감이 도모될 수 있는 새로운 형태의 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템을 제공하는 것을 목적으로 한다.Therefore, the present invention improves the problems of the prior art, and a camera module and an object separation distance measuring device are installed in an unmanned ship or a ship body with a minimum number of people on board, and a CNN-based deep dive for 360° omnidirectional images of the camera module Through running image recognition and laser positioning, recognition of obstacles to be avoided (ships/offshore structures) and marine waste and location information detection are performed, and autonomous navigation of the ship body and collection of offshore waste are induced to increase and improve the improvement efficiency of the marine environment. It aims to provide a new type of deep learning-image recognition-based autonomous ship system for removing marine waste that can reduce work costs.

또한 본 발명은 회피대상 장애물 및 해상폐기물의 영상인식이 실사영상과 3D 가상모델로부터 대량으로 생성되는 양태이미지를 학습이미지로 한 딥 러닝을 통해 수행되도록 함으로써 해상폐기물의 인식률 증대 및 인식 정밀도/정확도 향상이 도모될 수 있는 한편 자율운항 안정성 증대가 도모될 수 있는 새로운 형태의 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템을 제공하는 것을 목적으로 한다.In addition, the present invention increases the recognition rate of marine wastes and improves recognition precision/accuracy by allowing the image recognition of obstacles to be avoided and marine wastes to be performed through deep learning using aspect images generated in large quantities from live-action images and 3D virtual models as learning images. The purpose of this is to provide a new type of deep learning-image recognition-based autonomous navigation ship system for removing marine waste that can achieve this and increase the stability of autonomous navigation.

상술한 목적을 달성하기 위한 본 발명의 특징에 의하면, 본 발명은 선체 추진장치(110)가 구비되어 해상에서 운항되고, 부유물 수집장치(120)가 구비되어 해상폐기물을 수집하게 되는 선박 본체(100); 상기 선박 본체(100)의 상부에 시야 간섭없이 설치되어 선박 본체(100)를 중심으로 한 360°전방향 영상을 촬영하게 되는 카메라모듈(200); 상기 카메라모듈(200)로부터 360°전방향 영상을 전달받게 되고, 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)이 구비되어 선박과 해상구조물을 포함하는 회피대상 장애물 및 비닐, 스티로폼, 플라스틱을 포함하는 해상폐기물을 식별하게 되며, 회피대상 장애물 식별정보와 해상폐기물 식별정보를 생성하게 되는 영상분석장치(300); 상기 선박 본체(100)의 상부에 시야 간섭없이 설치되며, 상기 영상분석장치(300)로부터 회피대상 장애물 식별정보와 해상폐기물 식별정보를 전달받게 되고, 회피대상 장애물 식별정보와 해상폐기물 식별정보에 각각 포함된 회피대상 장애물 방향정보와 해상폐기물 방향정보에 맞추어 지향방향을 설정한 다음 선박과 회피대상 장애물 간 이격거리(이하 장애물 거리) 및 선박과 해상폐기물 간 이격거리(이하 부유물 거리)를 측정하게 되는 대상체 이격거리 측정장치(400); 및 상기 선체 추진장치(110)에 대한 동작제어를 통해 상기 선박 본체(100)의 해상 운항을 유도하게 되되, 상기 영상분석장치(300)와 대상체 이격거리 측정장치(400)와 연동되어 장애물 방향정보와 장애물 거리 및 해상폐기물 방향정보와 부유물 거리를 전달받게 되고, 회피대상 장애물 방향정보와 장애물 거리 및 해상폐기물 방향정보와 부유물 거리에 맞추어 상기 선체 추진장치(110)에 대한 동작제어를 수행하는 추진장치 제어장치(500);를 포함하는 구성으로 이루어져 회피대상 장애물을 회피하면서 해상폐기물이 위치한 영역으로 이동하여 해상폐기물을 수집하게 되는 것을 특징으로 하는 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템을 제공한다.According to the features of the present invention for achieving the above object, the present invention is a ship body 100 that is provided with a hull propulsion device 110 to operate on the sea, and a floating object collection device 120 is provided to collect marine waste. ); a camera module 200 that is installed on the upper portion of the ship body 100 without visual interference and takes a 360° omnidirectional image centered on the ship body 100; A 360° omnidirectional image is received from the camera module 200, and an image recognition algorithm 310 for identifying obstacles to be avoided and an image recognition algorithm 320 for identifying marine waste are provided to avoid including ships and offshore structures. An image analysis device 300 that identifies target obstacles and marine wastes including vinyl, Styrofoam, and plastics, and generates avoidance target obstacle identification information and marine waste identification information; It is installed on the upper part of the ship body 100 without visual interference, and receives obstacle identification information and marine waste identification information to be avoided from the image analysis device 300, and to the obstacle identification information to be avoided and marine waste identification information, respectively. After setting the direction according to the included obstacle direction information and marine waste direction information, the separation distance between the vessel and the obstacle to be avoided (hereinafter referred to as the obstacle distance) and the separation distance between the vessel and the marine waste (hereinafter the floating object distance) are measured. Object separation distance measuring device 400; And through operation control of the hull propulsion device 110, the navigation of the ship body 100 is induced, and the image analysis device 300 and the object separation distance measuring device 400 are interlocked to provide obstacle direction information. and obstacle distance, marine waste direction information, and floating object distance are received, and the operation control for the hull propulsion device 110 is performed according to the obstacle direction information to be avoided, the obstacle distance, the marine waste direction information and the floating object distance. A control device 500; A self-driving ship system for removing marine waste based on deep learning-image recognition, characterized in that it moves to the area where the marine waste is located while avoiding the obstacle to be avoided and collects the marine waste provides

이와 같은 본 발명에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템은 상기 선박 본체(100)와의 원격 무선통신으로 상기 선체 추진장치(110)에 대한 원격 동작제어를 수행하게 되는 관제센터 서버(600);를 더 포함하되, 상기 영상분석장치(300)와 추진장치 제어장치(500)는 상기 선박 본체(100)와 관제센터 서버(600) 중에서 선택된 어느 하나에 설치될 수 있다.The deep learning-image recognition based autonomous navigation ship system for removing marine waste according to the present invention is a control center that performs remote operation control for the hull propulsion device 110 through remote wireless communication with the ship body 100 Server 600; further comprising, the image analysis device 300 and the propulsion device control device 500 may be installed in any one selected from the ship body 100 and the control center server (600).

이와 같은 본 발명에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템은, 해상 환경에서의 회피대상 장애물과 해상폐기물의 실사영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 실사영상 데이터셋 DB(700); 해상 환경에서의 회피대상 장애물과 해상폐기물의 3D 가상모델로 구성되는 가상모델링영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 가상모델링영상 데이터셋 DB(800); 상기 실사영상 데이터셋 DB(700)의 실사영상 데이터셋에 대한 3D 실사모델링에 의해 생성되는 3D 실사모델로 구성되는 실사모델링영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 실사모델링영상 데이터셋 DB(900); 상기 가상모델링영상 데이터셋 DB(800)와 실사모델링영상 데이터셋 DB(900)로부터 전달되는 3D 가상모델과 3D 실사모델 각각에 대하여 해상환경조건, 기상조건, 주야간 조건, 촬영각도 별로 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지를 생성시켜 학습이미지 데이터로 데이터베이스화하게 되되, 상기 해상환경조건에는 풍향, 풍속, 파랑이 포함되고, 상기 기상조건에는 강우량, 강설량, 안개가 포함되는 학습이미지 DB(1000); 딥러닝 영상인식 알고리즘(1110)을 통해 상기 학습이미지 DB(1000)의 3D 가상모델 양태이미지와 3D 실사모델 양태이미지에 대한 기계학습을 수행하여 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 되는 딥러닝 영상인식 학습모듈(1100);을 포함할 수 있다.The deep learning-image recognition-based autonomous navigation ship system for marine waste removal according to the present invention, as described above, is a live-action image dataset in which the collection, storage, and databaseization of actual image datasets of obstacles to be avoided and marine wastes in a marine environment are performed. DB (700); A virtual modeling image dataset DB 800 for collecting, storing, and databaseizing a virtual modeling image dataset composed of a 3D virtual model of an obstacle to be avoided and a marine waste in a marine environment; A live-action modeling image dataset DB ( 900); 3D virtual model that varies according to marine environmental conditions, weather conditions, day/night conditions, and shooting angles for each of the 3D virtual model and the 3D live-action model delivered from the virtual modeling image dataset DB 800 and the live-action modeling image dataset DB 900 A mode image and a 3D live-action model mode image are created and converted into a database as learning image data, wherein the marine environmental conditions include wind direction, wind speed, and waves, and the weather conditions include rainfall, snowfall, and fog. Learning image DB ( 1000); Through the deep learning image recognition algorithm 1110, machine learning is performed on the 3D virtual model aspect image of the learning image DB 1000 and the 3D live-action model aspect image to identify the obstacle to be avoided image recognition algorithm 310 and marine waste It may include; a deep learning image recognition learning module 1100 that calculates the image recognition algorithm 320 for identification.

이와 같은 본 발명에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템에서 상기 실사영상 데이터셋 DB(700)는, 실사 촬영으로 생성된 실사 영상데이터셋이 저장, 데이터베이스화되는 촬영 실사영상 데이터셋 DB(710);와 웹 크롤링(web crawling)으로 수집된 웹수집 실사영상 데이터셋이 저장, 데이터베이스화되는 웹수집 실사영상 데이터셋 DB(720);를 포함하는 구성으로 이루어질 수 있다.In such a deep learning-image recognition-based autonomous navigation ship system for removing marine waste according to the present invention, the live-action image data set DB 700 is a photo-realistic image in which the live-action image dataset generated by the live-action shooting is stored and converted into a database. A data set DB 710; and a web collected live-action image dataset DB 720 in which a web-collected live-action image dataset collected by web crawling is stored and converted into a database; may be configured to include.

이와 같은 본 발명에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템에서 상기 딥러닝 영상인식 학습모듈(1100)은 CNN(Convolutional Neural Network) 기반 딥러닝 영상인식 알고리즘(1110)의 파라미터 조절을 통해 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 되되, CNN 기반 딥러닝 영상인식 알고리즘(1110)의 파라미터에는 필터(filter), 스트라이드(stride), 패딩(padding)이 포함될 수 있다.In the deep learning-image recognition-based autonomous navigation ship system for removing marine waste according to the present invention, the deep learning image recognition learning module 1100 is a Convolutional Neural Network (CNN)-based deep learning image recognition algorithm 1110 parameter adjustment An image recognition algorithm 310 for identifying obstacles to be avoided and an image recognition algorithm 320 for identifying marine waste are calculated through the parameters of the CNN-based deep learning image recognition algorithm 1110 include filters and strides ), padding may be included.

본 발명에 의한 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템에 의하면, 무인선 또는 최소인원이 탑승한 선박 본체에 구비된 카메라모듈을 통한 영상인식으로 스캐닝되는 해상폐기물을 자율운항으로 이동하는 선박 본체의 부유물 수집장치로 제거하도록 하므로, 해양환경의 개선효율 증대 및 개선작업비용 절감이 도모되는 효과가 있다. 그리고 본 발명에 의한 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템에 의하면, 회피대상 장애물 및 해상폐기물의 영상인식이 실사영상과 3D 가상모델로부터 대량으로 생성되는 양태이미지에 대한 기계학습을 통해 수행되도록 하므로, 해상폐기물의 인식률 증대 및 인식 정밀도/정확도 향상이 도모되는 한편 자율운항 안정성 증대가 도모되는 효과가 있다.According to the deep learning-image recognition-based autonomous navigation ship system for marine waste removal according to the present invention, marine waste scanned by image recognition through a camera module provided in an unmanned ship or a ship body with a minimum number of people on board is moved to autonomous navigation. Since it is removed by the floating object collecting device of the ship body, there is an effect of increasing the improvement efficiency of the marine environment and reducing the improvement work cost. And, according to the deep learning-image recognition-based autonomous navigation ship system for marine waste removal according to the present invention, image recognition of obstacles to be avoided and marine waste is machine learning for aspect images generated in large quantities from live-action images and 3D virtual models. Therefore, it has the effect of increasing the recognition rate of marine waste and improving the recognition precision/accuracy while increasing the stability of autonomous navigation.

도 1은 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템의 기본 구성블록도;
도 2와 도 3은 본 발명의 실시예에 따른 선박 본체의 구성을 보여주기 위한 도면;
도 4는 본 발명의 실시예에 따른 선박 본체의 조타기 원격제어기를 보여주기 위한 도면;
도 5는 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템의 자율운항 관련 구성 블록도;
도 6과 도 7은 본 발명의 실시예에 따른 영상분석장치에 의해 회피대상 장애물과 해상폐기물이 인식되는 것을 보여주기 위한 예시도;
도 8은 본 발명의 실시예에 따른 영상분석장치에서 기준 픽셀영역 이상의 크기를 갖는 회피대상 장애물과 해상폐기물만을 식별하게 되는 것을 보여주기 위한 예시도;
도 9는 본 발명의 실시예에 따른 대상체 이격거리 측정장치를 보여주기 위한 도면;
도 10은 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템의 기계학습 구성 블록도;
도 11은 본 발명의 실시예에 따른 실사모델링영상 데이터셋 DB에서 실상영상으로부터 3D 실사모델이 생성되는 것을 보여주기 위한 예시도;
도 12는 본 발명의 실시예에 따른 학습이미지 DB에서 기상조건에 따라 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지를 보여주기 위한 예시도;
도 13은 본 발명의 실시예에 따른 학습이미지 DB에서 촬영각도 별로 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지가 하나의 3D 가상모델과 하나의 3D 실사모델에 대한 전방위 회전 렌더링을 통해 생성되는 것을 보여주기 위한 예시도이다.1 is a basic block diagram of a deep learning-image recognition based autonomous navigation ship system for removing marine waste according to an embodiment of the present invention;
2 and 3 are views for showing the configuration of a ship body according to an embodiment of the present invention;
4 is a view for showing a steering gear remote controller of the ship body according to an embodiment of the present invention;
5 is a block diagram of an autonomous navigation-related configuration of an autonomous navigation ship system for deep learning-image recognition-based marine waste removal according to an embodiment of the present invention;
6 and 7 are exemplary views showing that the obstacle to be avoided and the marine waste are recognized by the image analysis apparatus according to an embodiment of the present invention;
8 is an exemplary view for showing that only obstacles to be avoided and marine wastes having a size greater than or equal to a reference pixel area are identified in the image analysis apparatus according to an embodiment of the present invention;
9 is a view showing an object separation distance measuring apparatus according to an embodiment of the present invention;
10 is a machine learning configuration block diagram of an autonomous navigation ship system for deep learning-image recognition-based marine waste removal according to an embodiment of the present invention;
11 is an exemplary view for showing that a 3D live-action model is generated from a real image in the real-life modeling image dataset DB according to an embodiment of the present invention;
12 is an exemplary diagram for showing a 3D virtual model aspect image and a 3D live-action model aspect image that vary depending on weather conditions in the learning image DB according to an embodiment of the present invention;
13 is a 3D virtual model aspect image and a 3D live-action model aspect image that vary by shooting angle in the learning image DB according to an embodiment of the present invention are generated through omnidirectional rotation rendering for one 3D virtual model and one 3D live-action model It is an example diagram to show.

이하, 본 발명의 실시예를 첨부된 도면 도 1 내지 도 13에 의거하여 상세히 설명한다. 한편, 도면과 상세한 설명에서 일반적인 선박, 선체, 자율운항 선박, 무인선, 선체 추진장치, 조타기, 카메라, 영상분석기술, 거리측정기술, 자동노출 조절기술, 딥 러닝(deep learning), CNN(Convolutional Neural Network) 기반 딥러닝 영상인식 알고리즘, 웹 크롤링(web crawling) 등으로부터 이 분야의 종사자들이 용이하게 알 수 있는 구성 및 작용에 대한 도시 및 언급은 간략히 하거나 생략하였다. 특히 도면의 도시 및 상세한 설명에 있어서 본 발명의 기술적 특징과 직접적으로 연관되지 않는 요소의 구체적인 기술적 구성 및 작용에 대한 상세한 설명 및 도시는 생략하고, 본 발명과 관련되는 기술적 구성만을 간략하게 도시하거나 설명하였다.Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings 1 to 13. Meanwhile, in the drawings and detailed description, general ships, hulls, autonomous ships, unmanned ships, hull propulsion devices, steering gear, cameras, image analysis technology, distance measurement technology, automatic exposure control technology, deep learning, CNN (Convolutional) Neural Network)-based deep learning image recognition algorithm, web crawling, etc., the illustration and description of the composition and operation that can be easily known by those involved in this field are simplified or omitted. In particular, in the drawings and detailed descriptions, detailed descriptions and illustrations of specific technical configurations and actions of elements not directly related to the technical features of the present invention are omitted, and only the technical configurations related to the present invention are briefly illustrated or described. did

본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템(1)은 도 1에서와 같이 선박 본체(100), 카메라모듈(200), 영상분석장치(300), 대상체 이격거리 측정장치(400), 추진장치 제어장치(500), 관제센터 서버(600)를 포함하는 구성으로 이루어지는 것으로, 선박 본체(100)가 회피대상 장애물(선박, 해상구조물 등)을 회피하면서 해상폐기물(비닐, 스티로폼, 플라스틱 등)이 위치한 영역으로 이동하여 해상폐기물을 수집하도록 하기 위한 시스템이다.The deep learning-image recognition-based autonomous navigation ship system 1 for removing marine waste according to an embodiment of the present invention is a ship body 100, a camera module 200, an image analysis device 300, and an object as shown in FIG. It consists of a configuration including a separation distance measuring device 400, a propulsion device control device 500, and a control center server 600, and the ship body 100 avoids obstacles to be avoided (ships, offshore structures, etc.) It is a system for collecting marine waste by moving to the area where waste (vinyl, styrofoam, plastic, etc.) is located.

선박 본체(100)는 도 2와 도 3에서와 같이 선체 추진장치(110)와 부유물 수집장치(120)를 구비하여 해상을 운항하면서 해상폐기물을 수집하게 된다. 선박 본체(100)는 무인선으로 자율 운항될 수도 있고, 최소인원이 탑승한 상태로 직접 조종되거나 자율 운항될 수도 있다. 자율 운항은 관제센터에 의한 원격제어를 통해 수행될 수도 있고, 선박 본체(100)에 설치된 영상분석장치(300), 추진장치 제어장치(500)를 통해 수행될 수도 있다. The ship body 100 is provided with a hull propulsion device 110 and a floating object collecting device 120 as in FIGS. 2 and 3 to collect marine waste while operating the sea. The ship body 100 may be operated autonomously as an unmanned ship, or may be directly operated or operated autonomously with a minimum number of people on board. The autonomous navigation may be performed through remote control by the control center, or may be performed through the image analysis device 300 and the propulsion device control device 500 installed in the ship body 100 .

또한 선박 본체(100)에는 조타기(130)가 설치되어 방향제어가 수행되도록 하는데, 본 발명의 실시예에 따른 선박 본체(100)는 도 4에서와 같이 조타기 원격제어기(140)를 구비하여 조타기(130)에 의한 방향제어도 원격으로 이루어질 수 있도록 한다.In addition, the steering gear 130 is installed in the ship body 100 to perform direction control, and the ship body 100 according to an embodiment of the present invention includes a steering gear remote controller 140 as shown in FIG. 130), so that the direction control can also be performed remotely.

카메라모듈(200)은 선박 본체(100)의 상부에 시야 간섭없이 설치되어 선박 본체(100)를 중심으로 한 360°전방향 영상을 촬영하게 된다. 여기서 본 발명의 실시예에 따른 카메라모듈(200)은 자동노출 조절유닛(210)을 구비하여 영상 밝기가 설정범위 내에서 유지되도록 한다.The camera module 200 is installed on the upper part of the ship body 100 without interference of the field of view to take a 360° omnidirectional image centered on the ship body 100 . Here, the camera module 200 according to the embodiment of the present invention is provided with an automatic exposure control unit 210 so that the image brightness is maintained within a set range.

영상분석장치(300)는 카메라모듈(200)로부터 360°전방향 영상을 전달받게 되는 것으로, 도 5에서와 같이 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)이 구비되어 도 6에서와 같이 회피대상 장애물과 해상폐기물을 식별한 다음, 회피대상 장애물 식별정보와 해상폐기물 식별정보를 생성하게 된다. 회피대상 장애물 식별정보와 해상폐기물 식별정보에는 회피대상 장애물 방향정보와 해상폐기물 방향정보, 회피대상 장애물 종류정보와 해상폐기물 종류정보, 크기정보 등이 포함될 수 있다. 회피대상 장애물에는 선박, 해상구조물 등이 포함될 수 있고, 해상폐기물에는 비닐, 스티로폼, 플라스틱, 고무제품, 목재품 등이 포함될 수 있다.The image analysis device 300 receives a 360° omnidirectional image from the camera module 200, and as shown in FIG. 5, an image recognition algorithm 310 for identifying an obstacle to be avoided and an image recognition algorithm 320 for identifying marine waste ) is provided to identify the obstacle to be avoided and the marine waste as shown in FIG. 6 , and then to generate the obstacle identification information and the marine waste identification information to be avoided. The avoidable obstacle identification information and marine waste identification information may include avoidable obstacle direction information, marine waste direction information, avoidable obstacle type information, marine waste type information, size information, and the like. Obstacles to be avoided may include ships and offshore structures, and marine wastes may include vinyl, styrofoam, plastic, rubber products, wood products, and the like.

한편 본 발명의 실시예에 따른 영상분석장치(300)는 기준 픽셀영역(N×N 픽셀, N는 기준 픽셀값) 이상의 크기를 갖는 회피대상 장애물과 해상폐기물만을 식별하게 되는데, 이는 선박 본체(100)의 운항에 영향을 미치지 않는 크기의 장애물과 집적되어 있는 정도가 떨어져 소요에너지 대비 제거효율이 떨어지는 폐기물은 식별되지 않도록 하기 위함이다. 예를 들어 기준 픽셀영역이 3×3 픽셀영역으로 설정될 경우, 도 8에서 A 내부의 회피대상 장애물이나 해상폐기물은 식별되지 않고, B와 C 내부의 회피대상 장애물이나 해상폐기물은 식별되게 된다.On the other hand, the image analysis apparatus 300 according to the embodiment of the present invention identifies only obstacles to be avoided and marine wastes having a size greater than or equal to the reference pixel area (N × N pixels, N is the reference pixel value), which is the ship body 100 ) to prevent the identification of obstacles of a size that do not affect the operation and wastes with low removal efficiency compared to the required energy due to the low degree of accumulation. For example, when the reference pixel area is set to a 3×3 pixel area, in FIG. 8 , the obstacle to be avoided or the marine waste inside A is not identified, and the obstacle or the marine waste to be avoided inside the B and C is identified.

대상체 이격거리 측정장치(400)는 선박 본체(100)의 상부에 시야 간섭없이 설치되는 것으로, 영상분석장치(300)로부터 회피대상 장애물 식별정보와 해상폐기물 식별정보를 전달받게 되고, 회피대상 장애물 식별정보와 해상폐기물 식별정보에 각각 포함된 회피대상 장애물 방향정보와 해상폐기물 방향정보에 맞추어 지향방향을 설정한 다음 선박과 회피대상 장애물 간 이격거리(이하 장애물 거리) 및 선박과 해상폐기물 간 이격거리(이하 부유물 거리)를 측정하게 된다. 여기서 본 발명의 실시예에 따른 대상체 이격거리 측정장치(400)는 회피대상 장애물과 해상폐기물로의 레이저 펄스 조사를 통해 장애물 거리 및 부유물 거리를 측정하게 되는 레이저 측위장치(400a)로 이루어지는데, 본 발명의 실시예에 따른 레이저 측위장치(400a)는 도 9에서와 같이 4개의 레이저 광원(411)이 4행 1열로 배열되어 있는 레이저 조사유닛(410)과 레이저 조사유닛(410) 양측에 배치되는 레이저 수광유닛(420)을 포함하는 구성으로 이루어진다.The object separation distance measuring device 400 is installed on the upper part of the ship body 100 without visual interference, and receives the target obstacle identification information and the marine waste identification information from the image analysis device 300, and identifies the obstacle to be avoided After setting the direction according to the direction information of the obstacle to be avoided and the direction information of the marine waste included in the information and the marine waste identification information, respectively, the separation distance between the vessel and the obstacle to be avoided (hereinafter referred to as the obstacle distance) and the separation distance between the vessel and the marine waste ( Hereinafter, the float distance) is measured. Here, the object separation distance measuring device 400 according to the embodiment of the present invention consists of a laser positioning device 400a that measures the obstacle distance and the floating object distance by irradiating laser pulses to the obstacle to be avoided and the marine waste. The laser positioning device 400a according to the embodiment of the present invention is disposed on both sides of the laser irradiation unit 410 and the laser irradiation unit 410 in which four laser light sources 411 are arranged in four rows and one column as shown in FIG. It consists of a configuration including a laser light receiving unit (420).

추진장치 제어장치(500)는 선체 추진장치(110)에 대한 동작제어를 통해 선박 본체(100)의 해상 운항을 유도하게 되는 것으로, 영상분석장치(300)와 대상체 이격거리 측정장치(400)와 연동되어 장애물 방향정보와 장애물 거리 및 해상폐기물 방향정보와 부유물 거리를 전달받게 되고, 회피대상 장애물 방향정보와 장애물 거리 및 해상폐기물 방향정보와 부유물 거리에 맞추어 선체 추진장치(110)에 대한 동작제어를 수행하게 된다. 추진장치 제어장치(500)는 선박 본체(100)에 설치될 수도 있고, 관제센터 서버(600)에 설치될 수도 있다.The propulsion device control device 500 induces maritime navigation of the ship body 100 through operation control for the hull propulsion device 110 , and includes the image analysis device 300 and the object separation distance measuring device 400 and It is linked to receive obstacle direction information, obstacle distance, marine waste direction information, and floating object distance, and controls the operation of the hull propulsion device 110 according to the obstacle direction information to be avoided, the obstacle distance, the marine waste direction information and the floating object distance. will perform The propulsion control device 500 may be installed in the ship body 100 or may be installed in the control center server 600 .

관제센터 서버(600)는 육상의 관제센터에 구비되는 것으로, 필요시 선박 본체(100)와의 원격 무선통신으로 선체 추진장치(110)에 대한 원격 동작제어를 수행하게 된다.The control center server 600 is provided in the control center on land, and if necessary, performs remote operation control for the hull propulsion device 110 through remote wireless communication with the ship body 100 .

여기서 영상분석장치(300)와 추진장치 제어장치(500)는 선박 본체(100)와 관제센터 서버(600) 중에서 선택된 어느 하나에 설치될 수 있다.Here, the image analysis device 300 and the propulsion device control device 500 may be installed in any one selected from the ship body 100 and the control center server 600 .

한편 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템(1)에는 도 10에서와 같이 실사영상 데이터셋 DB(700), 가상모델링영상 데이터셋 DB(800), 실사모델링영상 데이터셋 DB(900), 학습이미지 DB(1000), 딥러닝 영상인식 학습모듈(1100)이 구비된다. On the other hand, in the deep learning-image recognition-based autonomous navigation ship system 1 for removing marine waste according to an embodiment of the present invention, as shown in FIG. 10 , the live image data set DB 700 , the virtual modeling image data set DB 800 , A live-action modeling image dataset DB 900 , a learning image DB 1000 , and a deep learning image recognition learning module 1100 are provided.

실사영상 데이터셋 DB(700)는 해상 환경에서의 회피대상 장애물과 해상폐기물의 실사영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 DB이다. 여기서 본 발명의 실시예에 따른 실사영상 데이터셋 DB(700)는 촬영 실사영상 데이터셋 DB(710)와 웹수집 실사영상 데이터셋 DB(720)를 포함하는 구성으로 이루어지는데, 촬영 실사영상 데이터셋 DB(710)는 직접적인 실사 촬영으로 생성된 실사 영상데이터셋이 저장, 데이터베이스화되는 DB이고, 웹수집 실사영상 데이터셋 DB(720)는 웹 크롤링(web crawling)으로 수집된 웹수집 실사영상 데이터셋이 저장, 데이터베이스화되는 DB이다. The actual image dataset DB 700 is a DB in which the collection, storage, and databaseization of the actual image dataset of obstacles to be avoided and marine waste in the marine environment is performed. Here, the live-action image data set DB 700 according to the embodiment of the present invention is composed of a photographing live-action image dataset DB 710 and a web-collected live-action image dataset DB 720 , and the photographing live-action image dataset The DB 710 is a DB in which a live-action image data set generated by direct real-world shooting is stored and converted to a database, and the web-collected live-action image dataset DB 720 is a web-collected live-action image dataset collected by web crawling. This is the DB that is stored and converted into a database.

가상모델링영상 데이터셋 DB(800)는 해상 환경에서의 회피대상 장애물과 해상폐기물의 3D 가상모델로 구성되는 가상모델링영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 DB이다. 3D 가상모델은 라이노(Rhino), 3D MAX와 같은 3D 모델링 툴을 통해 제작된다. The virtual modeling image dataset DB 800 is a DB in which collection, storage, and databaseization of a virtual modeling image dataset composed of a 3D virtual model of an obstacle to be avoided and marine waste in a marine environment is performed. 3D virtual models are created through 3D modeling tools such as Rhino and 3D MAX.

실사모델링영상 데이터셋 DB(900)는 도 11에서와 같이 실사영상 데이터셋 DB(700)의 실사영상 데이터셋에 대한 3D 실사모델링에 의해 생성되는 3D 실사모델로 구성되는 실사모델링영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 DB이다. 3D 실사모델도 라이노(Rhino), 3D MAX와 같은 3D 모델링 툴을 통해 제작된다. The live-action modeling image dataset DB 900 is a collection of live-action modeling image datasets composed of a 3D live-action model generated by 3D live-action modeling for the live-action image dataset of the live-action image dataset DB 700 as shown in FIG. 11 . It is a DB where , storage, and databaseization are performed. 3D live-action models are also created through 3D modeling tools such as Rhino and 3D MAX.

학습이미지 DB(1000)는 가상모델링영상 데이터셋 DB(800)와 실사모델링영상 데이터셋 DB(900)로부터 전달되는 3D 가상모델과 3D 실사모델 각각에 대하여 해상환경조건, 기상조건, 주야간 조건, 촬영각도 별로 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지를 생성시켜 학습이미지 데이터로 데이터베이스화한 DB이다. 여기서 해상환경조건에는 풍향, 풍속, 파랑이 포함되고, 기상조건에는 강우량, 강설량, 안개가 포함되는데, 도 12에는 기상조건에 따라 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지가 예시되어 있다.The learning image DB 1000 is a 3D virtual model and a 3D live-action model delivered from the virtual modeling image dataset DB 800 and the live-action modeling image dataset DB 900, respectively, for marine environmental conditions, weather conditions, day and night conditions, and shooting It is a DB that creates a 3D virtual model aspect image that varies by angle and a 3D live-action model aspect image and turns it into a database as learning image data. Here, marine environmental conditions include wind direction, wind speed, and waves, and weather conditions include rainfall, snowfall, and fog.

본 발명의 실시예에 따른 학습이미지 DB(1000)는 해상환경에서의 3D 가상모델과 3D 실사모델에 대한 3D 애니메이션 구현을 통해 3D 가상모델 양태이미지와 3D 실사모델 양태이미지가 생성되도록 하는 한편, 도 13에서와 같이 촬영각도 별로 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지가 하나의 3D 가상모델과 하나의 3D 실사모델에 대한 전방위 회전 렌더링을 통해 생성되도록 한다. 즉 하나의 3D 모델에서 수백장 단위의 학습이미지(3D 가상모델 양태이미지와 3D 실사모델 양태이미지)를 생성하게 된다.The learning image DB 1000 according to an embodiment of the present invention generates a 3D virtual model aspect image and a 3D live-action model aspect image through 3D animation implementation for a 3D virtual model and a 3D live-action model in a marine environment. As in 13, the aspect image of the 3D virtual model and the aspect image of the 3D live-action model, which vary for each shooting angle, are generated through omnidirectional rotation rendering of one 3D virtual model and one 3D live-action model. That is, hundreds of learning images (3D virtual model aspect image and 3D live-action model aspect image) are generated from one 3D model.

딥러닝 영상인식 학습모듈(1100)은 딥러닝 영상인식 알고리즘(1110)을 통해 학습이미지 DB(1000)의 3D 가상모델 양태이미지와 3D 실사모델 양태이미지에 대한 기계학습을 수행하여 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 된다. 특히 본 발명의 실시예에 따른 딥러닝 영상인식 학습모듈(1100)은 CNN(Convolutional Neural Network) 기반 딥러닝 영상인식 알고리즘(1110)의 파라미터 조절을 통해 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 되는데, CNN 기반 딥러닝 영상인식 알고리즘(1110)의 파라미터에는 필터(filter), 스트라이드(stride), 패딩(padding)이 포함된다. 여기서 딥러닝 영상인식 학습모듈(1100)의 딥러닝 영상인식 알고리즘(1110)은 기준 픽셀영역(N×N 픽셀, N는 기준 픽셀값) 이상의 크기를 갖는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지만을 식별하도록 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 된다.The deep learning image recognition learning module 1100 performs machine learning on the 3D virtual model aspect image and the 3D live-action model aspect image of the learning image DB 1000 through the deep learning image recognition algorithm 1110 to identify obstacles to avoid An image recognition algorithm 310 and an image recognition algorithm 320 for identifying marine waste are calculated. In particular, the deep learning image recognition learning module 1100 according to an embodiment of the present invention includes an image recognition algorithm 310 for identifying obstacles to be avoided through parameter adjustment of a CNN (Convolutional Neural Network)-based deep learning image recognition algorithm 1110 and The image recognition algorithm 320 for identifying marine waste is calculated, and parameters of the CNN-based deep learning image recognition algorithm 1110 include a filter, a stride, and a padding. Here, the deep learning image recognition algorithm 1110 of the deep learning image recognition learning module 1100 is a 3D virtual model aspect image and a 3D live-action model aspect image having a size greater than or equal to the reference pixel area (N × N pixels, N is the reference pixel value) An image recognition algorithm 310 for identifying obstacles to be avoided and an image recognition algorithm 320 for identifying marine waste are calculated to identify the bay.

상기와 같이 구성된 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템(1)은 무인선 또는 최소인원이 탑승한 선박 본체에 카메라모듈과 대상체 이격거리 측정장치가 설치되고, 카메라모듈의 360°전방향 영상에 대한 CNN 기반 딥러닝 영상인식 및 레이저 측위를 통해 회피대상 장애물(선박/해상구조물)과 해상폐기물에 대한 인식과 위치정보 검출이 수행되면서 선박 본체의 자율운항과 해상폐기물 수집이 유도되므로, 해양환경의 개선효율 증대 및 개선작업비용 절감이 도모될 수 있게 된다. 또한 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템(1)은 회피대상 장애물 및 해상폐기물의 영상인식이 실사영상과 3D 가상모델로부터 대량으로 생성되는 양태이미지를 학습이미지로 한 딥 러닝을 통해 수행되도록 함으로써 해상폐기물의 인식률 증대 및 인식 정밀도/정확도 향상이 도모될 수 있는 한편 자율운항 안정성 증대가 도모될 수 있게 된다.The deep learning-image recognition-based autonomous navigation ship system 1 for marine waste removal according to an embodiment of the present invention configured as described above includes a camera module and an object separation distance measuring device installed in an unmanned ship or a ship body with a minimum number of people on board. Through CNN-based deep learning image recognition and laser positioning of the 360° omnidirectional image of the camera module, recognition of obstacles to be avoided (ships/offshore structures) and offshore wastes and location information detection are performed, autonomous navigation of the ship body and marine waste collection are induced, so it is possible to increase the improvement efficiency of the marine environment and reduce the improvement work cost. In addition, in the deep learning-image recognition-based autonomous navigation ship system 1 for removing marine waste according to an embodiment of the present invention, the image recognition of obstacles to be avoided and marine waste is mass-generated from live-action images and 3D virtual models. By performing deep learning as a learning image, it is possible to increase the recognition rate of marine waste and improve recognition precision/accuracy, while also increasing the stability of autonomous navigation.

상술한 바와 같은, 본 발명의 실시예에 따른 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템을 상기한 설명 및 도면에 따라 도시하였지만, 이는 예를 들어 설명한 것에 불과하며 본 발명의 기술적 사상을 벗어나지 않는 범위 내에서 다양한 변화 및 변경이 가능하다는 것을 이 분야의 통상적인 기술자들은 잘 이해할 수 있을 것이다.As described above, although the deep learning-image recognition-based autonomous navigation ship system for removing marine waste according to the embodiment of the present invention as described above is illustrated according to the above description and drawings, this is only an example and the technical spirit of the present invention It will be well understood by those skilled in the art that various changes and modifications can be made without departing from the above.

1 : 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템
100 : 선박 본체
110 : 선체 추진장치
120 : 부유물 수집장치
130 : 조타기
140 : 조타기 원격제어기
200 : 카메라모듈
210 : 자동노출 조절유닛
300 : 영상분석장치
310 : 회피대상 장애물 식별용 영상인식 알고리즘
320 : 해상폐기물 식별용 영상인식 알고리즘
400 : 대상체 이격거리 측정장치
400a : 레이저 측위장치
410 : 레이저 조사유닛
411 : 레이저 광원
420 : 레이저 수광유닛
500 : 추진장치 제어장치
600 : 관제센터 서버
700 : 실사영상 데이터셋 DB
710 : 촬영 실사영상 데이터셋 DB
720 : 웹수집 실사영상 데이터셋 DB
800 : 가상모델링영상 데이터셋 DB
900 : 실사모델링영상 데이터셋 DB
1000 : 학습이미지 DB
1100 : 딥러닝 영상인식 학습모듈
1110 : 딥러닝 영상인식 알고리즘1: Deep Learning-Image Recognition-based Self-Operating Vessel System for Marine Waste Removal
100: ship body
110: hull propulsion device
120: floating matter collecting device
130: steering gear
140: steering gear remote controller
200: camera module
210: automatic exposure control unit
300: image analysis device
310: image recognition algorithm for identifying obstacles to be avoided
320: image recognition algorithm for marine waste identification
400: object separation distance measuring device
400a: laser positioning device
410: laser irradiation unit
411: laser light source
420: laser light receiving unit
500: propulsion control device
600: control center server
700: live image data set DB
710: Live-action image data set DB
720: Web-collected live-action image data set DB
800: virtual modeling image dataset DB
900: live-action modeling image dataset DB
1000: Learning image DB
1100: Deep learning image recognition learning module
1110: Deep learning image recognition algorithm

Claims (5)

선체 추진장치(110)가 구비되어 해상에서 운항되고, 부유물 수집장치(120)가 구비되어 해상폐기물을 수집하게 되는 선박 본체(100);
상기 선박 본체(100)의 상부에 시야 간섭없이 설치되어 선박 본체(100)를 중심으로 한 360°전방향 영상을 촬영하게 되는 카메라모듈(200);
상기 카메라모듈(200)로부터 360°전방향 영상을 전달받게 되고, 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)이 구비되어 선박과 해상구조물을 포함하는 회피대상 장애물 및 비닐, 스티로폼, 플라스틱을 포함하는 해상폐기물을 식별하게 되며, 회피대상 장애물 식별정보와 해상폐기물 식별정보를 생성하게 되는 영상분석장치(300);
상기 선박 본체(100)의 상부에 시야 간섭없이 설치되며, 상기 영상분석장치(300)로부터 회피대상 장애물 식별정보와 해상폐기물 식별정보를 전달받게 되고, 회피대상 장애물 식별정보와 해상폐기물 식별정보에 각각 포함된 회피대상 장애물 방향정보와 해상폐기물 방향정보에 맞추어 지향방향을 설정한 다음 선박과 회피대상 장애물 간 이격거리(이하 장애물 거리) 및 선박과 해상폐기물 간 이격거리(이하 부유물 거리)를 측정하게 되는 대상체 이격거리 측정장치(400); 및
상기 선체 추진장치(110)에 대한 동작제어를 통해 상기 선박 본체(100)의 해상 운항을 유도하게 되되, 상기 영상분석장치(300)와 대상체 이격거리 측정장치(400)와 연동되어 장애물 방향정보와 장애물 거리 및 해상폐기물 방향정보와 부유물 거리를 전달받게 되고, 회피대상 장애물 방향정보와 장애물 거리 및 해상폐기물 방향정보와 부유물 거리에 맞추어 상기 선체 추진장치(110)에 대한 동작제어를 수행하는 추진장치 제어장치(500);
해상 환경에서의 회피대상 장애물과 해상폐기물의 실사영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 실사영상 데이터셋 DB(700);
해상 환경에서의 회피대상 장애물과 해상폐기물의 3D 가상모델로 구성되는 가상모델링영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 가상모델링영상 데이터셋 DB(800);
상기 실사영상 데이터셋 DB(700)의 실사영상 데이터셋에 대한 3D 실사모델링에 의해 생성되는 3D 실사모델로 구성되는 실사모델링영상 데이터셋의 수집, 저장, 데이터베이스화가 수행되는 실사모델링영상 데이터셋 DB(900);
상기 가상모델링영상 데이터셋 DB(800)와 실사모델링영상 데이터셋 DB(900)로부터 전달되는 3D 가상모델과 3D 실사모델 각각에 대하여 해상환경조건, 기상조건, 주야간 조건, 촬영각도 별로 달라지는 3D 가상모델 양태이미지와 3D 실사모델 양태이미지를 생성시켜 학습이미지 데이터로 데이터베이스화하게 되되, 상기 해상환경조건에는 풍향, 풍속, 파랑이 포함되고, 상기 기상조건에는 강우량, 강설량, 안개가 포함되는 학습이미지 DB(1000);
딥러닝 영상인식 알고리즘(1110)을 통해 상기 학습이미지 DB(1000)의 3D 가상모델 양태이미지와 3D 실사모델 양태이미지에 대한 기계학습을 수행하여 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 되는 딥러닝 영상인식 학습모듈(1100);을 포함하는 구성으로 이루어져 회피대상 장애물을 회피하면서 해상폐기물이 위치한 영역으로 이동하여 해상폐기물을 수집하게 되는 것을 특징으로 하는 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템.The ship body 100 is provided with a hull propulsion device 110 and is operated in the sea, and a floating object collection device 120 is provided to collect marine waste;
a camera module 200 that is installed on the upper portion of the ship body 100 without visual interference and takes a 360° omnidirectional image centered on the ship body 100;
A 360° omnidirectional image is received from the camera module 200, and an image recognition algorithm 310 for identifying obstacles to be avoided and an image recognition algorithm 320 for identifying marine waste are provided to avoid including ships and offshore structures. An image analysis device 300 that identifies target obstacles and marine wastes including vinyl, Styrofoam, and plastics, and generates avoidance target obstacle identification information and marine waste identification information;
It is installed on the upper part of the ship body 100 without visual interference, and receives obstacle identification information and marine waste identification information to be avoided from the image analysis device 300, and to the obstacle identification information to be avoided and marine waste identification information, respectively. After setting the direction according to the included obstacle direction information and marine waste direction information, the separation distance between the vessel and the obstacle to be avoided (hereinafter referred to as the obstacle distance) and the separation distance between the vessel and the marine waste (hereinafter the floating object distance) are measured. Object separation distance measuring device 400; and
The maritime navigation of the ship body 100 is induced through the operation control of the hull propulsion device 110, and the image analysis device 300 and the object separation distance measuring device 400 are interlocked to provide obstacle direction information and Propulsion device control that receives obstacle distance, marine waste direction information, and float distance, and controls the operation of the hull propulsion device 110 according to the obstacle direction information to be avoided, the obstacle distance, and the marine waste direction information and the floating object distance device 500;
The actual image data set DB 700 for collecting, storing, and databaseizing the actual image data set of obstacles to be avoided and marine waste in the marine environment;
A virtual modeling image dataset DB 800 for collecting, storing, and databaseizing a virtual modeling image dataset composed of a 3D virtual model of an obstacle to be avoided and a marine waste in a marine environment;
A live-action modeling image dataset DB ( 900);
3D virtual model that is different for each of the 3D virtual model and the 3D live-action model delivered from the virtual modeling image dataset DB 800 and the live-action modeling image dataset DB 900, depending on marine environmental conditions, weather conditions, day/night conditions, and shooting angles A mode image and a 3D live-action model mode image are created and converted into a database as learning image data, wherein the marine environmental conditions include wind direction, wind speed, and waves, and the weather conditions include rainfall, snowfall, and fog. Learning image DB ( 1000);
Through the deep learning image recognition algorithm 1110, machine learning is performed on the 3D virtual model aspect image of the learning image DB 1000 and the 3D live-action model aspect image to identify the obstacle to be avoided image recognition algorithm 310 and marine waste The deep learning image recognition learning module 1100 that calculates the image recognition algorithm 320 for identification is configured to include; while avoiding the obstacle to be avoided, it moves to the area where the marine waste is located and collects the marine waste Deep learning-image recognition-based autonomous navigation ship system for marine waste removal. 제 1항에 있어서,
상기 선박 본체(100)와의 원격 무선통신으로 상기 선체 추진장치(110)에 대한 원격 동작제어를 수행하게 되는 관제센터 서버(600);를 더 포함하되,
상기 영상분석장치(300)와 추진장치 제어장치(500)는 상기 선박 본체(100)와 관제센터 서버(600) 중에서 선택된 어느 하나에 설치되는 것을 특징으로 하는 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템.The method of claim 1,
A control center server 600 that performs remote operation control for the hull propulsion device 110 through remote wireless communication with the ship body 100; further comprising,
The image analysis device 300 and the propulsion device control device 500 are installed in any one selected from the ship body 100 and the control center server 600 for deep learning-image recognition-based marine waste removal Autonomous ship system. 삭제delete 제 1항에 있어서,
상기 실사영상 데이터셋 DB(700)는,
실사 촬영으로 생성된 실사 영상데이터셋이 저장, 데이터베이스화되는 촬영 실사영상 데이터셋 DB(710);와 웹 크롤링(web crawling)으로 수집된 웹수집 실사영상 데이터셋이 저장, 데이터베이스화되는 웹수집 실사영상 데이터셋 DB(720);를 포함하는 것을 특징으로 하는 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템.The method of claim 1,
The live-action image data set DB 700,
The real-life image data set created by photo-realistic shooting is stored and databased, and the photo-realistic image data set DB 710; and the web-collected live-action image data set collected by web crawling are stored and converted into a database. Image dataset DB (720); Deep learning-image recognition-based autonomous navigation ship system for removing marine waste, characterized in that it includes. 제 1항에 있어서,
상기 딥러닝 영상인식 학습모듈(1100)은 CNN(Convolutional Neural Network) 기반 딥러닝 영상인식 알고리즘(1110)의 파라미터 조절을 통해 회피대상 장애물 식별용 영상인식 알고리즘(310)과 해상폐기물 식별용 영상인식 알고리즘(320)을 산출하게 되되,
CNN 기반 딥러닝 영상인식 알고리즘(1110)의 파라미터에는 필터(filter), 스트라이드(stride), 패딩(padding)이 포함되는 것을 특징으로 하는 딥러닝-영상인식 기반 해상폐기물 제거용 자율운항 선박 시스템.The method of claim 1,
The deep learning image recognition learning module 1100 includes an image recognition algorithm 310 for identifying obstacles to be avoided through parameter adjustment of a CNN (Convolutional Neural Network)-based deep learning image recognition algorithm 1110 and an image recognition algorithm for identifying marine waste. (320) is calculated,
The parameters of the CNN-based deep learning image recognition algorithm 1110 include a filter, a stride, and padding.
KR1020190084762A 2019-07-12 2019-07-12 Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition KR102339465B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020190084762A KR102339465B1 (en) 2019-07-12 2019-07-12 Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020190084762A KR102339465B1 (en) 2019-07-12 2019-07-12 Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition

Publications (2)

Publication Number Publication Date
KR20210007767A KR20210007767A (en) 2021-01-20
KR102339465B1 true KR102339465B1 (en) 2021-12-15

Family

ID=74304586

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020190084762A KR102339465B1 (en) 2019-07-12 2019-07-12 Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition

Country Status (1)

Country Link
KR (1) KR102339465B1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102578809B1 (en) * 2022-11-23 2023-09-15 주식회사 제이디 Method for providing data of floating object using autonomous water drone and system thereof

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102606919B1 (en) * 2021-03-24 2023-11-24 김대인 Autonomous marine waste collecton system
CN113344885B (en) * 2021-06-15 2024-07-02 温州大学 River float detection method based on cascade convolution neural network
CN114063619B (en) * 2021-11-15 2023-09-19 浙江大学湖州研究院 Unmanned ship obstacle detection and breaking method based on carpet type scanning mode
CN114872842A (en) * 2022-05-13 2022-08-09 中交天津航道局有限公司 Intelligent collecting and processing system and method for water hyacinth in water area
CN114802617B (en) * 2022-06-09 2023-03-21 生态环境部华南环境科学研究所(生态环境部生态环境应急研究所) Pollutant salvaging method for water ecological environment treatment
CN115223162A (en) * 2022-08-22 2022-10-21 北京海舶无人船科技有限公司 Floater identification method and system
CN116280039A (en) * 2022-09-07 2023-06-23 广东工业大学 Unmanned ship garbage recycling method and system
CN116503767B (en) * 2023-06-02 2023-09-22 合肥众安睿博智能科技有限公司 River course floater recognition system based on semantic image processing

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101805564B1 (en) * 2017-06-12 2018-01-18 (주)지인테크 Alarm system for prevent ship collision and method thereby
KR101880437B1 (en) * 2016-10-28 2018-08-17 한국해양과학기술원 Unmanned surface vehicle control system for providing wide viewing angle using real camera image and virtual camera image
KR101985436B1 (en) * 2019-01-04 2019-06-03 (주)미래해양 Cleaning unit using the deep-learning

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR200324102Y1 (en) 2003-03-14 2003-08-21 주식회사 건화엔지니어링 Barge for collect of waste of dam and sea
KR101803364B1 (en) 2016-07-04 2017-11-30 주식회사 유경시스템 Suspended matter removal apparatus of the surface of the water

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101880437B1 (en) * 2016-10-28 2018-08-17 한국해양과학기술원 Unmanned surface vehicle control system for providing wide viewing angle using real camera image and virtual camera image
KR101805564B1 (en) * 2017-06-12 2018-01-18 (주)지인테크 Alarm system for prevent ship collision and method thereby
KR101985436B1 (en) * 2019-01-04 2019-06-03 (주)미래해양 Cleaning unit using the deep-learning

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102578809B1 (en) * 2022-11-23 2023-09-15 주식회사 제이디 Method for providing data of floating object using autonomous water drone and system thereof
WO2024111761A1 (en) * 2022-11-23 2024-05-30 주식회사 제이디 Apparatus for providing marine floating object data by using autonomous water drone, and method therefor

Also Published As

Publication number Publication date
KR20210007767A (en) 2021-01-20

Similar Documents

Publication Publication Date Title
KR102339465B1 (en) Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition
KR102231286B1 (en) Ship surrounding monitoring device and ship surrounding monitoring method
KR102112935B1 (en) Path planning method using obstacle map
Bovcon et al. WaSR—A water segmentation and refinement maritime obstacle detection network
KR102235787B1 (en) Device and method for monitoring a berthing
Bonnin-Pascual et al. On the use of robots and vision technologies for the inspection of vessels: A survey on recent advances
Maki et al. Tracking a sea turtle by an AUV with a multibeam imaging sonar: Toward robotic observation of marine life
US20220024549A1 (en) System and method for measuring the distance to an object in water
KR101888171B1 (en) Method and device for recognizing environment based on position information of unmanned surface vessel
KR102466804B1 (en) Autonomous navigation method using image segmentation
Livanos et al. Intelligent navigation and control of a prototype autonomous underwater vehicle for automated inspection of aquaculture net pen cages
Leedekerken et al. Mapping complex marine environments with autonomous surface craft
CN112146654B (en) Foresight imaging sonar underwater positioning and navigation method based on key constraint frame
Thompson et al. Efficient lidar-based object segmentation and mapping for maritime environments
CN113010958B (en) Simulation system of self-propelled ship and operation method thereof
Maki et al. Low-altitude and high-speed terrain tracking method for lightweight AUVs
CN116255908B (en) Underwater robot-oriented marine organism positioning measurement device and method
Maki et al. Autonomous tracking of sea turtles based on multibeam imaging sonar: Toward robotic observation of marine life
KR20220055555A (en) Method and device for monitoring harbor and ship
KR102278674B1 (en) Method and device for monitoring harbor and ship considering sea level
Karapetyan et al. Human diver-inspired visual navigation: Towards coverage path planning of shipwrecks
JP2023072588A (en) Unmanned maintenance system and unmanned maintenance method
Cardaillac et al. Application of maneuvering based control for autonomous inspection of aquaculture net pens
Ruscio et al. Autonomous boundary inspection of Posidonia oceanica meadows using an underwater robot
CN116859948A (en) Autonomous navigation control method and system for unmanned ship for channel sweep based on target detection algorithm

Legal Events

Date Code Title Description
E701 Decision to grant or registration of patent right