WO2024100616A1 - Marqueur de repère multimodal, appareil de perception hétérogène et système multimodal comprenant les deux - Google Patents

Marqueur de repère multimodal, appareil de perception hétérogène et système multimodal comprenant les deux Download PDF

Info

Publication number
WO2024100616A1
WO2024100616A1 PCT/IB2023/061384 IB2023061384W WO2024100616A1 WO 2024100616 A1 WO2024100616 A1 WO 2024100616A1 IB 2023061384 W IB2023061384 W IB 2023061384W WO 2024100616 A1 WO2024100616 A1 WO 2024100616A1
Authority
WO
WIPO (PCT)
Prior art keywords
marker
component
vehicle
thermal
target location
Prior art date
Application number
PCT/IB2023/061384
Other languages
English (en)
Inventor
Rafael Marques Claro
Diogo Alexandre Brandão Da Silva
Andry MAYKOL GOMES PINTO
Original Assignee
Inesc Tec - Instituto De Engenharia De Sistemas E Computadores, Tecnologia E Ciência
Universidade Do Porto
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from EP22212945.4A external-priority patent/EP4369308A1/fr
Application filed by Inesc Tec - Instituto De Engenharia De Sistemas E Computadores, Tecnologia E Ciência, Universidade Do Porto filed Critical Inesc Tec - Instituto De Engenharia De Sistemas E Computadores, Tecnologia E Ciência
Publication of WO2024100616A1 publication Critical patent/WO2024100616A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G06V10/245—Aligning, centring, orientation detection or correction of the image by locating a pattern; Special marks for positioning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/17—Terrestrial scenes taken from planes or by drones
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects

Definitions

  • the present disclosure relates generally to detection systems and methods. More particularly, the present application relates to systems, methods and devices for aid unmanned or manned vehicles in detecting target locations.
  • Unmanned vehicles are autonomous robots that have been used in several applications due to the ability to maintain a very stable navigation, and to approach not easily access areas, resulting in the collection of high-quality data.
  • Said vehicles may be of an aquatic, ground-based or air-based type and its main application areas include several industries, like the cinematographic, military, agricultural and surveillance.
  • several inspection tasks are being carried out by said robotic vehicles, representing a reduction of the costs and of the total time consumed.
  • the usual cycle of a robotic vehicle mission includes its departure from a base station, the execution of its task, and the returning to the base station or another target location.
  • An important feature that allows the autonomous operation is the capability of the vehicle to precise landing/docking on a predefined target location.
  • a crucial task for enabling the vehicle to successfully land/dock on the desired target location is its ability to autonomously detect and recognize said target location in real-time. If the vehicle misses the target location, it can compromise the continuity of the mission, as well as damaging the vehicle equipment.
  • Circular shaped markers such as the CCTag and the STag, are also used in several robotics applications. Although there are optimization techniques to improve the detectability of these markers, their major drawback is that they are dependent on lighting and environment conditions, only performing efficiently indoors or in daylight with favourable weather.
  • the present document discloses a multimodal fiducial marker for relative pose estimation comprising a first component and a second component arranged to provide a surface comprising a first section of the first component and a second section of the second component; a heat source arranged to heat the surface by thermal conduction through said components; the first section having a reflectivity index different of a reflectivity index of the second section; and, the first component having a thermal conductivity coefficient different of the thermal conductivity coefficient of the second component; wherein said sections are arranged as a pattern of geometric shapes encoding data.
  • the marker according to the previous claim wherein the pattern of geometric shapes is a binary code, in particular, a thermal light-retroreflective binary identification pattern.
  • the marker according to any of the previous claims wherein said reflectivity index is a visible-light reflectivity index is a visible-light reflectivity index.
  • the marker according to any of the previous claims wherein said surface is substantially non reflective in the infrared-light spectrum.
  • Figure 1 representation of an embodiment of the multimodal fiducial marker of the present application, where the numerical reference signs represent: 1 - marker's surface first section; 2 - marker's surface second section; 3 - heat source; 4 - multimodal fiducial marker.
  • Figure 2 representation of an embodiment of the heterogeneous perception apparatus of the multimodal system described in the present application, where the numerical reference signs represent: 5 - 3D-LiDAR unit; 6 - visual camera unit; 7 - thermal camera unit; 8 - heterogeneous perception apparatus.
  • Figure 3 illustrative example of the multimodal system described in the present application, where the numerical reference signs represent: 4 - multimodal fiducial marker; 8 - heterogeneous perception apparatus; 9 - unmanned Vehicle (air-based type); 10 - target location.
  • Figure 4 flowchart depicting the position estimation procedure executed by the heterogeneous perception apparatus, in an embodiment of the multimodal system described in the present application, where the numerical reference signs represent: 5 - 3D-LiDAR unit; 6 - visual camera unit; 7 - thermal camera unit; 8 - heterogeneous perception apparatus; 8.1 - processor-based device; 9 - unmanned Vehicle (air-based type); 10 - target location.
  • Figures 5A, 5B thermal images tests comparing an acrylic marker with the multimodal fiducial marker, where the numerical reference signs represent: 4 - multimodal fiducial marker, 503 - acrylic marker, and 505 - a hot spot.
  • Figure 6 thermal images tests for testing the IR reflection, where the numerical reference sign represents: 601 - an IR reflection.
  • Figure 7 illustrative acrylic marker and the multimodal fiducial marker, where the numerical reference signs represent: 4 - multimodal fiducial marker, and 503 - acrylic marker.
  • Such system therefore provides the ability to operate under severe environment and light conditions (such as at different heights, with intense sunlight, no-light or in dark environments; thus, it includes operations subject to rain and fog), by means of a multimodal approach that resorts to photometric and radiometric data, in order to perform a robust, redundant and reliable detection of the vehicle's target location.
  • the system includes at least one multimodal fiducial marker and a heterogeneous perception apparatus to be coupled to the vehicle.
  • the multimodal fiducial marker can be detected and localized through the analysis of visual, thermal and point cloud data. This is an active marker that is able to improve the relative localization of vehicles, which is very relevant for navigational manoeuvres, especially in unmanned vehicles.
  • the heterogeneous perception apparatus collects both photometric and radiometric data, resorting to cameras and range sensors. Data is fused and combined together by means of a particular method also described in the present application.
  • the multimodal system of the present application improves the situational awareness of the vehicles, increasing its detection capabilities and navigational abilities.
  • the system is comprised by: at least one multimodal fiducial marker as described in the present application; each marker positioned at a target location; and at least one heterogeneous perception apparatus as described in the present application.
  • the multimodal fiducial marker of the present application is adapted to create an unique thermal retroreflective binary identification pattern.
  • the marker comprises: a surface of a defined geometry, having a first section of a first component and at least a second section of a second component.
  • Such sections are arranged in order to form the marker's layout which is configured to encode an unique identification pattern in binary code, each of the first and the second components being of a different binary colour.
  • the first component has a reflectivity index different from the reflectivity index of the second component; and a heat source operable to heat the marker's surface; and wherein, the first and the second components have a different thermal conductivity coefficient.
  • the heterogeneous perception apparatus of the present application is adapted to be coupled to a vehicle and configured to detect the multimodal fiducial marker described in the present application that is positioned in a vehicle's target location.
  • the apparatus comprises: a visual camera unit configured to collect image data in a visible light spectrum of a target location area and to estimate the marker's pose relative to a visual camera unit's coordinate system.
  • a thermal camera unit configured to collect both thermal and radiometric data of the target location area and to estimate the marker's pose relative to a thermal camera unit's coordinate system.
  • a 3D-LiDAR unit configured to collect range and radiometric data from the target location area and to estimate the marker's pose relative to a 3D-LiDAR unit's coordinate system.
  • a processor-based device programmed to process marker's pose estimation data obtained by the visual, thermal and 3D-LiDAR units, to determine a relative pose estimation between the apparatus and the marker, and to determine the location of the marker; the relative positioning between the visual and thermal camera unit and the 3D- LiDAR unit being known; and operable to transmit said location information to the Vehicle.
  • a method detecting multimodal fiducial markers using the heterogeneous perception apparatus coupled to a Vehicle comprises the following steps: scanning a target location area using the heterogeneous perception apparatus to obtain target location data; identifying a marker from the target location data; collecting image, thermal, radiometric and range data from the marker using the sensor units of the apparatus; estimating a marker's pose relative to the coordinate system of each sensor units of the apparatus; determining the relative pose estimation between the apparatus and the marker based on the estimated poses of the marker obtained in iv., and determine the correspondent location of the marker; transmit the location information of the marker to the Vehicle.
  • Figure 1 shows a representation of an embodiment of the multimodal fiducial marker of the present application.
  • a multimodal fiducial marker (4) for relative pose estimation of a vehicle.
  • the marker (4) is operational for any type of vehicle (9), whether manned or unmanned, aquatic, ground-based or air-based type.
  • the marker comprises a surface of a defined geometry, that is, a geometry that is pre-defined and is recognizable by a perception apparatus and by the relative pose detection and estimation algorithms.
  • the marker (4) has a quadrangular shape with dimensions 0.22 x 0.22 x 0.02 meters, being detectable by first extracting the edges of an image collected by the apparatus' visual camera, followed by filtering the counters to make up a polygon with four vertices. Circular shaped markers may also be used.
  • the marker's surface has a first section (1) of a first component and at least a second section (2) of a second component, such sections and respective components being of a different binary colour and are arranged in such a way to form a marker's specific layout that is able to encode a unique identification pattern in binary code. Additionally, the first component has a reflectivity index different from the reflectivity index of the second component.
  • the marker (4) also comprises a heat source (3) operable to heat the marker's surface, and wherein the first and the second components have a different thermal conductivity coefficient.
  • the marker (4) is an active marker that is adapted to create a unique thermal retroreflective binary identification pattern, allowing it to be detectable and localizable through the analysis of visual, thermal and point cloud data.
  • This active marker improves the relative localization of vehicles (9) endowed with the heterogeneous perception apparatus (8) of the present application, in particular for the precise landing or docking maneuvers (depending on the type of vehicle (9)).
  • the marker (4) of the present application its surface's geometry is a planar geometry. More particularly, the first section (1) and at least the second section (2) are arranged on the two-dimensional surface of the marker (4). Consequently, the marker (4) is adapted to create a two-dimensional thermal retroreflective binary identification pattern.
  • the marker's geometry is a spatial geometry. More particularly, the first and at least the second sections (1, 2) are arranged as a multiplicity of differently shapes. Consequently, the marker (4) is adapted to create a three-dimensional thermal retroreflective binary identification pattern.
  • the binary code used to encode the marker's unique identification pattern can be any one of a library of binary code, such as an ArUco or an AprilTAg or an ARTag code.
  • the first component is of a white colour and the second component is of a black colour.
  • the first component is of a blue colour and the second component is of a red colour.
  • Other combination of colours for the first and second components are presented by way of example: green/blue, yellow/brown or light gray/dark gray.
  • the first component has a reflectivity index of at least 70% and the second component is of a non-retroreflective type.
  • the first component is comprised by a layer of retroreflective material having a reflectivity index of at least 70%, said layer of retroreflective material being applied on top of at least a first component's material.
  • Said first component's material may have a thermal conductivity of at least 88 W/m-lK-1.
  • the second component it is comprised by at least one material of a non-retroreflective type, said material having a maximum thermal conductivity of 0.38 W/mlK-1.
  • the first material is aluminium and the second material is cork.
  • the heat source (3) is operable to heat the marker's surface to at least a temperature of 100° C.
  • the heat source (3) may be an electric heated bed which is powered by an electrical source plug or by a battery unit.
  • an heterogeneous perception apparatus (8) to be coupled to a vehicle (9) and configured to detect the multimodal fiducial marker (4) already described, said marker (4) being positioned in a vehicle's target location (10).
  • the apparatus (8) is adapted to perceive multimodal information, to allow the vehicle (9) to which it is coupled to successfully land/dock in adverse environments. Moreover, the apparatus is designed for the severe offshore environment and obtains both photometric and radiometric data, such as visual, thermal and point cloud information.
  • the apparatus comprises: a visual camera unit (6) configured to collect image data in a visible light spectrum of a target location area and to estimate the marker's pose relative to a visual camera unit's coordinate system; a thermal camera unit (7) configured to collect both thermal and radiometric data of the target location area and to estimate the marker's pose relative to a thermal camera unit's coordinate system; a 3D-LiDAR unit (5) configured to collect range and radiometric data from the target location area and to estimate the marker's pose relative to a 3D-LiDAR unit's coordinate system; a processor-based device (8.1): programmed to process marker's pose estimation data obtained by the visual, thermal and 3D-LiDAR units (6, 7, 5), to determine a relative pose estimation between the apparatus (8) and the marker (4), and to determine the location (10) of the marker (4); the relative positioning between the visual and thermal camera unit (6, 7) and the 3D-LiDAR unit (5) being known; and operable to transmit said location information to the vehicle (9
  • the visual camera (6) collects images in the visible light spectrum and the thermal camera (7) gathers both thermal and radiometric information of the scene, it represents a more robust sensory approach that does not depend on the lighting conditions.
  • the 3D LiDAR (5) directly acquires range data from the surrounding environment using laser beams, that are represented in point clouds. While the camera sensors (6) collect denser data for close range procedures, the 3D LiDAR (5) has a larger field of view and range, suitable for long range operations.
  • the apparatus (8) not only allows to collect multimodal and complementary information about a target location, but also, being coupled to the vehicle (9), plays an important and significant role in the navigation manoeuvres, increasing situational awareness of a scenario of operation, contributing to a safer operation of the vehicle (9).
  • the pose estimation given by every sensor (5, 6, 7) needs to be fused to output a single and redundant localization of the detected marker. Forthat purpose, a weighted average is applied that ensures a short processing time and increases the computational efficiency for an embedded system, guaranteeing a real time detection and relative pose estimation.
  • X F . x y , x z ) is the position estimation of the apparatus (8) in relation to the marker (4);
  • Xi is a Boolean variable that equals 1 when its correspondent sensor detects the marker (4) and equals 0 otherwise; and w t is a dynamic weight, calculated according to the following expression: wherein, e represents the mean error and a the standard deviation.
  • a method for detecting multimodal fiducial markers (4) comprising: scanning a target location area using the heterogeneous perception apparatus (8) to obtain target location data; identifying a marker (4) from the target location data; collecting image, thermal, radiometric and range data from the marker (4) using the sensor units (5, 6, 7) of the apparatus (8); estimating a marker's pose relative to the coordinate system of each sensor units (5, 6, 7) of the apparatus (8); determining the relative pose estimation between the apparatus (8) and the marker (4) based on the estimated poses of the marker (4) obtained in iv., and determine the correspondent location (10) of the marker (4); transmit the location information of the marker (4) to the vehicle (9).
  • the present application also describes a multimodal system comprising: at least one multimodal fiducial marker (4) as described in the present application; each marker (4) positioned at a target location (10); and at least one heterogeneous perception apparatus as described in the present application.
  • the system comprises one or more vehicles (9), each vehicle (9) having coupled one heterogeneous perception apparatus, and the system being configured to operate according to the method for detecting multimodal fiducial markers described in the present application.
  • the vehicle (9) may be of an unmanned or manned type and being of an aquatic, ground-based or air-based type.
  • the vehicle (9) is a drone or a vessel or an automated guided vehicle.
  • the drone (9) has the apparatus (8) coupled to its structure, enabling the detection of the marker (4) placed in a specific target location (10), landing and delivery of the package. Another possibility consists of detecting the target (10) and dropping the package in the air (with or without a parachute). [0059] Windfarm inspection:
  • the use of the system enables accurate relative localization of the docking station (10) for both surface vessels and ground vehicles (9), such as rovers and AGVs, assisting in the docking manoeuvre.
  • Tests were performed comparing an acrylic marker of a 4mm thick plate with the disclosed multimodal fiducial marker for the same dimensions, code, and thermal bed as the heat source.
  • the thermal radiation reflection test was carried out using a soldering iron as a hot body ( ⁇ 450 °C), which was moved above the marker so that the cameras only picked up indirect/reflected radiation.
  • acrylic fiducial markers More generally there are several problems with acrylic fiducial markers, namely: they are not mechanically robust to heating, as they can warp, nor to atmospheric conditions in outdoor environments; although the acrylic acts as a filter at first, it is continuously heating up, specially the center of the acrylic marker, and therefore the temperature contrast tends to get significantly worse over time; furthermore acrylic is not a visual marker and is not radiometric at the frequency of LiDARs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Remote Sensing (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

La présente invention concerne un système multimodal pour un véhicule pour détecter et naviguer avec précision vers un emplacement cible. Le système comprend au moins un marqueur de repère multimodal (4) et un appareil de perception hétérogène (8), à coupler à un véhicule avec ou sans pilote (9). Le marqueur (4) peut être détecté par l'analyse de données visuelles, thermiques et de nuage de points. L'appareil de perception hétérogène (8) collecte à la fois des données photométriques et radiométriques, reposant sur des caméras et des capteurs de portée. Ce système multimodal améliore la perception situationnelle du véhicule (9), augmentant ses capacités de détection et ses capacités de navigation. Par conséquent, le système (4, 8) fournit au véhicule (9) la capacité de fonctionner dans des conditions d'environnement de serveur, au moyen d'une approche multimodale afin d'effectuer une détection robuste, redondante et fiable de l'emplacement cible du véhicule (10).
PCT/IB2023/061384 2022-11-10 2023-11-10 Marqueur de repère multimodal, appareil de perception hétérogène et système multimodal comprenant les deux WO2024100616A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
PT118328 2022-11-10
PT11832822 2022-11-10
EP22212945.4 2022-12-12
EP22212945.4A EP4369308A1 (fr) 2022-11-10 2022-12-12 Un marqueur fiduciaire multimodal, un appareil de perception hétérogène et un système multimodal comprenant les deux

Publications (1)

Publication Number Publication Date
WO2024100616A1 true WO2024100616A1 (fr) 2024-05-16

Family

ID=89121872

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2023/061384 WO2024100616A1 (fr) 2022-11-10 2023-11-10 Marqueur de repère multimodal, appareil de perception hétérogène et système multimodal comprenant les deux

Country Status (1)

Country Link
WO (1) WO2024100616A1 (fr)

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
MARIUS BEUL ET AL: "Fast Autonomous Flight in Warehouses for Inventory Applications", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 18 September 2018 (2018-09-18), XP081084321, DOI: 10.1109/LRA.2018.2849833 *
RAEI HAMIDREZA ET AL: "Autonomous landing on moving targets using LiDAR, Camera and IMU sensor Fusion", 2022 13TH ASIAN CONTROL CONFERENCE (ASCC), ACA, 4 May 2022 (2022-05-04), pages 419 - 423, XP034152449, DOI: 10.23919/ASCC56756.2022.9828342 *
SHEHRYAR KHATTAK ET AL: "Marker based Thermal-Inertial Localization for Aerial Robots in Obscurant Filled Environments", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 2 March 2019 (2019-03-02), XP081125243, DOI: 10.1007/978-3-030-03801-4_49 *

Similar Documents

Publication Publication Date Title
Alam et al. A survey of safe landing zone detection techniques for autonomous unmanned aerial vehicles (UAVs)
Kalinov et al. High-precision uav localization system for landing on a mobile collaborative robot based on an ir marker pattern recognition
US20180273173A1 (en) Autonomous inspection of elongated structures using unmanned aerial vehicles
Huh et al. Integrated navigation system using camera and gimbaled laser scanner for indoor and outdoor autonomous flight of UAVs
Breitenmoser et al. A monocular vision-based system for 6D relative robot localization
CN109885086B (zh) 一种基于复合多边形标志引导的无人机垂直降落方法
Nieuwenhuisen et al. Multimodal obstacle detection and collision avoidance for micro aerial vehicles
US11614743B2 (en) System and method for navigating a sensor-equipped mobile platform through an environment to a destination
CN109376587A (zh) 基于物联网的检测查勘通信铁塔智能巡检系统和方法
EP2439605A2 (fr) Navigation de dispositifs mobiles
CN106408601A (zh) 一种基于gps的双目融合定位方法及装置
Claro et al. Artuga: A novel multimodal fiducial marker for aerial robotics
JP2019050007A (ja) 移動体の位置を判断する方法および装置、ならびにコンピュータ可読媒体
Oszust et al. A vision-based method for supporting autonomous aircraft landing
Lin et al. Real-time 6DoF deck pose estimation and target tracking for landing an UAV in a cluttered shipboard environment using on-board vision
CN112074706A (zh) 精确定位系统
EP4369308A1 (fr) Un marqueur fiduciaire multimodal, un appareil de perception hétérogène et un système multimodal comprenant les deux
WO2024100616A1 (fr) Marqueur de repère multimodal, appareil de perception hétérogène et système multimodal comprenant les deux
CN116957360A (zh) 一种基于无人机的空间观测与重建方法及系统
KR101537324B1 (ko) 영상기반 비행체 자동 이착륙 유도 시스템
CN111402324A (zh) 一种目标测量方法、电子设备以及计算机存储介质
CN115755950A (zh) 一种基于激光雷达和相机数据融合的无人机定点降落方法
Bostelman et al. Obstacle detection using a time-of-flight range camera for automated guided vehicle safety and navigation
Dias et al. Thermographic and visible spectrum camera calibration for marine robotic target detection
Moura et al. Autonomous UAV Landing Approach for Marine Operations