EP3659116A1 - Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method - Google Patents

Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method

Info

Publication number
EP3659116A1
EP3659116A1 EP18837761.8A EP18837761A EP3659116A1 EP 3659116 A1 EP3659116 A1 EP 3659116A1 EP 18837761 A EP18837761 A EP 18837761A EP 3659116 A1 EP3659116 A1 EP 3659116A1
Authority
EP
European Patent Office
Prior art keywords
workspace
remotely operated
video feed
location
operated vehicle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP18837761.8A
Other languages
German (de)
French (fr)
Other versions
EP3659116A4 (en
Inventor
Lyman J. Petrosky
Martin Bolander
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Westinghouse Electric Co LLC
Original Assignee
Westinghouse Electric Co LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Westinghouse Electric Co LLC filed Critical Westinghouse Electric Co LLC
Priority to EP24210701.9A priority Critical patent/EP4524679A1/en
Publication of EP3659116A1 publication Critical patent/EP3659116A1/en
Publication of EP3659116A4 publication Critical patent/EP3659116A4/en
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G01C21/206—Instruments for performing navigational calculations specially adapted for indoor navigation
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G01C21/203—Instruments for performing navigational calculations specially adapted for water-borne vessels
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0033—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/04—Control of altitude or depth
    • G05D1/06—Rate of change of altitude or depth
    • G05D1/0692—Rate of change of altitude or depth specially adapted for under-water vehicles
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/24—Arrangements for determining position or orientation
    • G05D1/244—Arrangements for determining position or orientation using passive navigation aids external to the vehicle, e.g. markers, reflectors or magnetic means
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/24—Arrangements for determining position or orientation
    • G05D1/247—Arrangements for determining position or orientation using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
    • G05D1/249—Arrangements for determining position or orientation using signals provided by artificial sources external to the vehicle, e.g. navigation beacons from positioning sensors located off-board the vehicle, e.g. from cameras
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/40—Control within particular dimensions
    • G05D1/46—Control of position or course in three dimensions [3D]
    • G05D1/467—Control of position or course in three dimensions [3D] for movement inside a confined volume, e.g. indoor flying
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/60—Intended control result
    • G05D1/656—Interaction with payloads or external entities
    • G05D1/678—Interaction with payloads or external entities for tethered vehicles
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G21—NUCLEAR PHYSICS; NUCLEAR ENGINEERING
    • G21C—NUCLEAR REACTORS
    • G21C17/00—Monitoring; Testing ; Maintaining
    • G21C17/003—Remote inspection of vessels, e.g. pressure vessels
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2105/00—Specific applications of the controlled vehicles
    • G05D2105/80—Specific applications of the controlled vehicles for information gathering, e.g. for academic research
    • G05D2105/89—Specific applications of the controlled vehicles for information gathering, e.g. for academic research for inspecting structures, e.g. wind mills, bridges, buildings or vehicles
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2107/00—Specific environments of the controlled vehicles
    • G05D2107/70—Industrial sites, e.g. warehouses or factories
    • G05D2107/75—Electric power generation plants
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2109/00—Types of controlled vehicles
    • G05D2109/30—Water vehicles
    • G05D2109/38—Water vehicles operating under the water surface, e.g. submarines
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2111/00—Details of signals used for control of position, course, altitude or attitude of land, water, air or space vehicles
    • G05D2111/10—Optical signals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
    • Y02E30/00—Energy generation of nuclear origin
    • Y02E30/30—Nuclear fission reactors

Definitions

  • the present invention is directed to methods of locating a remotely operated vehicle within a workspace.
  • the present invention is also directed to remote inspection systems which employ such methods.
  • ROVs Remotely operated vehicles
  • tasks e.g., repairs, inspections, etc.
  • An example of such use/environment is in inspecting regions within a nuclear reactor vessel which would not only be deadly to a human inspector, but would also destroy many modem electrical components instantly or in a very short period of time.
  • a method of locating a remotely operated vehicle within a three-dimensional workspace comprises: receiving a video feed of the workspace from a video camera; processing the video feed to identify landmarks and features of known physical structures in or near the workspace; determining a correlation between the features and landmarks and known physical structures; calibrating the video feed from the camera to the known physical structures using the correlation; determining the location in the calibrated video feed of a number of fiducial markers which are positioned on the remotely operated vehicle; and determining the three-dimensional position of the remotely operated vehicle within the workspace using the location of the number of fiducial markers in the calibrated video feed.
  • Determining a correlation between the landmarks and features and known physical structures may comprise constructing a three-dimensional reference framework based on previously known dimensional information of the known physical structures.
  • Determining the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle may comprise determining the location of at least three distinct fiducial markers.
  • Determining the location of at least three distinct fiducial markers may comprise determining the location of at least three spherical elements.
  • Determining the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle may comprise determining the location of at least two distinct fiducial markers.
  • a remote inspection system comprises: a remotely operated vehicle operable within a three-dimensional workspace, the remotely operated vehicle having a number of fiducial markers positioned thereon; a video camera positioned at or about a boundary of the workspace, the video camera positioned and structured to capture images of landmarks and features located in or near the workspace; an electronic storage medium having dimensional details of the landmarks and the features stored therein; and a processing device in electrical communication with both the camera and the electronic storage medium, the processing device programmed to: receive images of the workspace in a video feed from the video camera, process the video feed to identify the features and landmarks, determine a correlation between the features and landmarks and known physical structures, calibrate the video feed from the camera to the known physical structures using the correlation, determine the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle, and determine the three- dimensional position of the remotely operated vehicle within the workspace using the location of the number of fiducial markers in the
  • the number of fiducial markers may comprise a plurality of spherical elements.
  • the spherical elements may be of a high contrast color.
  • the workspace may comprises an interior of a nuclear reactor vessel.
  • the video camera may be positioned at or about an upper portion of the workspace.
  • the electronic storage medium and the processing device may comprise elements of a computing device.
  • the computing device may comprise a desktop or laptop computer.
  • the computing device may comprise a handheld computing device.
  • Figure 1 shows a schematic view of a remote inspection system in accordance with an example embodiment of the present invention
  • Figure 2 shows an example screenshot in accordance with an example embodiment of the present invention of a video feed from a camera of a remote inspection system such as shown in Figure 1;
  • Figure 3 shows the general steps of a method of locating a remotely operated vehicle within a workspace in accordance with an example embodiment of the present invention.
  • Remote inspection system 10 includes a remotely operated vehicle (ROV) 14 which is operable within a three-dimensional workspace (e.g., the water-filled interior of reactor vessel 12) as shown generally at 16.
  • ROV remotely operated vehicle
  • workspace shall be used to refer to the three-dimensional space in which an ROV may be operated.
  • ROV 14 is a remotely operated submersible vehicle which is tethered to a remote control system (not shown) via an umbilical 18, which includes a number of communications cables for the transmission of data/control signals to ROV 14 and data/video images from ROV 14. It is to be appreciated that other ROVs (e.g., without limitation flying drones), wired or wirelessly operable, may be employed without varying from the scope of the present invention.
  • ROVs e.g., without limitation flying drones
  • wired or wirelessly operable may be employed without varying from the scope of the present invention.
  • ROV 14 includes a number of fiducial markers 20 positioned thereon (three of which are illustrated in the embodiment of Figure 1).
  • the position and relative spacing (e.g., di and d2 of Figure 2) of fiducial markers 20 is predetermined such that fiducial markers 20 may be used to determine one or more of the orientation and/or angle of ROV 14.
  • spherical elements approximately the size of a golf ball have been utilized. It is to be appreciated that such spherical objects remain the shape/size regardless of viewing angle (when distance is kept constant).
  • markers are preferably of a high contrast color (e.g., without limitation, fluorescent orange).
  • other quantities and or shapes (e.g., without limitation, disk, partial sphere) of fiducial markers may be employed without varying from the scope of the present invention.
  • remote inspection system 10 further includes a video camera 22 in electrical communication with a processing device 24 and a suitable electronic storage medium 26 which may be integral with, or separate from, processing device 24.
  • Video camera 22 is positioned at or about a boundary of workspace 16 and is positioned and structured to provide a video feed, via wired or wireless means, of images of workspace 16, including ROV 14 disposed therein along with landmarks (e.g., wall 30, floor 34) and features of such landmarks (e.g., vertical joints 32 of wall 30 and apertures 36 of floor 34), located in or near workspace 16.
  • Figure 2 shows an example of an image 28 from a video feed captured by camera 22 in accordance of an example embodiment of the present invention.
  • video camera 22 is positioned at or about an upper portion of workspace 16 and is structured to be able to pan and zoom upon request, however, it is to be appreciated that other cameras with other functionality, and positioning may be employed without varying from the scope of the present invention.
  • Processing device 24 may be any suitable programmable analog and/or digital device (including an associated memory part or portion) that can store, retrieve, execute and process data (e.g., software routines and/or information used by such routines), including, without limitation, a computer, a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a programmable system on a chip (PSOC), an application specific integrated circuit (ASIC), a microprocessor, a microcontroller, a programmable logic controller, or any other suitable processing device or apparatus.
  • FPGA field programmable gate array
  • CPLD complex programmable logic device
  • PSOC programmable system on a chip
  • ASIC application specific integrated circuit
  • Electronic storage medium 26 can be any one or more of a variety of types of internal and/or external storage media such as, without limitation, RAM, ROM, EPROM(s), EEPROM(s), FLASH, and the like that provide a storage register, i.e., a non-transitory machine readable medium, for data and program code storage such as in the fashion of an internal storage area of a computer, and can be volatile memory or nonvolatile memory.
  • Electronic storage medium 26 includes dimensional details of the landmarks and the features stored therein (e.g., distance between vertical joints 32, diameter of apertures 36, spacing between apertures 36, etc.).
  • Electronic storage medium 26 may be integral with processing unit 24.
  • processing device 24 and electronic storage medium 26 comprise elements of a computing device such as, without limitation, a desktop or laptop computer, or a handheld computing device (e.g., without limitation a tablet computer or a smartphone).
  • Method 100 begins at step 102 where a video feed of workspace 16 is received from video camera 22.
  • step 104 the video feed is processed to identify landmarks 30, 34 and features 32, 36 thereof in or near workspace 16.
  • step 104 After the landmarks and features in or near the workspace have been identified in step 104, dimensional data of such known physical structures which was stored in advance of such time in electronic storage medium 26 is used to determine a correlation between the landmarks and features identified in the video feed and known physical structures, such as shown in step 106.
  • a correlation is determined by constructing a three-dimensional reference framework based on the previously known dimensional information of the known physical structures and how those structures appear in the video feed.
  • the correlation is then used in step 108 to calibrate the video feed from video camera 22 to the known physical structures.
  • step 110 the three-dimensional location in the calibrated video feed of the number of fiducial markers 20 on ROV 14, is determined in step 110.
  • step 1 12 the position of ROV 14 within workspace 16, as well as the orientation/tilt thereof, is determined using the location of the number of fiducial markers 20 in the calibrated video feed.
  • determination of the ROV's x,y,z position and yaw is desired to adequately identify working position. These parameters can be uniquely determined with as few as 3 markers being visible simultaneously. Alternatively, if prior state information can be used to designate ROV's "front" end, only two markers visible may be adequate.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Plasma & Fusion (AREA)
  • General Engineering & Computer Science (AREA)
  • High Energy & Nuclear Physics (AREA)
  • Monitoring And Testing Of Nuclear Reactors (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Image Analysis (AREA)
  • Multimedia (AREA)
  • Electromagnetism (AREA)

Abstract

L'invention concerne un procédé (100) de localisation d'un véhicule télécommandé (14) dans un espace de travail tridimensionnel (16) qui comprend les étapes consistant à : recevoir (102) un flux vidéo de l'espace de travail à partir d'une caméra vidéo (22); traiter (104) le flux vidéo pour identifier des points de repère (30, 34) et des caractéristiques (32, 36) de structures physiques connues dans l'espace de travail ou à proximité de celui-ci ; déterminer (106) une corrélation entre les caractéristiques et les points de repère et les structures physiques connues ; étalonner (108) le flux vidéo provenant de la caméra vers les structures physiques connues à l'aide de la corrélation ; déterminer (110) l'emplacement dans le flux vidéo étalonné d'un certain nombre de marqueurs de repère (20) sur le véhicule télécommandé ; et déterminer (112) la position du véhicule télécommandé dans l'espace de travail à l'aide de l'emplacement du nombre de marqueurs de référence dans le flux vidéo étalonné.A method (100) for locating a remotely operated vehicle (14) in a three-dimensional workspace (16) includes the steps of: receiving (102) a video stream of the workspace from a video camera (22); processing (104) the video stream to identify landmarks (30, 34) and features (32, 36) of known physical structures in or near the workspace; determining (106) a correlation between features and landmarks and known physical structures; calibrating (108) the video stream from the camera to the known physical structures using the correlation; determining (110) the location in the calibrated video stream of a number of markers (20) on the remotely operated vehicle; and determining (112) the position of the remotely operated vehicle in the workspace using the location of the number of reference markers in the calibrated video stream.

Description

METHOD OF LOCATING A REMOTELY OPERATED VEHICLE
WITHIN A WORKSPACE AND REMOTE INSPECTION SYSTEM EMPLOYING SUCH METHOD
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This patent application claims the priority benefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 62/537,571 filed on July 27, 2017, the contents of which are herein incorporated by reference.
FIELD OF THE INVENTION
[0002] The present invention is directed to methods of locating a remotely operated vehicle within a workspace. The present invention is also directed to remote inspection systems which employ such methods.
BACKGROUND
[0003] Remotely operated vehicles (ROVs), are commonly employed for performing various tasks (e.g., repairs, inspections, etc.) in areas and/or environments which are not readily accessible or are inaccessible (for various reasons) by humans. An example of such use/environment is in inspecting regions within a nuclear reactor vessel which would not only be deadly to a human inspector, but would also destroy many modem electrical components instantly or in a very short period of time.
[0004] In order to carry out a safe and thorough inspection of a nuclear reactor vessel, the location and movement of an ROV used in such environment must be monitored. Presently there are no low-cost solutions that adequately allow for determining and tracking the position of an ROV in such an environment or in other inhospitable environments (e.g., outer space, deep sea).
[0005] There are many locating methods available that utilize various sensors located on the ROV, but the radiation present in nuclear reactors renders such methods inapplicable. The few methods that do not require sensors on the ROV typically require the use of lasers or other costly equipment. Accordingly, there exists room for improvement in methods of locating a remotely operated vehicle within a workspace and for remote inspection systems employing such methods.
SUMMARY OF THE INVENTION
[0006] In accordance with an embodiment of the invention, a method of locating a remotely operated vehicle within a three-dimensional workspace is provided. The method comprises: receiving a video feed of the workspace from a video camera; processing the video feed to identify landmarks and features of known physical structures in or near the workspace; determining a correlation between the features and landmarks and known physical structures; calibrating the video feed from the camera to the known physical structures using the correlation; determining the location in the calibrated video feed of a number of fiducial markers which are positioned on the remotely operated vehicle; and determining the three-dimensional position of the remotely operated vehicle within the workspace using the location of the number of fiducial markers in the calibrated video feed.
[0007] Determining a correlation between the landmarks and features and known physical structures may comprise constructing a three-dimensional reference framework based on previously known dimensional information of the known physical structures.
[0008] Determining the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle may comprise determining the location of at least three distinct fiducial markers.
[0009] Determining the location of at least three distinct fiducial markers may comprise determining the location of at least three spherical elements.
[0010] Determining the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle may comprise determining the location of at least two distinct fiducial markers.
[0011] In accordance with another embodiment of the invention, a remote inspection system is provided. The remote inspection system comprises: a remotely operated vehicle operable within a three-dimensional workspace, the remotely operated vehicle having a number of fiducial markers positioned thereon; a video camera positioned at or about a boundary of the workspace, the video camera positioned and structured to capture images of landmarks and features located in or near the workspace; an electronic storage medium having dimensional details of the landmarks and the features stored therein; and a processing device in electrical communication with both the camera and the electronic storage medium, the processing device programmed to: receive images of the workspace in a video feed from the video camera, process the video feed to identify the features and landmarks, determine a correlation between the features and landmarks and known physical structures, calibrate the video feed from the camera to the known physical structures using the correlation, determine the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle, and determine the three- dimensional position of the remotely operated vehicle within the workspace using the location of the number of fiducial markers in the calibrated video feed.
[0012] The number of fiducial markers may comprise a plurality of spherical elements. The spherical elements may be of a high contrast color.
[0013] The workspace may comprises an interior of a nuclear reactor vessel.
[0014] The video camera may be positioned at or about an upper portion of the workspace.
[0015] The electronic storage medium and the processing device may comprise elements of a computing device. The computing device may comprise a desktop or laptop computer. The computing device may comprise a handheld computing device.
[0016] These and other objects, features, and characteristics of the present invention, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] A full understanding of the invention can be gained from the following description of the preferred embodiments when read in conjunction with the accompanying drawings in which:
[0018] Figure 1 shows a schematic view of a remote inspection system in accordance with an example embodiment of the present invention;
[0019] Figure 2 shows an example screenshot in accordance with an example embodiment of the present invention of a video feed from a camera of a remote inspection system such as shown in Figure 1; and
[0020] Figure 3 shows the general steps of a method of locating a remotely operated vehicle within a workspace in accordance with an example embodiment of the present invention.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of the invention are shown. The invention may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein. Rather, these examples are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout.
[0022] As used herein, the singular form of "a", "an", and "the" include plural references unless the context clearly dictates otherwise. As used herein, the statement that two or more parts or components are "coupled" shall mean that the parts are joined or operate together either directly or indirectly, i.e., through one or more intermediate parts or components, so long as a link occurs. As used herein, "directly coupled" means that two elements are directly in contact with each other. As used herein, "fixedly coupled" or "fixed" means that two components are coupled so as to move as one while maintaining a constant orientation relative to each other.
[0023] Directional phrases used herein, such as, for example and without limitation, top, bottom, left, right, upper, lower, front, back, and derivatives thereof, relate to the orientation of the elements shown in the drawings and are not limiting upon the claims unless expressly recited therein. As used herein, the term "number" shall mean one or an integer greater than one (i.e., a plurality).
[0024] Referring to Figure 1, a schematic view of a remote inspection system 10 in accordance with an example embodiment of the present invention is shown positioned in a water-filled interior of a nuclear reactor vessel 12. Remote inspection system 10 includes a remotely operated vehicle (ROV) 14 which is operable within a three-dimensional workspace (e.g., the water-filled interior of reactor vessel 12) as shown generally at 16. As used herein, "workspace" shall be used to refer to the three-dimensional space in which an ROV may be operated. In the example embodiment shown in FIG. 1, ROV 14 is a remotely operated submersible vehicle which is tethered to a remote control system (not shown) via an umbilical 18, which includes a number of communications cables for the transmission of data/control signals to ROV 14 and data/video images from ROV 14. It is to be appreciated that other ROVs (e.g., without limitation flying drones), wired or wirelessly operable, may be employed without varying from the scope of the present invention.
[0025] ROV 14 includes a number of fiducial markers 20 positioned thereon (three of which are illustrated in the embodiment of Figure 1). The position and relative spacing (e.g., di and d2 of Figure 2) of fiducial markers 20 is predetermined such that fiducial markers 20 may be used to determine one or more of the orientation and/or angle of ROV 14. In an example embodiment of the present invention, spherical elements, approximately the size of a golf ball have been utilized. It is to be appreciated that such spherical objects remain the shape/size regardless of viewing angle (when distance is kept constant). In order to increase visibility of markers 20, such markers are preferably of a high contrast color (e.g., without limitation, fluorescent orange). It is also to be appreciated that other quantities and or shapes (e.g., without limitation, disk, partial sphere) of fiducial markers may be employed without varying from the scope of the present invention.
[0026] Continuing to refer to Figure 1, remote inspection system 10 further includes a video camera 22 in electrical communication with a processing device 24 and a suitable electronic storage medium 26 which may be integral with, or separate from, processing device 24. Video camera 22 is positioned at or about a boundary of workspace 16 and is positioned and structured to provide a video feed, via wired or wireless means, of images of workspace 16, including ROV 14 disposed therein along with landmarks (e.g., wall 30, floor 34) and features of such landmarks (e.g., vertical joints 32 of wall 30 and apertures 36 of floor 34), located in or near workspace 16. Figure 2 shows an example of an image 28 from a video feed captured by camera 22 in accordance of an example embodiment of the present invention. In the example embodiment of Figure 1, video camera 22 is positioned at or about an upper portion of workspace 16 and is structured to be able to pan and zoom upon request, however, it is to be appreciated that other cameras with other functionality, and positioning may be employed without varying from the scope of the present invention.
[0027] Processing device 24 may be any suitable programmable analog and/or digital device (including an associated memory part or portion) that can store, retrieve, execute and process data (e.g., software routines and/or information used by such routines), including, without limitation, a computer, a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a programmable system on a chip (PSOC), an application specific integrated circuit (ASIC), a microprocessor, a microcontroller, a programmable logic controller, or any other suitable processing device or apparatus.
[0028] Electronic storage medium 26 can be any one or more of a variety of types of internal and/or external storage media such as, without limitation, RAM, ROM, EPROM(s), EEPROM(s), FLASH, and the like that provide a storage register, i.e., a non-transitory machine readable medium, for data and program code storage such as in the fashion of an internal storage area of a computer, and can be volatile memory or nonvolatile memory. Electronic storage medium 26 includes dimensional details of the landmarks and the features stored therein (e.g., distance between vertical joints 32, diameter of apertures 36, spacing between apertures 36, etc.). Electronic storage medium 26 may be integral with processing unit 24. In example embodiments of the present invention, processing device 24 and electronic storage medium 26 comprise elements of a computing device such as, without limitation, a desktop or laptop computer, or a handheld computing device (e.g., without limitation a tablet computer or a smartphone).
[0029] Having thus described the basic elements of a remote inspection system 10 in accordance with an example embodiment of the present invention, a method 100 (e.g., which is carried out by processing device 24) of determining the three-dimensional position of ROV 14 within workspace 16 will now be described in conjunction with Figure 3. Method 100 begins at step 102 where a video feed of workspace 16 is received from video camera 22. Next, as shown in step 104, the video feed is processed to identify landmarks 30, 34 and features 32, 36 thereof in or near workspace 16.
[0030] After the landmarks and features in or near the workspace have been identified in step 104, dimensional data of such known physical structures which was stored in advance of such time in electronic storage medium 26 is used to determine a correlation between the landmarks and features identified in the video feed and known physical structures, such as shown in step 106. In an example embodiment of the present invention such correlation is determined by constructing a three-dimensional reference framework based on the previously known dimensional information of the known physical structures and how those structures appear in the video feed. The correlation is then used in step 108 to calibrate the video feed from video camera 22 to the known physical structures.
[0031] After workspace 16 has been recognized/generally three- dimensionally mapped in steps 102-108, the three-dimensional location in the calibrated video feed of the number of fiducial markers 20 on ROV 14, is determined in step 110. Finally, in step 1 12, the position of ROV 14 within workspace 16, as well as the orientation/tilt thereof, is determined using the location of the number of fiducial markers 20 in the calibrated video feed. Generally, determination of the ROV's x,y,z position and yaw is desired to adequately identify working position. These parameters can be uniquely determined with as few as 3 markers being visible simultaneously. Alternatively, if prior state information can be used to designate ROV's "front" end, only two markers visible may be adequate.
[0032] While specific embodiments of the invention have been described in detail, it will be appreciated by those skilled in the art that various modifications and alternatives to those details could be developed in light of the overall teachings of the disclosure. Accordingly, the particular arrangements disclosed are meant to be illustrative only and not limiting as to the scope of invention which is to be given the full breadth of the claims appended and any and all equivalents thereof. [0033] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" or "including" does not exclude the presence of elements or steps other than those listed in a claim. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. In any device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain elements are recited in mutually different dependent claims does not indicate that these elements cannot be used in combination.

Claims

CLAIMS What is claimed is:
1. A method (100) of locating a remotely operated vehicle (14) within a three- dimensional workspace (16), the method comprising:
receiving (102) a video feed of the workspace from a video camera (22); processing (104) the video feed to identify landmarks (30, 34) and features (32, 36) thereof of known physical structures in or near the workspace;
determining (106) a correlation between the landmarks features and known physical structures;
calibrating (108) the video feed from the camera to the known physical structures using the correlation;
determining (110) the location in the calibrated video feed of a number of fiducial markers (20) on the remotely operated vehicle; and
determining the three-dimensional position of the remotely operated vehicle within the workspace using the location of the number of fiducial markers in the calibrated video feed.
2. The method of claim 1, wherein determining a correlation between the features and landmarks and known physical structures comprises constructing a three- dimensional reference framework based on previously known dimensional information of the known physical structures.
3. The method of claim 1 , wherein determining the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle comprises determining the location of at least three distinct fiducial markers.
4. The method of claim 3, wherein determining the location of at least three distinct fiducial markers comprises determining the location of at least three spherical elements.
5. The method of claim 1, wherein determining the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle comprises determining the location of at least two distinct fiducial markers.
6. A remote inspection system (10) comprising:
a remotely operated vehicle (14) operable within a three-dimensional workspace, the remotely operated vehicle having a number of fiducial markers (20) positioned thereon;
a video camera (22) positioned at or about a boundary of the workspace, the video camera positioned and structured to capture images of landmarks (30, 34) and features (32, 36) located in or near the workspace;
an electronic storage medium (26) having dimensional details of the landmarks and the features stored therein; and
a processing device (24) in electrical communication with both the camera and the electronic storage medium, the processing device programmed to:
receive (102) images of the workspace in a video feed from the video camera,
process (104) the video feed to identify the features and landmarks, determine (106) a correlation between the landmarks and features and known physical structures,
calibrate (108) the video feed from the camera to the known physical structures using the correlation,
determine (110) the location in the calibrated video feed of a number of fiducial markers on the remotely operated vehicle, and
determine (112) the three-dimensional position of the remotely operated vehicle within the workspace using the location of the number of fiducial markers in the calibrated video feed.
7. The remote inspection system of claim 6, wherein the number of fiducial markers comprises a plurality of spherical elements.
8. The remote inspection system of claim 7, wherein the spherical elements are of a high contrast color.
9. The remote inspection system of claim 6, wherein the workspace comprises an interior of a nuclear reactor vessel.
10. The remote inspection system of claim 9, wherein the video camera is positioned at or about an upper portion of the workspace.
1 1. The remote inspection system of claim 6, wherein the electronic storage medium and the processing device comprise elements of a computing device.
12. The remote inspection system of claim 11, wherein the computing device comprises a desktop or laptop computer.
13. The remote inspection system of claim 11, wherein the computing device comprises a handheld computing device.
EP18837761.8A 2017-07-27 2018-07-26 PROCESS FOR LOCATING A REMOTE-CONTROLLED VEHICLE IN A WORKSPACE AND REMOTE INSPECTION SYSTEM USING SUCH A PROCESS Ceased EP3659116A4 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP24210701.9A EP4524679A1 (en) 2017-07-27 2018-07-26 Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762537571P 2017-07-27 2017-07-27
PCT/US2018/043814 WO2019023416A1 (en) 2017-07-27 2018-07-26 Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method

Related Child Applications (1)

Application Number Title Priority Date Filing Date
EP24210701.9A Division EP4524679A1 (en) 2017-07-27 2018-07-26 Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method

Publications (2)

Publication Number Publication Date
EP3659116A1 true EP3659116A1 (en) 2020-06-03
EP3659116A4 EP3659116A4 (en) 2021-04-28

Family

ID=65037759

Family Applications (2)

Application Number Title Priority Date Filing Date
EP24210701.9A Pending EP4524679A1 (en) 2017-07-27 2018-07-26 Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method
EP18837761.8A Ceased EP3659116A4 (en) 2017-07-27 2018-07-26 PROCESS FOR LOCATING A REMOTE-CONTROLLED VEHICLE IN A WORKSPACE AND REMOTE INSPECTION SYSTEM USING SUCH A PROCESS

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP24210701.9A Pending EP4524679A1 (en) 2017-07-27 2018-07-26 Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method

Country Status (6)

Country Link
US (1) US11460304B2 (en)
EP (2) EP4524679A1 (en)
JP (1) JP7346308B2 (en)
KR (1) KR102599392B1 (en)
CN (1) CN111052188B (en)
WO (1) WO2019023416A1 (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019023416A1 (en) 2017-07-27 2019-01-31 Westinghouse Electric Company Llc Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method
US11608148B2 (en) 2019-04-05 2023-03-21 Fmc Technologies, Inc. Submersible remote operated vehicle tool change control
US11361869B2 (en) * 2019-08-13 2022-06-14 Westinghouse Electric Company Llc System and method enabling signals from replacement self-powered neutron detectors to be used to generate inputs to legacy software
CN115295184B (en) * 2022-07-05 2025-07-25 苏州热工研究院有限公司 Nuclear power plant pool inspection method and system
US11807349B1 (en) * 2022-09-16 2023-11-07 Fmc Technologies, Inc. Submersible remote operated vehicle vision assistance and control

Family Cites Families (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5050112A (en) * 1989-08-08 1991-09-17 The United States Of America As Represented By The United States Department Of Energy Specimen coordinate automated measuring machine/fiducial automated measuring machine
US5603318A (en) * 1992-04-21 1997-02-18 University Of Utah Research Foundation Apparatus and method for photogrammetric surgical localization
US5550758A (en) * 1994-03-29 1996-08-27 General Electric Company Augmented reality maintenance system with flight planner
JP3427125B2 (en) * 1997-02-21 2003-07-14 仗祐 中田 Semiconductor device with optical lens function
FR2770317B1 (en) 1997-10-24 2000-12-08 Commissariat Energie Atomique METHOD FOR CALIBRATING THE ORIGINAL POSITION AND ORIENTATION OF ONE OR MORE MOBILE CAMERAS AND ITS APPLICATION TO MEASURING THE THREE-DIMENSIONAL POSITION OF FIXED OBJECTS
JP2002065641A (en) * 2000-09-04 2002-03-05 Univ Kansai Finger movement measurement system
JP3467017B2 (en) * 2000-11-30 2003-11-17 キヤノン株式会社 Position and orientation determination method and apparatus, and storage medium
DE112005000738T5 (en) * 2004-03-29 2007-04-26 Evolution Robotics, Inc., Pasadena Method and device for determining position using reflected light sources
JP2007057357A (en) * 2005-08-24 2007-03-08 Toshiba Corp In-reactor inspection and maintenance method
JP4799088B2 (en) * 2005-09-06 2011-10-19 株式会社東芝 Method and apparatus for measuring work position in remote inspection
US8279283B2 (en) * 2005-11-18 2012-10-02 Utc Fire & Security Americas Corporation, Inc. Methods and systems for operating a video surveillance system
US7674804B2 (en) 2006-06-29 2010-03-09 Hoffmann-La Roche Inc. Pyrimidine and quinazoline derivatives as SST5 modulators
JP4890294B2 (en) * 2007-02-26 2012-03-07 株式会社日立製作所 Underwater mobile device position measurement system
WO2008111692A1 (en) * 2007-03-13 2008-09-18 Research Institute Of Industrial Science & Technology Landmark for position determination of mobile robot and apparatus and method using it
JP2009143722A (en) * 2007-12-18 2009-07-02 Mitsubishi Electric Corp Person tracking device, person tracking method, and person tracking program
US9251721B2 (en) * 2010-04-09 2016-02-02 University Of Florida Research Foundation, Inc. Interactive mixed reality system and uses thereof
US8918209B2 (en) * 2010-05-20 2014-12-23 Irobot Corporation Mobile human interface robot
CH703740B1 (en) 2010-09-10 2016-12-15 Ge Hitachi Nuclear Energy Americas Llc Photogrammetric system and method for testing an inaccessible system.
US8862395B2 (en) * 2011-01-31 2014-10-14 Raytheon Company Coded marker navigation system and method
EP2699474B1 (en) * 2011-04-22 2015-11-04 Westinghouse Electric Company Llc Underwater robotic inspection system
US9942521B2 (en) 2012-01-13 2018-04-10 Honeywell International Inc. Automatic configuration of cameras in building information modeling
US9395390B2 (en) * 2012-06-19 2016-07-19 Westinghouse Electric Company Llc Eddy current inspection probe
JP2014081212A (en) * 2012-10-12 2014-05-08 Toshiba Corp Core meltdown object holding device and nuclear reactor container
US9717461B2 (en) * 2013-01-24 2017-08-01 Kineticor, Inc. Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan
WO2014144435A1 (en) * 2013-03-15 2014-09-18 Rezvan Amir H Remotely viewing and auditing cell sites using a digital data structure
HUE043828T2 (en) * 2013-03-15 2019-09-30 Univ Carnegie Mellon A supervised autonomous robotic system for complex surface inspection and processing
EP2972478B1 (en) * 2013-03-15 2020-12-16 Uatc, Llc Methods, systems, and apparatus for multi-sensory stereo vision for robotics
US9043146B2 (en) * 2013-06-19 2015-05-26 The Boeing Company Systems and methods for tracking location of movable target object
US9928594B2 (en) * 2014-07-11 2018-03-27 Agt International Gmbh Automatic spatial calibration of camera network
US9560345B2 (en) * 2014-12-19 2017-01-31 Disney Enterprises, Inc. Camera calibration
JP2016177640A (en) * 2015-03-20 2016-10-06 三菱電機株式会社 Video surveillance system
US20160324580A1 (en) * 2015-03-23 2016-11-10 Justin Esterberg Systems and methods for assisted surgical navigation
US10001544B2 (en) * 2015-05-27 2018-06-19 Samsung Electronics Co., Ltd. Method and electronic device identifying indoor location
US10694155B2 (en) * 2015-06-25 2020-06-23 Intel Corporation Personal sensory drones
US10346797B2 (en) * 2016-09-26 2019-07-09 Cybernet Systems, Inc. Path and load localization and operations supporting automated warehousing using robotic forklifts or other material handling vehicles
WO2018204726A1 (en) * 2017-05-04 2018-11-08 Carnegie Mellon University State estimation and localization for rov-based structural inspection
WO2019023416A1 (en) 2017-07-27 2019-01-31 Westinghouse Electric Company Llc Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method

Also Published As

Publication number Publication date
CN111052188B (en) 2023-11-24
WO2019023416A1 (en) 2019-01-31
JP2020528999A (en) 2020-10-01
KR102599392B1 (en) 2023-11-06
JP7346308B2 (en) 2023-09-19
KR20200024335A (en) 2020-03-06
EP3659116A4 (en) 2021-04-28
US11460304B2 (en) 2022-10-04
CN111052188A (en) 2020-04-21
EP4524679A1 (en) 2025-03-19
US20190033080A1 (en) 2019-01-31

Similar Documents

Publication Publication Date Title
US11460304B2 (en) Method of locating a remotely operated vehicle within a workspace and remote inspection system employing such method
CN113643380B (en) Mechanical arm guiding method based on monocular camera vision target positioning
KR101988083B1 (en) Systems and methods for tracking location of movable target object
CN107687855B (en) Robot positioning method and device and robot
CN112485785A (en) Target detection method, device and equipment
CN104424630A (en) Three-dimension reconstruction method and device, and mobile terminal
CN107481288A (en) The inside and outside ginseng of binocular camera determines method and apparatus
JP2020528999A5 (en) A method for locating the position of a remotely operated vehicle within a workspace, and a remote inspection system using such method.
CN109444773A (en) A kind of magnetic source detection device of connected external magnets and array of magnetic sensors
KR101438514B1 (en) Robot localization detecting system using a multi-view image and method thereof
CN103226693A (en) Identification and space positioning device and method for fished targets based on omni directional vision
CN112288821A (en) Method and device for calibrating external parameters of camera
CN107504918A (en) Radio telescope surface shape measurement method and device
US20200257183A1 (en) Information processing apparatus, information processing method, and program as well as interchangeable lens
EP3987765A1 (en) Method and system for the calibration of an object reconstruction device
JP6581280B1 (en) Monitoring device, monitoring system, monitoring method, monitoring program
CN109272551B (en) Visual positioning method based on circular mark point layout
CN102096922B (en) Object point positioning method
CN114511634B (en) Methods, devices, control equipment, and readable storage media for determining the location of the refueling port
WO2020024150A1 (en) Map processing method, apparatus, and computer readable storage medium
KR102126305B1 (en) System for measuring vehicle component parts
JP6262949B2 (en) Calibration apparatus and calibration program
Manouchehri et al. Extrinsic Calibration of Rotating 2d Laser Range Finder and Camera Using Photogrammetric Test Field and Ping Pong Balls
CN116071431A (en) Calibration method and device, storage medium and electronic equipment
Fedorov et al. Towards calibration of outdoor multi-camera visual monitoring system

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20191213

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06T0007800000

Ipc: G01C0021200000

A4 Supplementary search report drawn up and despatched

Effective date: 20210326

RIC1 Information provided on ipc code assigned before grant

Ipc: G01C 21/20 20060101AFI20210322BHEP

Ipc: G05D 1/02 20200101ALI20210322BHEP

Ipc: G05D 1/06 20060101ALI20210322BHEP

Ipc: G06T 7/70 20170101ALI20210322BHEP

Ipc: G06T 7/73 20170101ALI20210322BHEP

Ipc: G06T 7/80 20170101ALI20210322BHEP

Ipc: G21C 17/003 20060101ALI20210322BHEP

Ipc: G21C 17/013 20060101ALI20210322BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220404

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20241112