EP3186606A2 - Procédé et système de mappage et de perception basés sur la polarisation - Google Patents

Procédé et système de mappage et de perception basés sur la polarisation

Info

Publication number
EP3186606A2
EP3186606A2 EP15859072.9A EP15859072A EP3186606A2 EP 3186606 A2 EP3186606 A2 EP 3186606A2 EP 15859072 A EP15859072 A EP 15859072A EP 3186606 A2 EP3186606 A2 EP 3186606A2
Authority
EP
European Patent Office
Prior art keywords
images
roadway
objects
image
polarization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP15859072.9A
Other languages
German (de)
English (en)
Other versions
EP3186606A4 (fr
Inventor
Todd M. AYCOCK
David B. Chenault
Jonathan B. HANKS
John S. Harchanko
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Polaris Sensor Technologies Inc
Original Assignee
Polaris Sensor Technologies Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/602,823 external-priority patent/US10311285B2/en
Application filed by Polaris Sensor Technologies Inc filed Critical Polaris Sensor Technologies Inc
Priority claimed from PCT/US2015/047008 external-priority patent/WO2016076936A2/fr
Publication of EP3186606A2 publication Critical patent/EP3186606A2/fr
Publication of EP3186606A4 publication Critical patent/EP3186606A4/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147—Details of sensors, e.g. sensor lenses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • LWIR Long Wave Infrared
  • thermal refers to placing objects in a scene relative to other objects or elements in the scene.
  • mapping refers to placing objects in a scene relative to other objects or elements in the scene.
  • that little rock is in the road next to that big rock just off the road.
  • roadway refers to any path along which a person, animal, or vehicle may traverse.
  • a method using Long Wave Infrared Imaging Polarimetry for improved mapping and perception of a roadway or path and for perceiving or detecting objects is disclosed herein.
  • the described method is not tied to any one specific polarimeter sensor architecture, and thus the method described pertains to all LWIR sensors capable of detecting the critical polarimetric signature.
  • the method comprises recording raw image data of an area using a polarimeter to obtain polarized images of the area.
  • the images are then corrected for non-uniformity, optical distortion, and registration in accordance with the procedure necessitated by the sensor's architecture.
  • IR and polarization data products are computed, and the resultant data products are converted to a multidimensional data set for exploitation.
  • Contrast enhancement algorithms are applied to the multidimensional imagery to form enhanced object images.
  • the enhanced object images may then be displayed to a user, and/or an annunciator may announce the presence of an object. Further, the vehicle may take evasive action based upon the presence of an object in the roadway.
  • a standard IR camera gives information about an IR signature (i.e., how bright a target looks), spatial information (i.e., where a target is in the scene), and temporal information (i.e., how the target changes in the scene from frame to frame).
  • IR signature i.e., how bright a target looks
  • spatial information i.e., where a target is in the scene
  • temporal information i.e., how the target changes in the scene from frame to frame.
  • FIG. 1 is a block diagram illustrating a system in accordance with an exemplary embodiment of the present disclosure.
  • FIG. 2 depicts an exemplary polarimeter and signal processing unit as depicted in Fig.
  • FIG. 3 is a flowchart depicting exemplary architecture and functionality of the system logic in accordance with an exemplary embodiment of the disclosure.
  • Fig. 4a depicts a visible image of a roadway at night.
  • Fig. 4b is a thermal image of the same roadway at night.
  • Fig. 4c depicts a polarization image of the roadway obtained with the system and method according to an exemplary embodiment of the present disclosure.
  • Fig. 5a is a visible image of a roadway during the daytime.
  • Fig. 5b is a thermal image of the roadway of Fig. 5a.
  • Fig. 5c is a polarization image of the roadway of Fig. 5a obtained with the system and method according to an exemplary embodiment of the present disclosure.
  • Fig. 6a is a visible image of a roadway that has a median and sidewalk, shown at night.
  • Fig. 6b is a thermal image of the roadway of Fig. 6a, also at night.
  • Fig. 6c is a polarization image of the roadway of Fig. 6a obtained with the system and method according to an exemplary embodiment of the present disclosure
  • Fig. 7a is a visible image of a dirt road depicting exemplary obstacles on the road.
  • Fig. 7b is a thermal image of the road of Fig. 7a.
  • Fig. 7c is a contrast enhanced thermal image of the road of Fig. 7a.
  • Fig. 7d is a polarization image of the road of Fig. 7a.
  • Fig. 7e is a ColorFuse image of the road of Fig. 7d.
  • Fig. 8a is a visible image of a dirt road depicting exemplary obstacles the road at night.
  • Fig. 8b is a thermal image of the road Fig. 8a.
  • Fig. 8c is a contrast enhanced thermal image of the road of Fig. 8a.
  • Fig. 8d is a polarization image of the road of Fig. 8a.
  • Fig. 8e is a ColorFuse image of the roadway of Fig. 8d.
  • Fig- 9a depicts a visible image of train tracks.
  • Fig. 9b is a thermal image of the train tracks of Fig. 9a.
  • Fig. 9c is a polarization image of the train tracks of Fig. 9c.
  • Fig. 10a is a Stokes vector image So of a maritime scene showing fishermen in a boat.
  • Fig. 10b is a ColorFuse image of the scene of Fig. 10a.
  • Fig. 10c is a Stokes vector images Si of the maritime of Fig. 10a.
  • Fig. lOd is a Stokes vector images S 2 of the maritime of Fig. 10a.
  • Fig. lOe is a DoLP image of the maritime scene of Fig. 10a.
  • FIG. 1 illustrates a system 100 in accordance with an exemplary embodiment of the present disclosure.
  • the system 100 comprises a polarimeter 101 mounted on a vehicle 103 and a signal processing unit 107, which collect and analyze images of a surface 1 1 1 for detection and annunciation of an object 102 .
  • Exemplary objects 102 shown in Fig. 1 include an obstacle 104, water or mud puddle 105, and a roadway edge 106.
  • the term "object” may refer to any object, pathway defect or area of interest, including in some embodiments humans or other animals.
  • the obstacle 104 and the puddle 105 are objects the vehicle would want to avoid.
  • the roadway edge 106 is an object that the vehicle would want to know the location of, in order to stay on a roadway.
  • the objects 102 are objects to be avoided or located in order to safely navigate the vehicle 103.
  • the objects 102 are items in need of location, for example, humans during search and rescue operations, as further discussed herein.
  • the polarimeter 101 comprises a polarizing imaging device for recording polarized images, such as a digital camera or thermal imager that collects images.
  • the vehicle 103 may be an automobile, watercraft, aircraft, or any navigable vehicle, or a human on foot.
  • the polarimeter 101 collects raw image data of the roadway environment consisting of the surface 1 1 1 (a roadway, for example), and objects 102 such as the obstacle 104, the water or mud puddle 105, and the roadway edge 106.
  • the polarimeter 101 transmits raw image data to the signal processing unit 107, which processes the data as further discussed herein. The processed data is then displayed to the operator on display 108 or detection is annunciated on an annunciator 1 10, as further discussed herein.
  • Fig. 1 shows the polarimeter 101, the signal processing unit 107, the display 109, and annunciator 1 10 as separate items, the polarimeter 101 and signal processing unit 107 are packaged into one device in certain embodiments and placed on the vehicle 103 such that the polarimeter has a view of the roadway, and with the display 109 and annunciator 110 packaged together and placed inside the vehicle.
  • the polarimeter 101 sends raw image data (not shown) to the signal processing unit 107 over a network or communication channel 108 and processed data sent to the display 109 and annunciator 1 10.
  • the signal processing unit 107 may be any suitable computer known in the art or future-developed.
  • the signal processing unit 107 receives the raw image data, filters the data, and analyzes the data as discussed further herein to provide enhanced imagery and detections and annunciations.
  • the network 108 may be of any type network or networks known in the art or future-developed, such as a simple communications cable, the internet backbone, Ethernet, Wifi, WiMax, broadband over power line, coaxial cable, and the like.
  • the network 108 may be any combination of hardware, software, or both. Further, the network 108 could be resident in a sensor (not shown) housing both the polarimeter 101 and the signal processing unit 107.
  • the vehicle 103 comprises manned or unmanned (autonomous) agricultural equipment in a farming environment and the objects 102 include obstacles along farm roads or in fields.
  • the vehicle 103 comprises manned or unmanned (autonomous) vessels that operate on waterways or oceans and the objects 102 are floating in the water.
  • the vehicle 103 comprises a person or vessel conducting search and rescue activities and objects 102 are victims of an incident involving bodies of water.
  • the vehicle 103 comprises manned or unmanned (autonomous) aircraft and objects 102 are those found in an airfield environment, including runways and the grassy areas in and around runways.
  • the vehicle 103 comprises railroad equipment and the objects 102 are those found in the environment around railroad tracks and switches.
  • Fig. 2 depicts an exemplary polarimeter 101 and signal processing unit 107 according to an embodiment of the present disclosure.
  • the polarimeter 101 comprises an objective imaging lens 1201 , a filter array 1203, and a focal plane array 1202.
  • the objective imaging lens 1201 comprises a lens pointed at the surface 1 1 1 (Fig. 1).
  • the filter array 1203 filters the images received from the objective imaging lens system 1201.
  • the focal plane array 1202 comprises an array of light sensing pixels.
  • the signal processing unit 107 comprises image processing logic 120 and system data 121.
  • image processing logic 120 and system data 121 are shown as stored in memory 1123.
  • the image processing logic 120 and system data 121 may be implemented in hardware, software, or a combination of hardware and software.
  • the signal processing unit 107 also comprises a processor 130, which comprises a digital processor or other type of circuitry configured to run the image processing logic 120 by processing the image processing logic 120, as applicable.
  • the processor 130 communicates to and drives the other elements within the signal processing unit 107 via a local interface 1 124, which can include one or more buses.
  • a local interface 1 124 can include one or more buses.
  • the image processing logic 120 and the system data 121 can be stored and transported on any computer-readable medium for use by or in connection with logic circuitry, a processor, an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.
  • a "computer-readable medium” can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
  • the computer readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.
  • the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
  • Exemplary system data 121 is depicted comprises:
  • Corrected image data (not pictured), which is the data that has been corrected for non- uniformity, optical distortion, and registration per step 1002 of the method 1000 (Fig. 3).
  • an external interface device 126 connects to and communicates with the display 109 and annunciator 1 10.
  • the external interface device 126 may also communicate with or comprise an input device, for example, a keyboard, a switch, a mouse, a touchscreen, and/or other type of interface, which can be used to input data from a user of the system 100.
  • the external interface device 126 may also or alternatively communicate with or comprise a personal digital assistant (PDA), computer tablet device, laptop, portable or non-portable computer, cellular or mobile phone, or the like.
  • PDA personal digital assistant
  • the external interface device 126 may also or alternatively communicate with or comprise a non-personal computer, e.g., a server, embedded computer, field programmable gate array (FPGA), microprocessor, or the like.
  • the external interface device 126 is shown as part of the signal processing unit 107 in the exemplary embodiment of Fig. 2. In other embodiments, the external interface device 126 may be outside of the signal processing unit 107.
  • the display device 109 may consist of a tv, led screen, monitor or any electronic device that conveys image data resulting from the method 1000 or is attached to a personal digital assistant (PDA), computer tablet device, laptop, portable or non-portable computer, cellular or mobile phone, or the like.
  • PDA personal digital assistant
  • the annunciator device 1 10 can consist of a warning buzzer, bell, flashing light, or any other auditory or visual or tactile means to warn the operator of the detection of an object or obstacle.
  • autonomous action may be taken based upon the objects 102
  • the vehicle 103 (Fig. 1) may automatically be directed to avoid objects 102.
  • the external interface device 126 may interface with the vehicle 103 such that the processor 130 may direct the vehicle to swerve around an object 102.
  • the annunciator 1 10 may not be required.
  • a Global Positioning System (“GPS”) device may interface with the external interface device 126 to provide a position of the objects 102 detected.
  • GPS Global Positioning System
  • the display 109 and annunciator 1 10 are shown as separate, but the annunciator 1 10 may be combined with the display 109, and in another embodiments, annunciation could take the form of highlighted boxes or regions or another means used to highlight the object as part of the image data display.
  • an indicator box e.g., a red box (not shown)
  • Fig. 3 is a flowchart depicting exemplary architecture and functionality of the image processing logic 120 (Fig. 2) in accordance with a method 1000.
  • the polarimeter 101 captures an image of a roadway scene from a vehicle on a roadway 1 1 1 (Fig. 1 ) and sends raw image data to the signal processing unit 107 (Fig. 1 ).
  • step 1002 the signal processing unit 107 (Fig. 1) corrects imager non-uniformity of the images received from the polarimeter 101.
  • imager non-uniformity include fixed pattern lines in the image, noisy pixels, bad pixels, bright spots, and the like. Algorithms that are known in the art may be used for correcting the imager non-uniformity.
  • step 1002 is not performed because the imager non-uniformity does not require correction.
  • the signal processing unit 107 removes image distortion from the image data.
  • image distortion is warping at the edges of the image caused by the objective imaging lens system. Algorithms that are known in the art may be used for correcting image distortion. Registration corrections may also be performed in step 1002, using methods known in the art.
  • step 1003 IR and polarization data products are computed.
  • Stokes parameters (So, Si, S 2 ) are calculated by weighted subtraction of the polarized image obtained in step 1002.
  • the LWIR imaging polarimeter measures both a radiance image and a polarization image.
  • a radiance image is a standard image whereby each pixel in the image is a measure of the radiance, typically expressed in Watts/cm2-sr, reflected or emitted from that con'esponding pixel area of the scene. Standard photographs and thermal images are radiance images, simply mappings of the radiance distribution emitted or reflected from the scene.
  • a polarization image is a mapping of the polarization state distribution across the image. The polarization state distribution is typically expressed in terms of a Stokes image.
  • S 0 represents the conventional LWIR thermal image with no polarization information.
  • Si and S 2 display orthogonal polarimetric information.
  • the Stokes vector first introduced by G.G. Stokes in 1852, is useful for describing partially polarized light and is defined as
  • / 0 is the radiance that is linearly polarized in a direction making an angle of 0 degrees with the horizontal plane
  • / 90 is radiance linearly polarized in a direction making an angle of 90 degrees with the horizontal plane.
  • 7 45 and / 135 are radiance values of linearly polarized light making an angle of 45° and 135° with respect to the horizontal plane.
  • I R and I L are radiance values for right and left circularly polarized light.
  • right and left circularly polarized light is not necessary and the imaging polarimeter does not need to measure these states of polarization. For this reason, the Stokes vectors that we consider will be limited to the first 3 elements which express linearly polarized light only,
  • the polarization state emitted or reflected from the roadway surface, surfaces to the side of the road, and objects or surfaces in the road depends on a number of factors including the angle of emission, the surface temperature of the surface, the micro-roughness of the surface (texture), the complex refractive index of the surface and the background temperature of the surrounding environment.
  • the invention here primarily makes use of the fact that the polarization state of light emitted and reflected from the surfaces and objects is a function of angle of emission and different surface texture.
  • the emissivity of an object is determined from Kirchoff s radiation law.
  • the more general equations for Kirchoff s law are given by
  • 3 ⁇ 4(0) l - r s (0) (6)
  • the p-state indicates the plane of emission for light that is linearly polarized in a plane that contains the surface normal and the line of sight to the camera. For example, if the camera is looking down at a horizontal surface, the p-state of polarization would appear vertically polarized.
  • the s-state of polarization is perpendicular to the p-state. Note that we have suppressed the temperature and wavelength dependence in equations 4-6.
  • Equation 8 can be written out more explicitly as
  • P(0) does not explicitly depend on the angle ⁇ that the plane of incidence makes with the horizontal plane.
  • the angle ⁇ is critical to determine the orientation of plane of incidence and ultimately the azimuthal angle of the surface normal.
  • the angle ⁇ can be determined from the following angle,
  • the angle 0 can be determined a number of ways.
  • a method for determining 0 and ⁇ from a normalized Stokes image (Equation 3) are known in the art.
  • a degree of linear polarization (DoLP) image is computed from the Stokes images.
  • a DoLP image is useful for providing contrast for roadway surface and objects in the road, and can be calculated as follows:
  • DoLP J( Sl /s o y + ⁇ is 2 /s o y (1 1) or D 0L
  • step 1004 may use polarization images derived from any combination of So, Si, S 2 , or S 3 and is not limited to DoLP.
  • the DoLP image is one available image used to view polarization contrast in an image.
  • Another alternative image to view polarization content is a "ColorFuse" image that is generated by mapping the radiance, DoLP, and orientation images to a color map.
  • Persons with skill in the art makes the following mapping of polarization data to a hue-saturation- value representation for color:
  • This representation enables display of all optical information (radiance and polarization) in a single image and provides a means to show both radiometric and polarization contrast enhancing understanding of the scene. In many cases where polarization contrast is strong, this representation provides scene context for the surfaces or objects that are polarized. Those experienced in the art can imagine other ways of doing this.
  • the ColorFuse is one embodiment of multidimensional representation that can be produced in step 1004. Those knowledgeable in the art can conceive similar mappings. For one example, the DoLP information may be emphasized when radiance values are low.
  • the polarization state emitted or reflected from the surface of objects or surfaces in the imaged scene depends on a number of factors including the angle of emission, the surface temperature of the surface, the micro-roughness or texture of the surface, and the complex refractive index of the surface.
  • the contrast of surfaces and objects in the scene due to polarization are dependent on the geometry and the material or surface properties of the objects in the scene. While surface temperature contributes to polarization signature contrast, temperature differences of objects in the scene are not necessary in order for there to be polarization contrast. This is important because frequently many objects in an imaged scene can be at the same or very similar temperatures and hence show little contrast.
  • step 1005 contrast enhancing algorithms that are known in the art are applied to the multidimensional image from step 1004.
  • the multi-dimensional data exploits the polarization data to significantly enhance the information content in a scene.
  • Non-restrictive examples include global mean, variance, and higher order moment analysis, Principal Component Analysis, or Linear Discriminate Analysis, computation of the statistics of the multidimensional data as a whole and then computation of local values based on a kernel convolved with the image as a whole and then normalized by global statistics of the scene.
  • step 1006 object detection algorithms that are known in the art are applied to the contrast enhanced data from step 1005.
  • object detection algorithms include setting manually or automatically a threshold value based on the image statistics, segmenting portions of the image based on the contrast enhancements, edge detection, and morphological properties.
  • detected objects may then be annunciated to the user through visual or auditory means.
  • Non-restrictive examples includes bells, buzzers or lights to draw the operator's attention to the display, or indications on the display such as distinctive colors or boxes in the region of the obstacle or surface.
  • enhanced contrast images may be displayed to the user (not shown).
  • steps 1003, 1004, 1005, and 1006 are used in combinations that omit one or more of the steps.
  • the polarization image data, or the multidimensional (Color Fuse) data may be viewed by humans for object detection, and no algorithms are applied.
  • Algorithms that exploit a combination of image features extracted from a LWIR imaging polarimeter can be used to detect potential obstacles or roadway edges. In the case of train tracks, algorithms could be used to confirm continuity of the tracks automatically. Once potential noteworthy features are detected, they can be automatically highlighted for the operator, and a warning can be given through some annunciation mechanism (buzzer or light). Algorithms could also potentially be used to exploit the orientation information to help improve understanding of the image such as segmentation or shape recognition.
  • the enhanced contrast enables the mapping of features in the imaged scene that, through operator perception or automated detection and warning, improves the safety of the operator, or in the case of autonomously operated equipment such as agricultural equipment, provides autonomous obstacle avoidance to the steering or navigation systems.
  • improved detection and recognition of obstacles will allow the operator to maneuver the vehicle (or vessel) to avoid obstacles.
  • Improved detection and perception of roadway edges will reduce chances of inadvertently leaving the roadway. This is especially true at night when the operator's vision is limited by darkness.
  • the system and method of the present disclosure adds a polarimetric signature to the information that was previously attainable by an IR camera, i.e., temporal, special and IR signature.
  • a polarimetric signature can be used simultaneously to classify/categorize objects detected.
  • the classification/categorization of the detected objects can influence evasive action to be taken by a vehicle. For example, a detected object in a roadway may be classified as an obstacle that needs to be avoided, rather than a pothole the vehicle is capable of driving over.
  • multiple objects may be independently and simultaneously classified into separate groups or sub-groups based on their temporal, spatial, IR, and/or polarimetric signatures in accordance with given criteria.
  • Fig. 4a depicts a visible image of a roadway 400 at night.
  • Fig. 4b is a thermal image of the same roadway 400 at night. Note that in Fig. 4b, the roadway 400 and surrounding terrain have nearly the same temperature and hence there is little contrast between the roadway 400 and a shoulder 401 of the road in the thermal image.
  • Fig. 4c depicts a polarization image of the roadway 400 obtained with the system and method according to an exemplary embodiment of the present disclosure. The polarization image in 4c shows strong contrast of the road 400 and the shoulder 401 is easily discernable. A white stripe 402 that parallels the roadway 400 on the left hand side is a sidewalk. The polarization image in 4c was obtained with no external light source.
  • Fig. 5a is a visible image of a roadway 500 during the daytime.
  • Fig. 5b is a thermal image of the roadway 500 of Fig. 5a.
  • the roadway 500 and other scene elements show confusing contrast in the thermal image of Fig 5b.
  • Fig. 5c is a polarization image of the roadway 500 of Fig. 5 a obtained with the system and method according to an exemplary embodiment of the present disclosure.
  • the polarization image of Fig. 5c shows strong contrast of only the roadway 500.
  • a sidewalk 501 that parallels the road on the left hand side and a driveway 502 on the right are easily discernable in the polarization image of Fig. 5c.
  • the sidewalk 501 and the driveway 502 are not easily perceptible in the thermal image of Fig. 5b.
  • Fig. 6a is visible image of a roadway 600 that has a median 601 and sidewalk 602, shown at night.
  • Fig. 6b is a thermal image of the roadway 600 of Fig. 6a, also at night.
  • the roadway and surrounding terrain have similar temperatures and hence there is weak contrast between the roadway and the media of the road in the thermal image.
  • Fig. 6c is a polarization image of the roadway 600 of Fig. 6a obtained with the system and method according to an exemplary embodiment of the present disclosure.
  • the polarization image of Fig. 6c shows strong contrast of the roadway 600.
  • the sidewalk 602 that parallels the road on the right hand side and the median 601 are easily discernable in the polarization image of Fig. 6c.
  • the sidewalk 602 and the median 601 are not easily perceptible in the thermal image of Fig. 6b.
  • Fig. 7a is a visible image of a dirt road 700 depicting exemplary obstacles 701 on a road 700.
  • the obstacles 701 comprise wood planks in the image.
  • Fig. 7b is a thermal image of the road 700 of Fig. 7a. In the image of Fig. 7b, the obstacles 701 are easier to discern than in the visible image of Fig. 7a.
  • Fig. 7c is a contrast enhanced thermal image of the road 700 of Fig. 7a.
  • Fig. 7d is a polarization image of the road 700 of Fig. 7a.
  • the obstacle 701 in this image is easily discerned, though the polarization image does not provide much context to the obstacle in relation to the road 700.
  • Fig. 7e is a ColorFuse image of the roadway of Fig. 7d.
  • the ColorFuse image shows both thermal and polarimetric data in a single image, and provides the greatest contrast.
  • Fig. 8a is a visible image of a dirt road 800 depicting exemplary obstacles 801 on a road 800 at night.
  • the obstacles 801 comprise wet dirt and mud in the image. These are potential hazards which might immobilize some ground vehicles.
  • Fig. 8b is a thermal image of the road 800 of Fig. 8a.
  • Fig. 8c is a contrast enhanced thermal image of the road 800 of Fig. 8a.
  • Fig. 8d is a polarization image of the road 800 of Fig. 8a.
  • the obstacles 801 in this image are easily discerned, though the polarization image does not provide much context to the obstacles in relation to the road 800.
  • Fig. 8e is a ColorFuse image of the roadway of Fig. 8d.
  • the ColorFuse image shows both thermal and polarimetric data in a single image, and provides the greatest contrast.
  • the ColorFuse image of Fig. 8e shows how the combination of thermal and polarization data products can be used to provide a good representation of road surfaces.
  • Fig. 9a is a visible image of train tracks 900 depicting exemplary segmentation of the rails in a railroad environment.
  • Fig. 9b is a thermal image, in which identification of the tracks is difficult due to different temperatures of the objects in areas adjacent to the tracks.
  • Fig. 9c is a polarimetric image of the train tracks 900 of Fig. 9a, and show good delineation of the tracks.
  • Fig. 10a, 10c, and l Od are Stokes vector images So, Si and S 2 , respectively, of a maritime scene showing fishermen in a boat.
  • Fig. lOe is a DoLP image of the same scene.
  • Fig. 10b is a ColorFuse image of the scene. The ColorFuse image shows improvement in contrast for obstacle avoidance for vessels or advantage for search and rescue.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Multimedia (AREA)
  • Automation & Control Theory (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Vascular Medicine (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Image Processing (AREA)
  • Traffic Control Systems (AREA)

Abstract

L'invention concerne un procédé utilisant une polarimétrie d'imagerie infrarouge à onde longue pour améliorer le mappage et la perception d'une route et d'un chemin et pour percevoir ou détecter des obstacles, ledit procédé consistant à enregistrer des données d'images brutes au moyen d'un polarimètre pour obtenir des images polarisées de la route ou de la zone. La non-uniformité, la distorsion optique et l'enregistrement des images sont ensuite corrigées. Le produits de données IR et de polarisation sont calculés, puis les produits de données obtenus sont convertis en un ensemble de données multidimensionnelles pour l'exploitation. Des algorithmes d'amélioration de contraste sont appliqués à l'imagerie multi-dimensionnelle pour former des images d'objets améliorées. Les images d'objets améliorées peuvent ensuite être affichées à l'attention d'un utilisateur et/ou un annonciateur peut annoncer la présence d'un objet. De plus, le véhicule peut prendre des mesures d'évitement en fonction de la présence d'un objet sur la route.
EP15859072.9A 2014-08-26 2015-08-26 Procédé et système de mappage et de perception basés sur la polarisation Withdrawn EP3186606A4 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201462041778P 2014-08-26 2014-08-26
US14/602,823 US10311285B2 (en) 2014-01-22 2015-01-22 Polarization imaging for facial recognition enhancement system and method
PCT/US2015/047008 WO2016076936A2 (fr) 2014-08-26 2015-08-26 Procédé et système de mappage et de perception basés sur la polarisation

Publications (2)

Publication Number Publication Date
EP3186606A2 true EP3186606A2 (fr) 2017-07-05
EP3186606A4 EP3186606A4 (fr) 2018-05-09

Family

ID=58397916

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15859072.9A Withdrawn EP3186606A4 (fr) 2014-08-26 2015-08-26 Procédé et système de mappage et de perception basés sur la polarisation

Country Status (3)

Country Link
EP (1) EP3186606A4 (fr)
CN (1) CN107076614A (fr)
AU (1) AU2015347259A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310607A (zh) * 2020-01-22 2020-06-19 交通运输部公路科学研究所 基于计算机视觉和人工智能的公路安全风险识别方法及系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102018125736B4 (de) * 2018-10-17 2021-02-18 Sick Ag Verfahren zum Schutz von Menschen in einer Umgebung einer beweglichen Maschine
CN109490867B (zh) * 2018-12-25 2020-05-22 北京理工大学 水面目标偏振遥感探测能力评价方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7426437B2 (en) * 1997-10-22 2008-09-16 Intelligent Technologies International, Inc. Accident avoidance systems and methods
WO2001081949A2 (fr) * 2000-04-26 2001-11-01 Arete Associates Imagerie a resolution temporelle tres rapide dans un espace de mesure multiparametrique
US6674532B2 (en) * 2001-11-02 2004-01-06 Vandelden Jay S. Interferometric polarization interrogating filter assembly and method
US20080059007A1 (en) * 2006-06-09 2008-03-06 Whittaker William L System and method for autonomously convoying vehicles
US20080129541A1 (en) * 2006-12-01 2008-06-05 Magna Electronics Black ice detection and warning system
US8462323B2 (en) * 2007-03-27 2013-06-11 Metrolaser, Inc. Integrated multi-sensor surveilance and tracking system
WO2008128024A1 (fr) * 2007-04-12 2008-10-23 The Arizona Board Of Regents On Behalf Of The University Of Arizona Appareil photo de polarimétrie d'instantané compact
JP4645714B2 (ja) * 2008-09-19 2011-03-09 株式会社デンソー 画像処理装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310607A (zh) * 2020-01-22 2020-06-19 交通运输部公路科学研究所 基于计算机视觉和人工智能的公路安全风险识别方法及系统

Also Published As

Publication number Publication date
EP3186606A4 (fr) 2018-05-09
CN107076614A (zh) 2017-08-18
AU2015347259A1 (en) 2017-03-30

Similar Documents

Publication Publication Date Title
US11386648B2 (en) Polarization-based mapping and perception method and system
US9589195B2 (en) Polarization-based mapping and perception method and system
US11022541B2 (en) Polarimetric detection of foreign fluids on surfaces
KR101610432B1 (ko) 적외선 카메라 시스템 및 듀얼 센서 응용 방법
KR20210036886A (ko) 항만 모니터링 장치 및 항만 모니터링 방법
US11900668B2 (en) System and method for identifying an object in water
WO2019067193A1 (fr) Détection nocturne
US20200404224A1 (en) Multispectrum, multi-polarization (msmp) filtering for improved perception of difficult to perceive colors
US10803590B2 (en) Multiple-wavelength images analysis electro optical system for detection of accident ship and submerged person and analysis method thereof
EP3631391A1 (fr) Procédé et système de détection et de cartographie basés sur une polarisation
US20220214222A1 (en) Radiometric thermal imaging improvements for navigation systems and methods
Kim et al. Artificial intelligence vision-based monitoring system for ship berthing
US20220291701A1 (en) Thermal imaging for navigation systems and methods
EP3186606A2 (fr) Procédé et système de mappage et de perception basés sur la polarisation
WO2016076936A2 (fr) Procédé et système de mappage et de perception basés sur la polarisation
US20220301303A1 (en) Multispectral imaging for navigation systems and methods
Rodríguez-Jiménez et al. A-contrario detection of aerial target using a time-of-flight camera
EP4067814A1 (fr) Améliorations d'imagerie thermique radiométrique pour systèmes et procédés de navigation
Low A wireless webcam-based robotic vision system
Rodri et al. A-contrario detection of aerial target using a time-of-flight camera
Shi Obstacle detection using thermal imaging sensors for large passenger airplane
Teoh Near-range water body detection and obstacle detection in rainforest terrain/tropical terrain
Sandoval et al. Real-time mapping and navigation by fusion of multiple electro-optic sensors

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170306

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20180409

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 5/00 20060101ALI20180403BHEP

Ipc: G01J 4/00 20060101AFI20180403BHEP

Ipc: G06K 9/00 20060101ALI20180403BHEP

Ipc: G06K 9/20 20060101ALI20180403BHEP

Ipc: G01J 4/04 20060101ALI20180403BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20210323

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20230301