WO2017120617A1

WO2017120617A1 - System and method for single lens 3d imagers for situational awareness in autonomous platforms

Info

Publication number: WO2017120617A1
Application number: PCT/US2017/017103
Authority: WO
Inventors: David Wayne RUSSELL
Original assignee: Russell David Wayne
Priority date: 2016-01-06
Filing date: 2017-02-09
Publication date: 2017-07-13

Abstract

Autonomous vehicle platforms require situational awareness for higher level functionality. This invention utilizes 3D single lens camera technologies with high speed preprocessing modules to achieve high levels of real-time situational awareness for autonomous platforms.

Description

System and Method for Single Lens 3D Imagers for Situational Awareness in Autonomous Platforms

FIELD

[0001] This invention relates generally to 3 -dimensional (3D) imaging and more specifically to the utilization of single lens 3D imagery to enhance situational awareness in unmanned vehicle platforms.

BACKGROUND

[0001] Situational Awareness (SA) is a critical function for autonomous and semi -autonomous vehicle platforms. At a minimum SA encompasses obstacle avoidance, but higher functioning systems this could also encompass functionality such as but not limited to tracking, scene analysis, pattern recognition, target acquisition, bearing and distance measurements, landmark recognition, position tri angulation, cooperative swarms, formation flying, and threat assessment. [0002] While stereoscopic imaging can derive 3D information from two images, the images must be closely correlated to be accurate or for high speed processing.

[0003] A system is needed for high speed capture and correlation of 3 -Dimensional information from single lens imaging for autonomous vehicle situational awareness.

BRIEF SUMMARY OF THE INVENTION

[0004] For real-time assessment and awareness of the space surrounding the platform high speed processing by dedicated pre-processors or hardware accelerators may be required. Single Lens 3D (SL3) imaging provides a simple and accurate 3D imaging capability for autonomous platforms.

[0005] There are several types of 3D imaging, such as but not limited to variable focus, de-focus, synthetic aperture, and Spatial Phase Integration (SPI). Each of these approaches has benefits and drawbacks specific to the technology and to be effective the 3D imaging system should be matched to the performance of the platform and mission parameters. In some cases the 3D image requires multiple images over time, in other cases high frame rates are possible or high resolution. In some systems the output is pixel based, and in other systems the 3D data model is the native output of the imager and structured data simplifies extracting significant features relevant to situational awareness functions.

[0006] In one embodiment the SA system may be composed of some combination of one or more 3D imagers coupled with one or more Scene Analysis units. Some number of the imagers may be mounted with a Pan/Tilt/Zoom configuration while others may be fixed. For highest functionality SA systems this embodiment couples a number of SPI native 3D cameras with a Flexible Pipeline Processor (FPP) for fast reduction of data stream to useable SA information. In another embodiment some number of hardware accelerator architectures could be coupled to each of one or more 3D imagers. Each hardware accelerator may have different characteristics such as but not limited to programming, architecture, input and output requirements, and speed depending on the mission profile for that imaging system.

[0007] In another embodiment multiple frames may be processed by one or more processors to correlate the movement of one or more target objects to derive information such as but not limited to distance, bearing, size, luminosity, speed, direction, and intercept information. In other embodiments tasks such as but not limited to target recognition, tracking and following may be required of the SA system.

BRIEF DESCRIPTION OF DRAWINGS

[008] The following detailed description illustrates embodiments of the invention by way of example and not by way of limitation. The description clearly enables one skilled in the art to make and use the disclosure, describes several embodiments, adaptations, variations,

alternatives, and use of the disclosure, including what is currently believed to be the best mode of carrying out the disclosure. The disclosure is described as applied to an exemplary embodiment namely, systems and methods of utilization of single lens 3D imaging for UAV situational awareness. However, it is contemplated that this disclosure has general application to vehicle management systems in industrial, commercial, military, and residential applications.

[0009] As used herein, an element or step recited in the singular and preceded with the word "a" or "an" should be understood as not excluding plural elements or steps, unless such exclusion is explicitly recited. Furthermore, references to "one embodiment" of the present invention are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

[0010] The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0011] The novel features believed characteristic of the illustrative embodiments are set forth in the appended claims. The illustrative embodiments, however, as well as a preferred mode of use, further objectives, and features thereof will best be understood by reference to the following detailed description of illustrative embodiments of the present disclosure when read in conjunction with the accompanying drawings, wherein:

[0012] FIG. 1 shows an overall diagram of a situational awareness subsystem for autonomous platforms.

DETAILED DESCRIPTION OF THE INVENTION [0013] Referring now to the invention in more detail, in Fig. 1 there is shown an overall diagram of a situational awareness subsystem for autonomous platforms. The different illustrative embodiments recognize and take into account a number of different considerations. "A number", as used herein with reference to items, means one or more items. For example, "a number of different considerations" means one or more different considerations. "Some number", as used herein with reference to items, may mean zero or more items.

[0014] In Figure 1, one or more 3D image / model capture sensors 100 provide a stream of structured 3D data to a hardware accelerator 110. The hardware accelerator may be composed of one or more processes such a but not limited to some number of standard CPUs running software 120, some number of flexible pipeline processors 130, some number of character recognition processors 140, or some number of 3D model comparison processors 150.

[0015] Connections between the sensors and the situational analysis engines may be direct or via a crossbar routing switch 160. Other hardware could be added depending on the image sensor, makeup of the data stream, and mission requirements. For example without limitation, Flexible Pipeline Processors are very efficient for pixel based data, while character recognition might be best handled by a neural network or deep learning system, and 3D model comparison might require a totally custom processing system. One or more of the processing modules is then attached to the platform's navigation system 170 to provide it with all SA information as required.

[0016] In another embodiment bypass paths 180 are also available which route data from one of the analysis systems to another, providing the ability to merge sensor information for greater analytic capabilities. This connectivity could be part of the crossbar switch or another path as required by the implementation.

[0017] In another embodiment some number of the imager may have pan/tilt/zoom capability, and the control of these parameters is derived from either the hardware accelerator managing that imager, the overall SA system, or in some cases another subsystem might have overall control of the imaging system and allocate use of the imagers to the SA system as available.

[0018] In one embodiment situational awareness may encompass edge detection of structures within a given radius of the platform. In another embodiment it may encompass creation of a 3D model of the environment and comparing that model against a stored virtual model of the same area. Comparison of a stored data derived from a model of the environment can, for example without limitation, be used to calculate the position of the vehicle from the differences in the models in order to validate, enhance, or replace external sources of position information such as GPS, for example in GPS-denied environments.

[0019] While the foregoing written description of the invention enables one of ordinary skill to make and use what is considered presently to be the best mode thereof, those of ordinary skill will understand and appreciate the existence of variations, combinations, and equivalents of the specific embodiment, method, and examples herein. The invention should therefore not be limited by the above described embodiment, method, and examples, but by all embodiments and methods within the scope and spirit of the invention. Further, different illustrative embodiments may provide different benefits as compared to other illustrative embodiments. The embodiment or embodiments selected are chosen and described in order to best explain the principles of the embodiments, the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

[0020] The flowcharts and block diagrams described herein illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various illustrative embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function or functions. It should also be noted that, in some alternative implementations, the functions noted in a block may occur out of the order noted in the figures. For example, the functions of two blocks shown in succession may be executed substantially concurrently, or the functions of the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

Claims

The invention claimed is:

1) A system that utilizes a number of single lens 3D imaging sensors coupled with some number of situational awareness analysis engines to provide high level information to a platform navigation system.

2) The system of 1 where 3D imaging is accomplished by a number of Spatial Phase

Integration 3D imagers.

3) The system of 1 where one or more processors such as but not limited to a Flexible

Pipeline Processor, standard CPU with programming, or other hardware accelerator is used to produce high level situational awareness triggers as required by mission parameters.

4) The system of 1 where 3D imaging is accomplished by synthetic aperture, focus, or de- focus techniques coupled with situational awareness processing.

5) The system of 1 where some number of the imagers is mounted on some combination of Pan, Tilt and/or Zoom platform.

6) The system of 1 where the direction and zoom parameters of the imagers are controlled by the associated hardware accelerators or combined SA functional block.

7) The system of 1 where the direction and zoom parameters of the imagers are specified or requested by the SA functional block but another subsystem has overall control.

8) The system of 1 including use of the situational awareness system for target recognition and/or tracking.

9) The system of 1 where temporally spaced 3D models are compared for some combination of SA data such as but not limited to distance, bearing, size, luminosity, speed, direction, tracking, target bearing and intercept capabilities.

10) The system of 1 where connectivity can be routed from a given imager to one or more of the acceleration processors simultaneously.

11) The system of 1 where output from one hardware accelerator or processing node may be routed to one or more of the other nodes in addition to or instead of the platform navigation system. 12) The system of 1 where processing capability within the 3D imager can pre-process pattern recognition and analysis functions before passing structured data to the hardware accelerator or SA system.

13) The system of 1 where data derived from the 3D imager can be compared against data derived from 3D models of the Area of Operation to compute location information.

14) The system of 1 where imaging sensors may be shared by different subsystems and some number allocated to situational awareness at any given time.