US20190049734A1 - System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display - Google Patents
System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display Download PDFInfo
- Publication number
- US20190049734A1 US20190049734A1 US16/165,210 US201816165210A US2019049734A1 US 20190049734 A1 US20190049734 A1 US 20190049734A1 US 201816165210 A US201816165210 A US 201816165210A US 2019049734 A1 US2019049734 A1 US 2019049734A1
- Authority
- US
- United States
- Prior art keywords
- image
- goggles
- scope
- block
- computer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims description 18
- 230000000153 supplemental effect Effects 0.000 title description 4
- 238000003672 processing method Methods 0.000 claims 1
- 238000004364 calculation method Methods 0.000 description 27
- 239000011295 pitch Substances 0.000 description 19
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 10
- 239000011159 matrix material Substances 0.000 description 10
- 238000012937 correction Methods 0.000 description 8
- 230000010363 phase shift Effects 0.000 description 7
- 230000004297 night vision Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 239000010432 diamond Substances 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000002349 favourable effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000008685 targeting Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 244000062327 Eupatorium odoratum Species 0.000 description 1
- 235000009199 Lamium album Nutrition 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 229910003460 diamond Inorganic materials 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000005670 electromagnetic radiation Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000001931 thermography Methods 0.000 description 1
Images
Classifications
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F41—WEAPONS
- F41G—WEAPON SIGHTS; AIMING
- F41G3/00—Aiming or laying means
- F41G3/14—Indirect aiming means
- F41G3/16—Sighting devices adapted for indirect laying of fire
- F41G3/165—Sighting devices adapted for indirect laying of fire using a TV-monitor
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0176—Head mounted characterised by mechanical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0189—Sight systems
-
- G06K9/6202—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H04N5/247—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
- G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0149—Head-up displays characterised by mechanical features
- G02B2027/0154—Head-up displays characterised by mechanical features with movable elements
- G02B2027/0159—Head-up displays characterised by mechanical features with movable elements with mechanical means other than scaning means for positioning the whole image
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Definitions
- Augmented reality systems offer a mechanism for providing a user with additional data about his or her environment.
- An example of such a system is a heads-up display (HUD) found in aircraft and automobiles.
- HUD heads-up display
- a HUD projects various navigational and/or operational data on a windshield or other transparent surface in a user's field of view. This allows a user to monitor various information without having to divert his or her gaze from the external environment.
- Augmented reality systems have also been developed for use in a combat military environment.
- commonly-owned U.S. patent application Ser. No. 11/000,934 (filed Dec. 2, 2004, titled “System and Method for Video Image Registration in a Heads Up Display,” published as Pub. No. US20060121993, and incorporated by reference herein) describes a system in which an image from a rifle-mounted video source is superimposed on an image seen through a pair of goggles.
- Sensors coupled to the rifle and to the goggles provide data indicating movement of the goggles and rifle.
- An image from the rifle-mounted source shows an external region within the source's field of view (FOV).
- FOV field of view
- the goggles have a wider FOV and provide an image that includes a portion showing the same region as is shown in the image from the rifle-mounted video source.
- the sensor data is then used to determine the relative orientation of the two sources and calculate a location for the rifle image within the image seen through the goggles.
- IMU inertial measurement unit
- Additional advantages could also be obtained by increasing the types of information provided by a HUD within goggles worn by a user.
- a computer receives images from two video sources. Each of those two video sources is movable independent of the other and generates images that represent a portion of an external environment within its field of view.
- One of the video sources may be contained within a pair of goggles worn by a user, and the other source may be mounted to a rifle carried by the user.
- Sensors coupled to the two video sources provide data to the computer that indicates the spatial orientations of those sources.
- the computer determines a location for placing a video image (or a portion thereof) from a second of the sources (e.g., a rifle-mounted source) in the video image from a first of the sources (e.g., a goggles-mounted source).
- the second source video image and a corresponding portion of the first source video image represent the same part of the external environment.
- Data from the two images are then compared in order to evaluate the location determined from the sensor data.
- the sensor-based location is either confirmed, or a new location is found based on additional image comparisons.
- Once a location is selected (either a confirmed sensor-based location or a location found using image comparison), the two images are displayed such that the second source image (or a portion of that image) overlays a corresponding portion of the first source image. Locations obtained using image comparisons are used to calibrate (adjust) the manner in which subsequent sensor-based locations are determined.
- a database contains information about external objects in the environment in which a system employing the two video sources is being used.
- the computer receives system location information and orientation information for one of the video sources (e.g., the source contained within a user's goggles).
- the computer uses the location and orientation information to calculate a spatial volume within the field of view of that video source.
- the computer parses the database for information about objects that may be located within the calculated spatial volume.
- icons or other graphical indicia are displayed for objects identified as being within the calculated spatial volume, the positions of the icons being imposed upon the locations of the objects.
- FIG. 1 illustrates a system, according to at least some embodiments, for providing an information-enhanced heads up display (HUD).
- HUD heads up display
- FIG. 2 is a block diagram of the control unit for the system of FIG. 1 .
- FIG. 3 is a block diagram showing data flows in the system of FIG. 1 .
- FIG. 4 is an example of a user display in the system of FIG. 1 .
- FIGS. 5A and 5B are a flow chart explaining operation of the system of FIG. 1 .
- FIGS. 6A through 6F illustrate positioning of one video image within another based on data from inertial measurement units.
- FIGS. 6G and 6H illustrate rotation of one video image within another based on data from inertial measurement units.
- FIG. 7 is a flow chart providing additional details for a block in the flow chart of FIGS. 5A and 5B .
- FIGS. 8A through 8K illustrate checking and/or correcting an IMU-based position for one video image within another video image.
- FIGS. 9A through 9C illustrate correction of an IMU-based position calculation based on image comparison results.
- FIG. 10 is an example of a tactical database.
- FIG. 11 is a flow chart providing additional details for another block in the flow chart of FIGS. 5A and 5B .
- FIGS. 12A through 12D show identification of real-world objects based on a spatial volume calculated from a position and orientation.
- FIGS. 13A and 13B show a goggles images and scope image, respectively, used to describe an alternate image comparison algorithm.
- FIG. 14 is a flow chart for an alternate image comparison algorithm.
- FIG. 15 is a partially schematic diagram corresponding to the scope image of FIG. 13B .
- FIG. 16 is a partially schematic diagram corresponding to the goggles image of FIG. 13A .
- FIGS. 17A through 17C show collection of gray-scale values from scope and goggles images.
- FIG. 18 is a plot showing an example of gray-scale values.
- FIG. 19 is a plot of the results of calculations in which image region center points coincide.
- FIG. 20 is a plot of the results of calculations similar to those used to generate the plot of FIG. 19 , but using random data.
- FIGS. 21 and 22 show a pattern by which additional points are selected.
- FIG. 1 illustrates a system 10 , according to at least some embodiments, that provides an information-enhanced heads-up display (HUD) for an infantryman or other armed tactical operator.
- HUD head-up display
- System 10 includes a set of goggles 11 that are configured for wear by the user. Goggles 11 may be worn directly, may be attached to a helmet or other headgear (not shown), or worn in some other manner. The front side of goggles 11 faces outward (i.e., away from the user) when worn, and includes eyepieces 12 and other apertures (not shown) for receiving light or other (e.g., IR) input from the user's field of view.
- eyepieces 12 and other apertures not shown
- goggles 11 faces the user when worn such that the user looks outward through eyepieces 12 and sees external objects within the goggles field of view.
- goggles 11 includes an image generator and a projector and creates a user display that combines video images and graphical indicia with what a user sees as a result of light passing from the external environment through eyepieces 12 .
- a sensor 13 is attached to goggles 11 or otherwise physically coupled so as to move as a single unit with goggles 11 .
- Sensor 13 includes an inertial measurement unit (IMU) and magnetometer.
- IMU inertial measurement unit
- sensor 13 is an IM 3 TM sensor (available from Archangel Systems, Inc.
- Control unit 16 includes a computer, radio receiver and other elements, as is also described below.
- System 10 further includes a video source (or “scope”) 17 and a sensor 18 configured to move as a single unit with scope 17 .
- scope 17 is affixed to a rifle 19 and is used to target rifle 19 .
- scope 17 is aligned with the barrel of rifle 19 such that a reticle corresponding to the optical centerline of scope 17 generally follows the path of a bullet fired from rifle 19 .
- Scope 17 may include a visible-light video camera, a thermal imaging (i.e., IR sensitive) video camera, a night-vision video camera, or some combination thereof.
- scope 17 is a Light Weapon Thermal Sight (available from DRS Optronics of Melbourne, Fla.).
- Sensor 18 is also an IMU (such as the aforementioned IM 3 TM sensor) in at least some embodiments.
- Sensor 18 and scope 17 communicate with control unit 16 via cables 20 and 21 .
- control unit 16 receives video feeds from scope 17 and goggles 11 .
- the video feeds may be delivered using any standard video format, for example analog formats like NTSC or PAL, digital formats like MPEG, or a non-standard format.
- Sensors 13 and 18 communicate angular pitch, yaw, and roll information to control unit 16 .
- a computer within control unit 16 determines orientation of the line of sight for goggles 11 relative to the line of sight for scope 17 . As described in more detail below, this permits superimposition of an image from scope 17 onto a display corresponding to a field of view of goggles 11 .
- sensors 13 and 18 communicate via Universal Asynchronous Receiver/Transmitter (UART) protocol on cables 14 and 21 . These cables, along with video cables 15 and 20 , may be sewn into a soldier's clothing, rifle sling, equipment harness, etc. to prevent entanglement.
- control unit 16 communicates wirelessly with sensors 13 and 18 , goggles 11 and/or scope 17 in other embodiments.
- ultra-wideband (UWB) transceivers may transmit video and sensor data from rifle 19 , as well as video and sensor data from goggles 11 .
- UWB may be used to transmit video from the control unit to goggles 11 .
- UWB radios such as Time Domain's PulsON® radio, are particularly desirable for their high bandwidth, low power consumption and for being virtually undetectable, any wireless standard may be used, including both Bluetooth and IEEE 802.11.
- UWB radios may be used for more than transmission of video and sensor data. Multiple radios may be placed on rifle 19 and on goggles 11 (or on a helmet, to which the goggles may be affixed), each of which can relay their precise position. In this fashion, control unit 16 is able to calculate the alignment of the rifle and goggles based on the relative location of radios rather than using separate orientation sensors.
- FIG. 2 is a block diagram of control unit 16 .
- Control unit 16 includes a computer 30 , global positioning system (GPS) chipset 31 and data communication chip set 32 .
- GPS chip set 31 receives signals from GPS satellites via antenna 33 and outputs GPS data to computer 30 . Based on that GPS data, computer 30 determines the location of system 10 (and thus of the user) to within the accuracy limits of the particular GPS chip set. For differential GPS systems intended for military use, the horizontal position information is typically accurate to less than three meters.
- RF data communication chip set 32 provides two-way data communication (using antenna 34 ) between system 10 and other elements in a tactical network. Those other elements may include additional systems 10 (e.g., worn by other soldiers in a squad), a remotely located commander, etc.
- the communication link(s) serviced by chip set 32 may include a Land Warrior radio local net, single channel ground and airborne radio system (SINCGARS), Force XXI Battle Command Brigade-and-Below (FBCB2) tracking system, etc
- Computer 30 includes a bus 35 connected to GPS chip set 31 and local communication chip set 32 via GPS interface 36 and communication interface 37 , respectively.
- a processor 38 communicates with interfaces 36 and 37 and with memory 40 via bus 35 .
- Memory 40 includes a combination of volatile (e.g., random access memory, or RAM) and non-volatile memory (e.g., removable and/or non-removable FLASH, EEPROM, hard drive, etc.). Stored in memory 40 are instructions for performing operations of system 10 as described herein and a tactical database (described below). Additional components communicating with processor 38 via bus 35 include a video input interface 41 , video output interface 42 and sensor interfaces 43 - 45 . Video input interface 41 receives video input from scope 17 and goggles 11 and makes that input available to processor 38 .
- sensor interfaces 43 and 44 receive input from sensors 13 and 18 and make sensor data available to processor 38 .
- Other sensors e.g., barometric sensors
- Processor 38 provides video output, via interface 42 , to goggles 11 for presentation in a display generated within goggles 11 .
- FIG. 2 is merely one example of a control unit.
- the components and/or functions shown may be combined and/or divided in various ways.
- a single input interface could receive all sensor and video feed data.
- some or all of the operations performed by processor 38 and other elements of control unit 16 could be performed by one or more application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.
- ASICs application specific integrated circuits
- FPGAs field-programmable gate arrays
- some or all of the operations described herein as being performed by processor 38 are hardwired into one or more ASICs as gates and other logic dedicated to the calculations and other operations described herein.
- FIG. 3 is a block diagram showing data flows in system 10 .
- Scope 17 receives electromagnetic input (e.g., visible light, infrared) from within its field of view.
- the field of view for scope 17 is shown schematically in FIG. 3 with dotted lines and labeled “scope FOV.”
- the scope FOV is 18°.
- an optical centerline (or “line of sight”) for scope 17 also referred to herein as the “scope FOV centerline”.
- the scope FOV is a cone symmetrically centered on the scope optical centerline.
- Scope 17 outputs a video feed corresponding to an image of external objects within the scope FOV.
- a video image output from scope 17 will be referred to as a “scope image.”
- Sensor 18 is coupled to scope 17 (because of both components being attached to rifle 19 ) and outputs data (P S , R S , Y S ) reflecting the rotational orientation of scope 17 about pitch (P), roll (R) and yaw (Y) axes (as shown in FIG.
- sensor 13 coupled to goggles 11 outputs data (P G , R G , Y G ) reflecting the rotational orientation of goggles 11 about pitch (P), roll (R) and yaw (Y) axes.
- P pitch
- R roll
- Y yaw
- sensors 13 and 18 output data at predetermined sampling intervals (e.g. 100 Hz).
- sensor 13 also includes a magnetometer. The magnetometer senses the earth's magnetic field and provides a signal (MagNorth) indicative of magnetic north.
- control unit 16 In addition to sensor data and a scope 17 video signal, control unit 16 also receives periodic position data updates via antenna 33 and tactical data via antenna 34 .
- position data via antenna 33 includes data from GPS satellites.
- Tactical data includes data providing positions and descriptions of various elements (e.g., friendly or hostile forces) in the environment in which system 10 is operating.
- Other types of data may be received from one or more additional sensors 55 .
- sensor 55 is a barometer providing data indicating changes in the elevation of system 10 , which data can be used to confirm and/or adjust GPS-based elevation data.
- Goggles 11 includes an array 56 of electromagnetic receptors (e.g., charge-coupled devices, photodiodes or other photo-sensitive elements, microbolometers) that receive electromagnetic radiation (e.g., visible light, IR) from within the receptor array field of view (“goggles FOV”).
- electromagnetic receptors e.g., charge-coupled devices, photodiodes or other photo-sensitive elements, microbolometers
- the goggles FOV is a cone symmetrically centered about an optical centerline/line of sight (also referred to herein as the “goggles FOV centerline”) extending from goggles 11 . This configuration is not required, however.
- An example of an image from a goggles array with a non-conical FOV is described below in connection with FIG. 13A .
- Visible light from the goggles FOV also passes through eyepiece 12 and can be seen by the user.
- eyepiece 12 and array 56 are offset slightly from one another, the eyepiece FOV (not indicated in FIG. 3 ) is the same or larger than the array field of view (i.e., the goggles FOV in FIG. 3 ) and generally has the same center line.
- Array 56 generates image data (e.g., a night vision view of the environment within the goggles FOV). This image data is then provided to an image generator 57 that processes the image data into an image of objects in the goggles FOV.
- an image corresponding to the data from array 56 will be referred to as a “goggles image.”
- the goggles image may be modified by computer 30 prior to being displayed to a user. Accordingly, the goggles image is also output via line 58 to computer 30 .
- a video feed from computer 30 (which may contain the goggles image modified to include the scope image and other graphics) is received by projector 59 via line 61 .
- Projector 59 through beam splitter 60 , combines the goggles image with the visible light passing directly through eyepiece 12 .
- the goggles image can also be provided directly to projector 59 from image generator 57 (e.g., if computer 30 or scope 17 becomes inoperative but night vision capability is still desired).
- a heads-up display (HUD) field of view (HUD FOV) is also shown in FIG. 3 , and is further discussed below.
- FIG. 3 only shows one array, one eyepiece, one projector and one beam splitter. In at least some embodiments, there is a separate arrangement of array, eyepiece, projector and beam splitter for each eye.
- Goggles 11 in some embodiments is a modified version of the AN/PVS-21 low-profile night vision goggles with M-2755 Enhanced Heads Up Display available from Sensor Technology Systems, Inc. of Beavercreek, Ohio. The modifications to a standard AN/PVS-21 (or other night vision goggles or enhanced vision system) required to implement various embodiments of the invention will be apparent to persons of skill in the art once such persons are provided with the information included herein.
- FIG. 4 shows an example of a user display 70 provided by goggles 11 .
- “user display” is used to refer to the image projected by projector 59 ( FIG. 3 ) through beam splitter 60 into eyepiece 12 , and which is visible by a wearer of goggles 11 .
- a user display may include the goggles image (or a portion thereof), the scope image (or a portion thereof) and other graphical objects.
- FIG. 4 assumes system 10 is being used in a forest setting, though such a setting is merely for purposes of illustration.
- goggles FOV Located within the goggles FOV (and thus in goggles image 82 ) are numerous trees and bushes such as might be present in a forest environment, as well as soldiers 71 (partially behind a tree in the lower left) and 72 (partially covered by foliage in the upper right).
- User display 70 is also slightly shaded so as to indicate that the goggles image is a night vision display.
- the outer circle of FIG. 4 generally corresponds to the limit of the goggles FOV.
- the HUD portion of user display 70 is shown as a rectangular region 73 in the center portion of the goggles FOV. As explained in further detail below, various graphical indicia are overlaid within HUD 73 .
- HUD 73 Also overlaid on HUD 73 is a weapon view 74 corresponding to (and generated from) the scope image.
- the weapon view is the same size as the scope image. This is not a requirement, however.
- the scope image may be cropped to provide a weapon view that is actually smaller than the scope image (or that corresponds to reduced scope FOV).
- HUD 73 is shown as a rectangle and weapon view 74 is shown as a circle in FIG. 4 , other shapes could be used.
- weapon view 74 is rectangular. Portions of the scope image at the outer edges of the scope FOV may similarly be cropped so as to give a weapon view a desired shape.
- the location and rotation of weapon view 74 within user display 70 is determined by computer 30 based on output from sensors 13 and 18 and based on comparison of the scope image with the goggles image.
- scope images or portions thereof are dynamically positioned within user display 70 so as to indicate where scope 17 (and thus rifle 19 ) is pointing.
- a user of system 10 is able to simultaneously survey a setting, acquire a target and point a weapon at the target without having to remove his or her goggles and adjust to a weapon sight.
- An aiming reticle 76 is located at the center of weapon view 74 , and corresponds to the rifle aim point (as zeroed to the rifle at a specified range) and lies near the scope FOV centerline.
- Various other graphical objects are also included in the HUD.
- “friendly” icons 77 and 78 are positioned upon soldiers 71 and 72 . In this manner, the user of system 10 is aware that soldiers 71 and 72 should not be fired upon.
- an icon 79 indicating the location of (or direction to) a rally point (i.e., a position determined in advance of a tactical operation and at which persons involved in the operation may regroup).
- Icon 80 shows a reported location of an enemy machine gun position.
- Colored path 81 indicates a preplanned route. Additional details of generating the graphical objects in display 70 , as well as examples of other graphical objects that could be included in display 70 , are provided below.
- FIGS. 5A and 5B are a flow chart explaining the operation of system 10 .
- the steps shown in FIGS. 5A and 5B can be reordered, combined, split, replaced, etc.
- FIGS. 5A and 5B are not intended to indicate a requirement that all steps be performed as part of a single algorithm.
- Many aspects of system 10 operation could be accommodated on multiple independently-running parallel program threads or as processes running in parallel on an ASIC, an FPGA, multiple processors, etc.
- all of the operations shown in the flow charts of FIGS. 5A through 5B would not necessarily be performed the same number of times as is suggested by the flow chart.
- refreshing and/or updating of icons might occur at a different rate than updating and/or repositioning of weapon view 74 .
- a user activates system 10 , and goggles 11 , control unit 16 and scope 17 are energized. Sensors 13 and 18 are also energized. The system is then calibrated (block 103 ).
- a user of system 10 remains motionless for a period of time so as to allow rate sensors and accelerometers in sensors 13 and 18 to stabilize, determine a down vector, etc.
- IMU sensors 13 and 18 can both determine a common down vector based on the earth's gravitational field. The sensors thus require no further initial calibration as to the pitch and roll axes. However, each sensor randomly chooses a zero-yaw orientation.
- sensors 13 and 18 must be further calibrated so that a baseline difference between the zero-yaw orientation can be determined.
- sensor 18 on rifle 19 might arbitrarily choose a zero-yaw position that corresponds to rifle 19 pointing along a 37 degree compass heading
- sensor 13 on goggles 11 might arbitrarily choose a zero-yaw position that corresponds to the optical centerline of the goggles FOV pointing along a 246 degree compass heading.
- this difference in zero-yaw orientations is determined by the user looking at a distant (e.g., over 300 meters away) object through goggles. A calibration reticle corresponding to the goggles FOV centerline is shown in the goggles display. The user then moves rifle 19 so that a reticle corresponding to the centerline of scope 17 is aligned with the goggles reticle and presses a “calibration” button.
- computer 30 receives position data for system 10 from GPS chipset 31 ( FIG. 2 ) and/or from data communication chipset 32 in block 105 .
- computer 30 receives data for a video frame (i.e., a scope image) from scope 17 and data for a goggles image from image generator 57 .
- Operation then proceeds to block 109 , where computer 30 receives angular pitch (P G ), roll (R G ) and yaw (Y G ) orientation data for goggles 11 from sensor 13 .
- Computer 30 also receives pitch (P S ), roll (R S ) and yaw (Y S ) data from sensor 18 .
- Data values from sensors 13 and 18 indicate the angle of vertical rise (pitch), the angle of horizontal rotation (yaw), and the angle of rotation around the line of sight (roll), for both goggles 11 and scope 17 , which because of calibration share the same coordinate system.
- the pitch, roll and yaw angle values from (goggles) IMU 13 and the pitch, roll and yaw angle values from (scope) IMU 18 are used to create an IMU-based rotation matrix.
- the IMU-based rotation matrix is then used in block 113 to calculate a location (i.e., horizontal and vertical position) and an image rotation for the scope image within the goggles image (and thus, the correct placement of weapon view 74 in user display 70 ).
- Rotation matrices and the use of same to place and orient one image within another are known in the art and/or would be readily apparent to persons of ordinary skill in the art once such persons are provided with the information contained herein.
- computer 30 adjusts for differences between scope 17 (and a scope image) and array 56 (and a goggles image).
- the goggles FOV in some embodiments is 40° and the scope FOV is 18°.
- array 56 and a similar sensor array in scope 17 may have different numbers of pixels and/or different pixel pitches. Adjusting for these and other differences between scope 17 and goggles 11 so as to determine a linear distance up or down on user display 70 (and an image rotation) is within the ability of persons skilled in the art once such persons are supplied with information provided herein.
- FIGS. 6A through 6H positioning and rotating a weapon view 74 within display 70 based on data from sensors 13 and 18 is generally illustrated.
- FIGS. 6A through 6H assume data from sensors 13 and 18 is accurate (e.g., not significantly affected by bias drift) and that weapon view 74 is generated based on the IMU sensor data only.
- HUD 73 and various icons and other indicia included in FIG. 4 are omitted from FIGS. 6B, 6D, 6F and 6H .
- goggles 11 are looking in exactly the same direction that rifle 19 is pointed. This results in a centered weapon view 74 ( FIG. 6B ), and computer 30 places an image from scope 17 directly over the center of goggles display 82 .
- FIG. 6C rifle 19 has yawed eight degrees to the left.
- FIG. 6D depicts the subsequent change in user display 70 .
- the pitch and roll values detected by sensors 13 and 18 remain unchanged, but the yaw value detected by sensor 18 will change by minus eight degrees (or positive eight degrees, depending on the chosen convention).
- the proper location for the scope image will be ⁇ 8 degrees from the HUD centerline. Accordingly, the placement of weapon view 74 in the heads up display is shifted to the left as shown (the same result would have been obtained if rifle 19 was held still and goggles 11 yawed eight degrees to the right).
- weapon view 74 remains at the edge of HUD 73 .
- computer 30 gives weapon view 74 a distinctive border to indicate to the user that rifle 19 is pointing outside the visual field of goggles 11 .
- Weapon view 74 remains visible in such embodiments so as to provide the user a view of where the weapon is pointing (e.g., allowing a user to aim and shoot around a corner).
- rifle 19 has been pitched upward six degrees.
- the yaw and roll values detected by sensors 13 and 18 remain unchanged, but the pitch values from sensor 13 and from sensor 18 differ by six degrees. This results in weapon view 74 being shifted upward within display 70 as shown in FIG. 6F .
- parallax is in many applications a non-issue. At least some implementations of system 10 are predominantly intended for use against targets at distances greater than 10 meters. Moreover, when targeting a weapon using system 10 , weapon view 74 will ultimately be the source of assurance that the weapon is pointed at a proper target. Even if weapon view 74 is slightly misaligned with the surrounding portions of the goggles view, the soldier will be primarily concerned that the weapon is pointed at the correct target, and weapon view 74 will still accurately show the intended target.
- FIGS. 6G and 6H show rotation of weapon view 74 in display 70 .
- computer 30 calculates an amount by which weapon view 74 should be rotated within display 70 so as to be properly registered with the remainder of display 70 .
- Various algorithms for rotating an image by a desired amount are known in the art. In the example of FIGS. 6G and 6H the image rotation of weapon view 74 is ten degrees.
- block 117 the IMU-based calculation for position and rotation of weapon view 74 within display 70 is checked using an image-based method. This permits computer 30 to adjust the manner in which IMU-based positions and rotations are calculated, thereby correcting for bias drift and helping to maintain proper registration of the scope image within the goggles image.
- registration refers to positioning of a scope image (or a portion of that scope image) within a goggles image so that the two images are properly aligned and positioned, and one image coincides with the other.
- IMU-based orientation determination is relatively simple from a computational standpoint and is generally quite robust over short periods of time, IMUs can have certain disadvantages.
- IMUs have bias drift rates and may be susceptible to significant overshoot if moved rapidly.
- the relative orientation of goggles 11 and scope 17 can be independently deduced by processing image data from image generator 57 and scope 17 if there is sufficient image content and contrast and if similar imaging technologies (e.g., microbolometers, CCD, etc.) are used.
- image data from image generator 57 and scope 17 e.g., microbolometers, CCD, etc.
- the image from scope 17 can be considered a dilated, translated and rotated distortion of the image generated from image generator 57 .
- the angle between the viewing vector of array 56 (i.e., the goggles FOV centerline) and the viewing vector of scope 17 (the scope FOV centerline) should not be too great (i.e., so the image from scope 17 is contained within the goggles image). If the angle is too great as determined by IMU readings (such as when an operator looks away from where the weapon is pointed), the image comparison algorithm is (in some embodiments) not invoked until the IMU data indicates that the angle is within acceptable limits for image comparison.
- the distance between goggles 11 and scope 17 should also be small compared to the distance from goggles 11 and scope 17 to objects being viewed. This ensures that parallax effects do not disrupt a simplifying assumption that the goggles FOV centerline and the scope FOV centerline are co-located.
- the location of the fixed point constrains the solution of the proper orientation of the scope image relative to the goggles image up to a twist rotation.
- the twist rotation is about an axis corresponding to a line defined by the center of projection of the scope camera and the point u′,v′ (i.e., the fixed point) on the scope camera focal plane. If the Brouwer fixed point for the goggles and scope images is at the center of the scope image, the twist rotation corresponds to the difference between the roll axis orientations of goggles 11 and scope 17 .
- the Brouwer fixed point can be found by scanning and comparing a goggles image and a scope image and determining a location where a rotationally invariant similarity metric is maximized.
- One such metric is the peak to sidelobe ratio (PSR) of a cross correlation of two linear signals.
- PSR peak to sidelobe ratio
- One of those signals is from a small annular region centered about a location in a goggles image and the other is from an annular region (of the same angular size) centered about a location in the scope image. If the PSR for those two annular regions is the maximum, the centers of those annular regions represent the Brouwer fixed point.
- PSR can be defined as (PEAK ⁇ mean)/ ⁇ , where PEAK is a peak cross correlation value for the centers of the two annular regions, and where mean and ⁇ are the respective mean and standard deviations of cross correlation values for the annular regions.
- the twist rotation about the fixed point can be deduced from the phase shift of the two signals.
- the phase shift is proportional to the displacement of the cross correlation peak used in the PSR calculation between the two signals. That is, starting from the zero point of each annulus (as defined by image orientation) and taking signal values around the annulus, determine the angular displacement from zero for each peak. The difference between these two angles is the phase shift.
- FIG. 7 is a flow chart providing additional details about the operations performed within block 117 of FIG. 5B in some embodiments.
- computer 30 performs an image-based check on a point of interest in the portion of the goggles image identified (using IMU data from sensors 13 and 18 ) as corresponding to the center of the scope image.
- Point of interest refers to a point in an image (a center of a portion of the goggles image in the current example) that is being evaluated as possibly corresponding to a point on the other image (the center of the scope image in the current example).
- the image-based check passes, the values for the IMU-based rotation matrix (and thus, the center for weapon view 74 predicted by that IMU-based rotation matrix) are verified, and computer 30 proceeds to block 204 on the “yes” branch. If the image-based check fails, computer 30 proceeds on the “no” branch to block 206 . As described in more detail below, computer 30 will in block 206 and in subsequent blocks expand the search (primarily in the global yaw plane) to perform image-based checks on additional points of interest in the goggles image.
- FIGS. 8A through 8G conceptually illustrate a case in which the image-based check verifies the position for the scope image calculated with IMU data from sensors 13 and 18 .
- FIGS. 6B, 6D, 6F and 6H HUD 73 and other elements shown in FIG. 4 have been omitted from FIGS. 8A through 8G .
- FIG. 8A shows a goggles image 82
- FIG. 8B shows a scope image 302 .
- the user is holding his head level but has tilted rifle 19 twenty degrees (20°) to the left. Shown in FIG.
- FIG. 8A (with broken lines) is the portion 303 of goggles image 82 where computer 30 intends (relying on the IMU-based rotation matrix) to place scope image 302 .
- the center 304 of portion 303 is also indicated; so as to avoid confusion with the aiming reticle of the weapon view, the center is shown as a “+” inside of a circle.
- the broken line 303 in FIG. 8A only shows the portion of goggles image 82 over which scope image 302 of FIG. 8B would be overlaid.
- the intended rotation of scope image 302 within goggles image 82 (according to the IMU-based rotation matrix) is shown in FIG. 8A as the arcuate dimension “20°.”
- computer 30 performs an image-based check of point 304 so as to evaluate the validity of point 304 as the center of a weapon view. Stated differently, computer 30 initially assumes that points 304 and 305 are the Brouwer fixed point. As part of the image-based check of point 304 , several PSRs are calculated using a region of goggles image portion 303 and regions of scope image 302 . As seen in FIG. 8C , a first PSR calculation is based on a cross correlation using the pixel intensity values of the equally-sized annular regions 308 and 309 respectively centered on the center points 304 and 305 of goggles image portion 303 and scope image 302 .
- the PSR value from this calculation (PSR[ 304 : 305 ]) is then compared to PSRs calculated for annular regions centered on locations around center 305 of scope image 302 .
- another PSR (PSR[ 304 : 305 - 1 ]) is calculated based on a cross correlation using the pixel intensity values of the equally-sized annular regions 308 and 309 - 1 respectively centered on points 304 and 305 - 1 of goggles image portion 303 and scope image 302 .
- PSR[ 304 : 305 - 2 ], PSR[ 304 : 305 - 3 ] and PSR[ 304 : 305 - 4 ] are calculated based on cross correlations using the pixel intensity values of equally-sized annular regions 308 and 309 - 2 (centered on point 305 - 2 , as shown in FIG. 8E ), equally-sized annular regions 308 and 309 - 3 (centered on point 305 - 3 , as shown in FIG. 8F ) and equally-sized annular regions 308 and 309 - 4 (centered on point 305 - 4 , as shown in FIG. 8G ).
- Point 304 passes the image-based check (i.e., the assumption that points 304 and 305 are the Brouwer fixed point is validated) when computer 30 determines that PSR[ 304 : 305 ], PSR[ 304 : 305 - 1 ], PSR[ 304 : 305 - 2 ], PSR[ 304 : 305 - 3 ] and PSR[ 304 : 305 - 4 ] are within an acceptable range of one another.
- An acceptable PSR range will depend on the characteristic performance of the sensor technology (microbolometer, etc.) being used in the imaging sources (scope 17 and array 56 in the present example), as well as on peculiarities of a given manufacturer's implementation of a particular class of sensor.
- the acceptable PSR range for a given combination of a goggles sensor type and a scope sensor type can be determined by comparing a series of exemplar images obtained using samples of those two sensor types. Determining an acceptable range is within the routine ability of a person of ordinary skill in the art once such a person is provided with the information contained herein.
- an image-based check of a point of interest calculates PSR values using regions centered on more or fewer points around the center 305 of scope image 302 . Those points around center 305 could also have locations (relative to center 305 ) other than as shown for points 305 - 1 through 305 - 4 .
- PSRs for an image-based check of a goggles image point of interest are based on annular regions centered on points around that goggles image point of interest instead of annular regions centered on points surrounding the scope image.
- computer 30 moves to block 204 from the “yes” branch of block 202 if the image-based check for the IMU-based center of the weapon view passes.
- computer 30 determines if there is a roll difference between the two images. In particular, computer 30 calculates the phase shift of the scope image and goggles image and deduces a twist rotation from the phase shift. If the difference between the twist rotation and the IMU-based roll difference is within acceptable limits (e.g., less than five degrees), the IMU-based value (and thus, the rotation of the scope image within the goggles image predicted by IMU data) is verified, and operation proceeds to block 119 of FIG. 5B . If the difference between the twist rotation and IMU-based rotation is not within acceptable limits, computer 30 proceeds on the “no” branch from block 204 to block 218 . Block 218 is discussed below.
- a failed image-based check in block 202 indicates that the IMU-based location for scope image 302 within goggles image 82 may be incorrect.
- the scope image and the part of the goggles image over which computer 30 intends (based on IMU data) to overlay the scope image do not sufficiently represent the same part of the external environment.
- computer 30 proceeds on the “no” branch from block 202 to block 206 .
- block 206 computer 30 attempts to find the proper location for scope image center 305 within goggles image 82 by comparing scope image 302 with different parts of goggles image 82 .
- computer 30 performs image-based checks for points of interest in goggles image 82 that are slightly offset from a center calculated from IMU data. This is illustrated conceptually in the example of FIGS. 8H through 8K , which assumes the same goggles image 82 and scope image 302 of FIGS. 8A and 8B , but in which the IMU-based position and image rotation are not correct.
- FIG. 8H shows goggles image 82 and a portion 311 of goggles image 82 where computer 30 now intends (relying on inaccurate IMU-based information) to place scope image 302 .
- the center 313 of portion 311 and the IMU-based image rotation angle (17°) are also shown.
- the correct position of scope image 302 within goggles image 82 is shown with a different type of broken line.
- FIG. 8I shows an enlarged area of goggles image 82 around center 313 .
- computer 30 performs image-based checks for each of points 315 a through 315 h . As seen in FIG. 8I , these points are slightly offset from center 313 .
- the image-based check of point 315 g is based on cross correlations of the pixel intensity values for annular region 316 (centered on point 315 g , as seen in FIG. 8J ) and each of equally-sized annular regions 309 through 309 - 4 shown in FIGS. 8C through 8G . Similar image-based checks are performed for the points 315 a - 315 f and 315 h.
- computer 30 determines if any of the image-based checks of block 206 passed. If so, computer 30 proceeds to block 216 , which block is explained below. If not, computer 30 proceeds on the “no” branch to block 210 .
- computer 30 expands its search and performs image-based checks on points in the goggles image that are even further offset from IMU-based center 313 . This is shown in FIG. 8K , where further offset locations 315 i through 315 hh are indicated. In at least some embodiments, the expanded search is biased in the global yaw plane. As shown in FIG.
- computer 30 proceeds to block 212 and determines if any of the image-based checks of block 210 passed. If so, computer 30 selects the location (e.g., one of points 315 i through 315 hh ) for which the highest PSR was obtained (e.g., the highest PSR for a point of interest that passed) and proceeds to block 216 along the “yes” branch. If none of the image-based checks in block 210 resulted in a pass, computer 30 proceeds on the “no” branch to block 214 . In block 214 , computer 30 determines if there are additional regions of goggles image 82 with which a portion of scope image 302 can be compared.
- computer 30 returns on the “yes” branch to block 210 and further expands its search to include locations that are farther away from center 313 . If computer 30 determines in block 214 that there are no more regions of goggles image 82 with which scope image 302 can be compared, computer 30 proceeds on the “no” branch to block 119 ( FIG. 5B ). In at least some embodiments, computer 30 treats a failure of all points of interest to pass an image-based check as evidence that there is temporarily insufficient contrast and/or content in the scope or goggles image to perform a PSR calculation, or that the scope FOV is not within the goggles FOV.
- any adjustment of IMU-based position calculation using image comparison results will wait until a better set of scope and goggles images are available.
- calibration can be performed manually (typically in less than ten seconds). Adjustment of IMU-based position calculation is described in more detail below.
- computer 30 When computer 30 does determine that a point of interest in the goggles image has passed an image-based check, computer 30 proceeds to block 216 of FIG. 7 (either from the “yes” branch of block 208 or from the “yes” branch of block 212 ). In block 216 , computer 30 identifies a portion of the goggles image centered on the point for which the highest PSR was found. Computer 30 then proceeds to block 218 and calculates the twist rotation of scope image 302 . The calculations of blocks 206 through 216 determine the amount by which the scope image is translated relative to the goggles image. The relative rotation of the two images (if any) is calculated separately in block 218 . As indicated above, this calculation is in some embodiments based on the phase shift of scope image 302 and the portion of goggles image 82 identified in block 216 .
- Computer 30 then proceeds to block 219 , where the IMU-based goggles portion ( 311 in the example of FIG. 8H ) and rotation matrix are replaced with the portion identified in block 216 and with the twist rotation (from block 218 ). If block 219 is reached from the “no” branch of block 204 , only the (IMU-based) roll is replaced. Computer 30 also adjusts the manner of IMU-based position calculation. This is illustrated conceptually in FIGS. 9A through 9C .
- FIG. 9A shows a goggles image 350 , the center point 351 for an IMU-based location for a scope image and the center point 352 for an image-comparison-based location for the scope image.
- IMU is the horizontal location of the scope image in the goggles image calculated with data from IMU sensors 13 and 18 .
- Image is the horizontal location of the scope image in the goggles image calculated using the image comparison technique described above.
- IMU is the vertical location of the scope image in the goggles image calculated with data from IMU sensors 13 and 18 ;
- Image is the vertical location of the scope image in the goggles image calculated using the image comparison technique described above.
- IMU are shown as linear distances in FIGS.
- FIGS. 9A and 9B the difference between H
- FIG. 9C shows values R
- computer 30 proceeds to block 119 of FIG. 5B .
- the scope image is cropped, resized and/or rotated.
- the resulting image is then overlaid on the goggles image as the weapon view (block 121 ).
- a reticle (which corresponds to the scope FOV centerline) may be generated, but in some embodiments an integral scope image reticle will be displayed as part of the image.
- the weapon view may be given a desired shape (e.g., rectangular or circular) and angular dimension(s), and portions of the scope view outside of the desired shape may be cropped.
- Computer 30 proceeds from block 121 to block 123 , where a tactical object database is updated.
- a tactical object database is updated.
- user display 70 graphically indicates locations of various real-world objects of interest to the user. Data for such objects is stored in a database, accessible by computer 30 , located in memory 40 (see FIG. 2 ), and periodically updated by messages received from other friendly units by means of the communications chip set 32 .
- FIG. 10 shows an example of a database 401 , according to at least some embodiments, storing information about real-world objects.
- the real-world objects are items of tactical interest to a soldier; accordingly, such items may also be referred to herein as “tactical objects.”
- Database 401 in tabular form for purposes of explanation.
- Database 401 need not be stored or organized as a table, however, and need not have other features shown in FIG. 10 .
- a database such as database 401 could be stored in numerous other formats, could include information other than that shown in FIG. 10 , and/or could be stored as multiple separate databases.
- the first column in FIG. 10 contains an identifier for an object of interest. For convenience, relatively simple identifiers are shown in FIG. 10 . The identifiers could take any form, however. In some cases, an identifier may be a name (e.g., of another soldier) or other characters that a human user would recognize, and that might be displayed in user display 70 under some circumstances.
- the second column of FIG. 10 indicates an object type. Although only four object types are shown in database 401 , numerous other types of objects could be included. For example, mission objectives, friendly outposts or emplacements, obstacles, contaminated areas, magnetic headings, stadia marks and other items can be included within database 401 and shown in a user display with an icon or other indicator.
- the third column of database 401 gives location information for each object.
- the location information provides the three-dimensional location of an object within a particular tactical region.
- the tactical region may be relatively small (e.g., several square miles that a military unit will patrol). In other cases, the tactical region may be much larger.
- the location data in FIG. 10 is represented generically as ⁇ x coord>, coord> (with a lower-case “y” in this case referring to a linear distance axis and not yaw) and ⁇ z coord>.
- actual location information may be in the form geographical coordinates, military grid reference system coordinates, or in any other desired format.
- the fourth column of FIG. 10 shows the type of icon or other graphical object used to mark or otherwise indicate a tactical object in user display 70 . Although the icon type is shown graphically in FIG. 10 for convenience, an actual database may contain a pointer or other data object corresponding to a particular icon.
- terrain and other geographic information is also stored in database 401 .
- This geographic information includes elevations of points in the tactical region, true north corrections for magnetic north values obtained from a magnetometer in sensor 13 , magnetic dip angle to determine if there is interference to the compass, etc.
- Data can be input into database 401 in various ways. Some data might be downloaded into database 401 (e.g., via an RS232 or USB port on computer 30 ) from a laptop computer as part of planning for a mission. Examples of such data could include planned routes, known enemy positions, etc. Other data is added to database 401 , while system 10 is in use, based on periodic updates over the radio communications link. Block 123 of FIG. 5B represents such updating of database 401 via radio communications. For example, changing locations of friendly personnel can be relayed from other systems 10 used by such personnel. As another example, an enemy unit location may be reported by a soldier using system 10 , and that enemy unit location is then broadcast to other soldiers (each of whom is also equipped with a system 10 ). Radio updates may also come from sources other than (or in addition to) soldiers equipped with systems 10 . As but one example, a remotely located commander or an airborne observation unit may provide tactical information about enemy units, changed mission objectives, etc.
- FIG. 11 is a flow chart providing additional details about the operations performed within block 125 of FIG. 5B .
- FIGS. 12A through 12D are used to conceptually explain various operations of FIG. 11 .
- Computer 30 would not necessarily generate the diagrams shown in FIGS. 12A through 12D as part of the operations described in connection with FIG. 11 .
- computer 30 first determines the location of system 10 based on GPS coordinates and/or other data received in block 105 of FIG. 5A .
- FIG. 12A a plan view of a portion of the tactical region for which information is stored in database 401 .
- Various friendly soldiers, enemy positions and other tactical objects are shown using the same types of icons shown in FIGS. 4 and 10 .
- Each of the tactical objects has one or more entries in database 401 .
- the position of system 10 (and thus of goggles 11 ) is also indicated.
- block 503 FIG.
- computer 30 uses magnetometer output from sensor 13 , or (in case of magnetic interference) the yaw Euler angle calculated from the IMU movements subsequent to the last acceptable magnetic fix, to determine the direction of the goggles FOV (i.e., the direction in which the user is looking).
- the value of MagNorth is corrected (using geographically-specific magnetic deviation and true-north correction data in database 401 ) so as to determine the direction in which goggles 11 are pointed. This is shown in FIG. 12B .
- FIG. 12B Also shown in FIG. 12B is the goggles FOV and HUD FOV.
- Flow then proceeds to block 505 of FIG. 11 where pitch axis data from goggles sensor 13 is used to determine the degree to which the user is looking up or down. This is shown in FIG. 12C .
- Computer 30 then proceeds to block 507 .
- computer 30 calculates a spatial volume corresponding to the boundary of HUD 73 in user display 70 .
- This HUD volume is a pyramid centered on the goggles FOV centerline and extending outward.
- the HUD FOV e.g., 32°
- the HUD pyramid is bounded by a plane located at a range (d) from goggles 11 . This is shown in FIG. 12D .
- Computer 30 searches database 401 for all objects within the just-calculated HUD pyramid.
- the range (d) is in some embodiments selected by the user. For example, a user might be more concerned about objects that are relatively close (e.g., within 500 meters) than about those that are further away. Moreover, providing icons or other display indicia for distant objects may unnecessarily clutter user display 70 . In certain embodiments, range (d) may be changed while system 10 is operating and the user display can be de-cluttered in other ways. As but one example, route information, control measures, etc. may be removed at the push of a button during a firefight so as to minimize potential distracters to the soldier, while friendly icons would be retained to help limit fratricide. Removed information could be restored by the operator when again needed.
- computer 30 then generates icons or other indicia in user display 70 corresponding to the objects, identified in the database search of block 509 , within the HUD pyramid.
- the size of the icons or other indicia is logarithmically scaled so that icons or other indicia for more distant objects are smaller.
- Computer 30 then returns to block 127 ( FIG. 5B ). If more video frames from goggles 11 and scope 17 are received (i.e., goggles 11 and scope 17 remain ON), then computer 30 returns to block 105 of FIG. 5A . Otherwise, operation of system 10 terminates.
- the weapon view can be a magnified version of the scope view (either at all times or selectively at the request of the user).
- the image-based comparison techniques described above can first be performed using an unmagnified scope image. Once the proper location and orientation for the scope image are deduced, the scope image can be magnified and cropped prior to overlay as the weapon view.
- Other types of information e.g., a battery life indicator, navigational headings, etc.
- a user can be provided with a great deal of flexibility in selecting the types of information to be shown on the display.
- Such selection can take place during set-up or while the system is in use. For example, a soldier on patrol may choose to see display indicators for routes, rally points, indirect fire targets, preplanned control measures, etc. When the soldier is attacked or is about to enter a firelight, he or she might then place the weapon in a close-combat mode (e.g., by pressing a mode control button located on a weapon). In such a close-combat mode, the user could avoid information overload by suppressing all information except icons for friendly forces. In some cases, the soldier might even choose to suppress “friendly” icons and opt for an audible warning if the weapon is pointed at friendly personnel.
- a soldier might configure his system to display icons for friendly forces within a first range (e.g., 300 meters) and for enemy positions within a second range (e.g., 1000 meters), to display graphics for patrol routes within a third range (e.g., 500 meters), to plot control measures with routes, etc.
- a first range e.g. 300 meters
- a second range e.g. 1000 meters
- a third range e.g. 500 meters
- location information can be provided via GPS transmissions.
- Location data can also be augmented by local information such as timing and ranges from nearby soldiers also equipped with a system such as described herein.
- Other types of navigational data e.g., WAAS, or wide area augmentation system
- Barometric pressure can be used to provide a relative elevation error and used as a check against a GPS-based elevation, as well as during system initialization.
- a terrain database e.g., within a database such as database 401
- Data from various sources can be updated at various rates (e.g., video frames at 30 Hz, IMU data at 100 Hz, GPS or other navigational updates at 1 Hz, range data from other soldiers at 8 Hz).
- IMU accelerometers can have bandwidths greater than 1000 Hz, resolution of 1.25 milligrams, and 2% sensitivity.
- a magnetometer can provide outputs at 200 Hz and have a sensitivity of 1 mGauss (1 part in 300-600 Earth field).
- Rate sensors can have a 2% error and 100 Hz bandwidth, and barometric pressure data provided at 100 Hz and within 1-2 meters of resolution. Range data from other soldiers may be within 1.3 meters of error.
- all of these values are only examples, and the invention is not limited to systems having specific values for any of these quantities.
- FIG. 7 Other embodiments also include numerous variations on the above-described methods for using a comparison of a scope image and a goggles image to check an IMU-based location and rotation for that scope image within that goggles image.
- the operations described in connection with FIG. 7 initially assume that the center for a scope image within a goggles image calculated from IMU data is correct. If the image comparison suggests that the IMU-based calculation is not correct, a search of other possible center locations moves outward from the IMU-based center. Assuming an IMU drift rate of approximately 1°/hour, a procedure such as that of FIG. 7 works well (and minimizes computational load on a processor) if the scope and goggles point in the same direction at least once every 10 to 15 minutes.
- a system could be configured so that a user can manually initiate a search (e.g., beginning at the goggles image center) across the entire goggles image for a scope image location. This could be used for initially calibrating the IMUs, or as a re-calibration if the user has not pointed his weapon forward for an extended period (e.g., if the weapon was slung over a shoulder).
- the search patterns shown in FIGS. 8J and 8K are only examples. Numerous other search patterns could be implemented.
- FIGS. 13A and 13B show a goggles images and scope image, respectively, that will be used to describe this alternate algorithm. Unlike the previous discussion, which assumed the fields of view for both goggles array 56 and scope 17 were conical, the images of FIGS. 13A and 13B instead assume that array 56 and a similar array inside of scope 17 are rectangular.
- FIG. 13A is a goggles image 601 generated when array 56 ( FIG. 3 ) is a 320 ⁇ 240 pixel microbolometer array (8-12 micron range) having a 14.4° horizontal FOV, a 10.8° vertical FOV, and an 18° diagonal FOV.
- FIG. 13B is a scope image 603 generated when an array inside scope 17 ( FIG. 3 ) is also 320 ⁇ 240 pixel microbolometer array (8-12 micron range) having a 14.4° horizontal FOV, a 10.8° vertical FOV, and an 18° diagonal FOV.
- the data from the scope array is also converted into a 640 ⁇ 480 pixel NTSC format image for transmission to control unit 16 and digitized for processing.
- the horizontal, vertical and diagonal goggles fields of view are the same as the horizontal, vertical and diagonal scope fields of view.
- the below-described alternative image comparison algorithm can also be used in connection with a goggles and/or scope image created from different arrays having different fields of view (including the conical fields of view described previously), as well as in circumstances when the fields of view for the goggles and the scope are different in horizontal, vertical and/or diagonal planes. Adaptation of the below described alternative algorithm to such other array configurations is within the routine ability of persons of ordinary skill in the art once such persons are provided with the information contained herein.
- each pixel corresponds to approximately 0.0225° of the FOV in each of the horizontal, vertical and diagonal planes.
- each of images 601 and 603 is based on data from a 320 ⁇ 240 pixel array.
- Each array pixel would thus correspond to approximately 0.045° of the FOV in each of the horizontal, vertical and diagonal planes.
- gray-scale values from the 320 ⁇ 240 pixel arrays are duplicated so as to create images 601 and 603 .
- a pixel in 320 ⁇ 240 array 56 outputting a gray-scale value of G corresponds to a 2 ⁇ 2 pixel block in image 601 , with all four pixels in that 2 ⁇ 2 block also having a gray-scale value of G. The significance of this will be apparent below.
- a center 604 is also indicated. Surrounding center 604 is a circle 605 having a diameter corresponding to 4° of the FOV. The region within circle 604 will be overlaid on goggles image 601 as a weapon view (similar to weapon view 74 of FIG. 4 ).
- FIG. 14 is a flow chart for an alternate image comparison algorithm according to at least some embodiments. Various blocks in the FIG. 14 flow chart will be further explained using diagrams and plots in FIGS. 15-22 .
- the alternative image comparison algorithm of FIG. 14 assumes that current IMU data is available from sensors 13 and 18 ( FIG. 3 ), that IMU drift since the last calibration (or image registration) has not been significant, and that IMU data is thus usable as a starting point in the algorithm (as described below).
- random noise in IMU output rarely exceeds 1 degree in yaw or 1 ⁇ 2 degree in pitch or roll.
- center area 605 is wholly contained within goggles image 601 (i.e., that the portion of the external environment corresponding to center area 605 is also represented by a region wholly contained in goggles image 601 ) at time intervals such that the bias drift will be less than 2 degrees.
- the image comparison algorithm of FIG. 14 is entered from block 115 of FIG. 5B .
- the algorithm of FIG. 14 is also performed by computer 30 ( FIG. 2 ).
- computer 30 resets a counter (N).
- the counter N is used to prevent excessive looping through the algorithm.
- computer 30 proceeds to block 703 and locates a region of high contrast, within scope image 603 , that is relatively near center 604 . This is shown in more detail by FIG. 15 .
- FIG. 15 is a partially schematic diagram in which the various image elements (e.g., trees, foliage, etc.) have been omitted for simplicity, but in which the boundary of scope image 603 and center 604 are shown.
- computer 30 searches image 603 for a square region 610 in which the pixels inside of region 610 have a high degree of contrast.
- computer 30 searches for the square region 610 in which the standard deviation of pixel gray-scale values is maximized.
- image 603 comprises numerous 2 ⁇ 2 pixel blocks in which all pixels in a 2 ⁇ 2 block have the same gray-scale value. Accordingly, when calculating a standard deviation for a region of image 603 , only every other pixel in the horizontal and vertical directions need be used.
- region 610 corresponds to a 3° horizontal FOV by 3° vertical FOV portion of image 603 .
- region 610 is approximately 133 pixels by 133 pixels.
- the center 606 of region 610 should also be within 2° of center 604 in the horizontal plane and within 1° of center 604 in the vertical plane.
- computer 30 thus searches for a 133 ⁇ 133 pixel region contained within a 311 ⁇ 222 pixel region of image 603 that is centered about center 604 .
- computer 30 proceeds from block 703 to block 705 and determines if the region 610 identified in block 703 is of sufficiently high contrast. If, for example, the maximum standard deviation for gray-scale values within any 133 ⁇ 133 pixel block is not sufficiently high, there may be insufficient contrast to continue with an image comparison. A sufficiently high standard deviation value could be determined for a given combination of goggles and scope imaging array types based on a series of test images. If the standard deviation for gray scale values in the region 610 identified in block 703 is not sufficiently high, computer 30 returns to block 119 of FIG. 5B via the “yes” branch. If the standard deviation of gray-scale values for the region 610 identified in block 703 is sufficiently high, computer continues with the image comparison and proceeds on the “no” branch to block 707 .
- FIG. 16 is a partially schematic diagram in which the various image elements (e.g., trees, foliage, etc.) have been omitted for simplicity, but in which the boundary of goggles image 601 is shown.
- Point 607 is the point which IMU data indicates corresponds to point 606 in the scope image. Also shown in FIG. 601 is a 1° by 1° (e.g., 44 pixels by 44 pixels) region 613 centered on point 607 . The significance of region 613 is explained below.
- computer 30 proceeds from block 707 to block 709 .
- computer 30 obtains pixel gray-scale values from 128 locations on a circle around center 606 of region 610 ( FIG. 15 ).
- FIG. 17A a schematic diagram of pixels surrounding center 606 in a portion of scope image 603 .
- Each of the squares in FIG. 17A represents one pixel of scope image 603 .
- Surrounding center 606 are 128 locations (shown as diamonds) that are equally spaced on a circle having a diameter of approximately 1°.
- computer 30 obtains the gray-scale value for the pixel in which the diamond's center is located.
- values are not limited to those for every other pixel.
- gray-scale values may be obtained for multiple pixels within a 2 ⁇ 2 pixel block corresponding to a single pixel of the microbolometer array in scope 17 . Indeed, there might be instances in which two of the points on the circle are located within the same pixel of image 603 .
- computer 30 proceeds from block 709 to block 711 .
- computer 30 performs an operation similar to that of block 709 , but instead using goggles image 601 .
- computer 30 obtains pixel gray-scale values from 128 locations on a circle around point 607 in goggles image 601 ( FIG. 16 ). This is shown in more detail in FIG. 17B , a schematic diagram of pixels surrounding point 607 in a portion of goggles image 601 .
- gray-scale values may be obtained for multiple pixels within a 2 ⁇ 2 pixel block corresponding to a single pixel of the microbolometer array 56 in goggles 11 .
- goggles image 601 and scope image 603 have the same horizontal, vertical and diagonal fields of view and are of the same pixel dimensions. This need not be the case, however. If one of images 601 or 603 was of a different pixel dimension or had a different field of view, the procedures of blocks 709 and 711 would be similar. In particular, gray-scale values would be obtained from pixels corresponding to the same number of points distributed on similarly-dimensioned circles in each image. If, for example, a goggles image has a 32° horizontal FOV, a 24° vertical FOV and a 40° diagonal FOV, each pixel in that image represents approximately 0.1°. An even distribution of 128 points on a 1° circle would be generally as shown in FIG. 17C . In FIG.
- point 607 ′ is a point (determined using IMU data) corresponding to the center of a high contrast region found in block 703 ( FIG. 14 ).
- Each square in FIG. 17C represents one pixel of a goggles image having the larger FOV values.
- FIG. 18 is a plot showing an example of raw data that might be obtained in blocks 709 and 711 .
- the plots of FIG. 18 and of subsequent FIGS. 19 and 20 are provided for purposes of explanation, and would not necessarily be generated by computer 30 .
- the solid line plot represents the 128 gray-scale values obtained in one of blocks 709 and 711
- the broken line plot represents the 128 values obtained in the other of blocks 709 and 711 .
- a 10% noise factor and a 22.5 degree phase shift (representing a rotation of one image relative to the other) have been added.
- computer 30 first performs a discrete Fast Fourier Transform (FFT) on each data set (i.e., a discrete FFT on the 128 scope image values and a discrete FFT on the 128 goggles image values).
- FFT Fast Fourier Transform
- Computer 30 then multiplies the complex conjugate of one of those discrete FFTs by the other of the discrete FFTs.
- Computer 30 next obtains an inverse FFT of the product.
- Computer 30 subtracts the DC component of the inverse FFT and obtains data that allows peak-to-sidelobe ratio (PSR) comparisons.
- PSR peak-to-sidelobe ratio
- FIG. 19 is a plot of the results of a calculation in block 713 ( FIG. 14 ) when the center points do coincide.
- the sidelobes in FIG. 19 can be seen as three regions (on either side of the peak) that are higher than their surroundings. The location of the peak can also be used to determine the rotation of one image relative to the other. In the example of FIG.
- FIG. 20 is an example of a plot resulting from the calculations of block 713 using random data.
- block 715 If the result of the evaluation in FIG. 14 block 715 is favorable (e.g., if the PSR 3 ), computer 30 proceeds to block 717 via the “Fay.” branch and stores the PSR, rotation and the pixel coordinates of the point in the goggles image 601 that was used in block 711 (e.g., point 607 ) as the circle center. Computer 30 then continues to block 719 . If the result in block 715 is not favorable (e.g., PSR ⁇ 3), computer 30 proceeds on the “Unfav.” branch to block 719 without storing a PSR, center location or rotation.
- FIGS. 21 and 22 show a pattern by which additional points in region 613 are selected in at least some embodiments.
- FIG. 21 schematically shows region 613 and location 607 (found in FIG. 14 block 707 ), with each of the squares representing a single pixel.
- FIG. 22 is an enlargement of the portion of region 613 indicated in FIG. 21 .
- computer 30 selects the pixel at position ⁇ circle around (2) ⁇ .
- Computer 30 then returns to block 711 and repeats block 711 using pixel position ⁇ circle around (2) ⁇ instead of point 607 , and then repeats blocks 713 through 719 (possibly omitting block 715 ).
- pixel position ⁇ circle around (3) ⁇ is selected. This continues until some stop condition is reached.
- a stop condition is reached when the pattern shown generally in FIG. 22 reaches the sides of region 613 along the horizontal plane.
- the pattern of FIG. 22 weights the search based on a higher probability that any random noise or bias drift will be greater along the yaw axis than along the pitch or roll axes.
- the stop condition is reached after every other point in both the horizontal and vertical directions has been checked.
- computer 30 proceeds to block 723 and determines if any of the data sets (with each set containing a PSR, rotation and center point location) stored in block 717 is unambiguously the best.
- the “best” data set is one in which the PSR is a predetermined percentage above the other stored PSRs. The percentage by which a “best” point should exceed others depends on the characteristics of the imaging technology involved, but one example value is an excess of 10%.
- computer 30 proceeds to block 725 .
- block 725 (which is substantially similar block 219 of FIG. 7 )
- computer 30 updates the IMU-based position for weapon view 74 based on the position and rotation of scope image region 605 within goggles image 601 resulting from the unambiguously best data set.
- Computer 30 also uses that data set to adjust the manner in which subsequent IMU-based position and image rotation calculations are made. This adjustment is performed as previously explained in connection with FIGS. 9A-9C .
- computer 30 proceeds to block 119 of FIG. 5B .
- computer 30 proceeds to block 727 . If counter N has reached a maximum vale (N MAX ), computer 30 returns (via the “yes” branch) to block 119 of FIG. 5B . Appropriate values of N MAX would depend of various design preferences (e.g., processor speed, acceptable latency, etc.). If counter N has not reached N MAX , computer 30 proceeds on the “no” branch to block 729 and increments counter N, and then proceeds to block 731 . In block 731 a new high contrast region is selected in the scope image.
- N MAX a maximum vale
- the selection of block 731 identifies the 3° ⁇ 3° pixel block having the next highest gray-scale value standard deviation, and which does not overlap a previously selected 3° ⁇ 3° pixel block in image 603 by more than a predetermined amount.
- Computer 30 then returns to block 707 .
- the search pattern of block 721 the sizes of image regions evaluated, and various other parameters can be varied.
- the above-described algorithm employs center 604 as one reference point and point 607 as a corresponding reference point.
- a point other than the center of the scope image can be used as the initial reference point.
- individual pixel gray-scale values are the image element data points used in the FFT analyses. Other image element data points could be used (e.g., an average of gray-scale values for several pixels clustered around a point, wavelength (color) information if sensors provide color information).
- 17A through 17C have a particular dimensioned arrangement relative to the reference point (center 604 ) and the corresponding reference point (point 607 ).
- different arrangements e.g., oval, square, concentric rings, etc.
- different dimensions e.g., closer to or farther from the reference point and corresponding reference point
- data sets are stored in block 717 for only the five highest (or some other predetermined number of highest) PSR values.
- the scope image has a small number of possible FOVs (e.g., 9° diagonal or 18° diagonal).
- drift may become obvious to the user. In such a case, the user can manually recalibrate the system.
- a heads up display need not be associated with a pair of goggles.
- the heads up display could appear before a windshield in a vehicle.
- a weapon mounted on the vehicle includes a video gun sight producing images processed and projected onto the heads up display.
- an orientation sensor may be placed to sense the orientation of the vehicle rather than a pair of goggles worn by the observer. This embodiment may be particularly useful for remotely controlled weapon systems, for example a robot carrying a weapon.
- controller generically refers to any of one or more FPGAs, microprocessors, ASICs, other types of computational devices, or combinations thereof. It is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. In the claims, various portions are prefaced with letter or number references for convenience. However, use of such references does not imply a temporal relationship not otherwise required by the language of the claims.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Optics & Photonics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computer Graphics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Controls And Circuits For Display Device (AREA)
- Processing Or Creating Images (AREA)
- Aiming, Guidance, Guns With A Light Source, Armor, Camouflage, And Targets (AREA)
Abstract
Description
- This application is a continuation of U.S. patent application Ser. No. 15/483,086, titled “System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display” and filed Apr. 10, 2017, which application is a continuation of U.S. patent application Ser. No. 14/950,643, titled “System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display” and filed Nov. 24, 2015, which application is a continuation of U.S. patent application Ser. No. 11/680,207, titled “System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display” and filed Feb. 28, 2007. U.S. patent application Ser. No. 11/680,207, U.S. patent application Ser. No. 14/950,643, and U.S. patent application Ser. No. 15/483,086, in their entireties, are incorporated by reference herein.
- Augmented reality systems offer a mechanism for providing a user with additional data about his or her environment. An example of such a system is a heads-up display (HUD) found in aircraft and automobiles. As is generally known, a HUD projects various navigational and/or operational data on a windshield or other transparent surface in a user's field of view. This allows a user to monitor various information without having to divert his or her gaze from the external environment.
- Augmented reality systems have also been developed for use in a combat military environment. For example, commonly-owned U.S. patent application Ser. No. 11/000,934 (filed Dec. 2, 2004, titled “System and Method for Video Image Registration in a Heads Up Display,” published as Pub. No. US20060121993, and incorporated by reference herein) describes a system in which an image from a rifle-mounted video source is superimposed on an image seen through a pair of goggles. Sensors coupled to the rifle and to the goggles provide data indicating movement of the goggles and rifle. An image from the rifle-mounted source shows an external region within the source's field of view (FOV). The goggles have a wider FOV and provide an image that includes a portion showing the same region as is shown in the image from the rifle-mounted video source. The sensor data is then used to determine the relative orientation of the two sources and calculate a location for the rifle image within the image seen through the goggles.
- Determining the relative orientations of two video sources based on inertial measurement unit (IMU) sensor data, as is described for at least some embodiments of the aforementioned application, can pose challenges. For example, many low-cost IMU sensors experience bias drift over time. Such drift can result in relative orientation errors of several degrees per hour. In order to prevent such errors from accumulating, the IMU sensors must be periodically recalibrated. This recalibration typically requires user action and can disrupt system operation. The ability to minimize the need for manually-initiated recalibration would be highly advantageous.
- Additional advantages could also be obtained by increasing the types of information provided by a HUD within goggles worn by a user.
- This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
- In at least some embodiments, a computer receives images from two video sources. Each of those two video sources is movable independent of the other and generates images that represent a portion of an external environment within its field of view. One of the video sources may be contained within a pair of goggles worn by a user, and the other source may be mounted to a rifle carried by the user. Sensors coupled to the two video sources provide data to the computer that indicates the spatial orientations of those sources. Using the sensor data, the computer determines a location for placing a video image (or a portion thereof) from a second of the sources (e.g., a rifle-mounted source) in the video image from a first of the sources (e.g., a goggles-mounted source). The second source video image and a corresponding portion of the first source video image represent the same part of the external environment. Data from the two images are then compared in order to evaluate the location determined from the sensor data. The sensor-based location is either confirmed, or a new location is found based on additional image comparisons. Once a location is selected (either a confirmed sensor-based location or a location found using image comparison), the two images are displayed such that the second source image (or a portion of that image) overlays a corresponding portion of the first source image. Locations obtained using image comparisons are used to calibrate (adjust) the manner in which subsequent sensor-based locations are determined.
- In at least some additional embodiments, a database contains information about external objects in the environment in which a system employing the two video sources is being used. The computer receives system location information and orientation information for one of the video sources (e.g., the source contained within a user's goggles). The computer uses the location and orientation information to calculate a spatial volume within the field of view of that video source. The computer then parses the database for information about objects that may be located within the calculated spatial volume. After parsing the database, icons or other graphical indicia are displayed for objects identified as being within the calculated spatial volume, the positions of the icons being imposed upon the locations of the objects.
- The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.
-
FIG. 1 illustrates a system, according to at least some embodiments, for providing an information-enhanced heads up display (HUD). -
FIG. 2 is a block diagram of the control unit for the system ofFIG. 1 . -
FIG. 3 is a block diagram showing data flows in the system ofFIG. 1 . -
FIG. 4 is an example of a user display in the system ofFIG. 1 . -
FIGS. 5A and 5B are a flow chart explaining operation of the system ofFIG. 1 . -
FIGS. 6A through 6F illustrate positioning of one video image within another based on data from inertial measurement units. -
FIGS. 6G and 6H illustrate rotation of one video image within another based on data from inertial measurement units. -
FIG. 7 is a flow chart providing additional details for a block in the flow chart ofFIGS. 5A and 5B . -
FIGS. 8A through 8K illustrate checking and/or correcting an IMU-based position for one video image within another video image. -
FIGS. 9A through 9C illustrate correction of an IMU-based position calculation based on image comparison results. -
FIG. 10 is an example of a tactical database. -
FIG. 11 is a flow chart providing additional details for another block in the flow chart ofFIGS. 5A and 5B . -
FIGS. 12A through 12D show identification of real-world objects based on a spatial volume calculated from a position and orientation. -
FIGS. 13A and 13B show a goggles images and scope image, respectively, used to describe an alternate image comparison algorithm. -
FIG. 14 is a flow chart for an alternate image comparison algorithm. -
FIG. 15 is a partially schematic diagram corresponding to the scope image ofFIG. 13B . -
FIG. 16 is a partially schematic diagram corresponding to the goggles image ofFIG. 13A . -
FIGS. 17A through 17C show collection of gray-scale values from scope and goggles images. -
FIG. 18 is a plot showing an example of gray-scale values. -
FIG. 19 is a plot of the results of calculations in which image region center points coincide. -
FIG. 20 is a plot of the results of calculations similar to those used to generate the plot ofFIG. 19 , but using random data. -
FIGS. 21 and 22 show a pattern by which additional points are selected. -
FIG. 1 illustrates asystem 10, according to at least some embodiments, that provides an information-enhanced heads-up display (HUD) for an infantryman or other armed tactical operator. For simplicity, a soldier or otherperson using system 10 will henceforth be referred to as a “user.”System 10 includes a set ofgoggles 11 that are configured for wear by the user.Goggles 11 may be worn directly, may be attached to a helmet or other headgear (not shown), or worn in some other manner. The front side ofgoggles 11 faces outward (i.e., away from the user) when worn, and includeseyepieces 12 and other apertures (not shown) for receiving light or other (e.g., IR) input from the user's field of view. The rear side ofgoggles 11 faces the user when worn such that the user looks outward througheyepieces 12 and sees external objects within the goggles field of view. As explained in more detail below in connection withFIG. 3 ,goggles 11 includes an image generator and a projector and creates a user display that combines video images and graphical indicia with what a user sees as a result of light passing from the external environment througheyepieces 12. Asensor 13 is attached togoggles 11 or otherwise physically coupled so as to move as a single unit withgoggles 11.Sensor 13 includes an inertial measurement unit (IMU) and magnetometer. In at least some embodiments,sensor 13 is an IM3™ sensor (available from Archangel Systems, Inc. of Auburn, Ala.) that uses accelerometers and rate sensors to sense and report angular orientation around three axes of rotation up to 100 times per second. The image projector and generator ingoggles 11 andsensor 13 communicate (overcables wearable control unit 16.Control unit 16 includes a computer, radio receiver and other elements, as is also described below. -
System 10 further includes a video source (or “scope”) 17 and asensor 18 configured to move as a single unit withscope 17. In the embodiment shown,scope 17 is affixed to arifle 19 and is used to targetrifle 19. In particular,scope 17 is aligned with the barrel ofrifle 19 such that a reticle corresponding to the optical centerline ofscope 17 generally follows the path of a bullet fired fromrifle 19.Scope 17 may include a visible-light video camera, a thermal imaging (i.e., IR sensitive) video camera, a night-vision video camera, or some combination thereof. In certain embodiments,scope 17 is a Light Weapon Thermal Sight (available from DRS Optronics of Melbourne, Fla.).Sensor 18 is also an IMU (such as the aforementioned IM3™ sensor) in at least some embodiments.Sensor 18 andscope 17 communicate withcontrol unit 16 viacables - As is also explained in more detail in conjunction with
FIG. 3 ,control unit 16 receives video feeds fromscope 17 andgoggles 11. The video feeds may be delivered using any standard video format, for example analog formats like NTSC or PAL, digital formats like MPEG, or a non-standard format.Sensors unit 16. Using information fromsensors control unit 16 determines orientation of the line of sight forgoggles 11 relative to the line of sight forscope 17. As described in more detail below, this permits superimposition of an image fromscope 17 onto a display corresponding to a field of view ofgoggles 11. - In at least some embodiments,
sensors cables video cables control unit 16 communicates wirelessly withsensors goggles 11 and/orscope 17 in other embodiments. For example, ultra-wideband (UWB) transceivers may transmit video and sensor data fromrifle 19, as well as video and sensor data fromgoggles 11. Likewise, UWB may be used to transmit video from the control unit togoggles 11. Although UWB radios, such as Time Domain's PulsON® radio, are particularly desirable for their high bandwidth, low power consumption and for being virtually undetectable, any wireless standard may be used, including both Bluetooth and IEEE 802.11. - In alternative embodiments, UWB radios may be used for more than transmission of video and sensor data. Multiple radios may be placed on
rifle 19 and on goggles 11 (or on a helmet, to which the goggles may be affixed), each of which can relay their precise position. In this fashion,control unit 16 is able to calculate the alignment of the rifle and goggles based on the relative location of radios rather than using separate orientation sensors. -
FIG. 2 is a block diagram ofcontrol unit 16.Control unit 16 includes acomputer 30, global positioning system (GPS)chipset 31 and data communication chip set 32. GPS chip set 31 receives signals from GPS satellites viaantenna 33 and outputs GPS data tocomputer 30. Based on that GPS data,computer 30 determines the location of system 10 (and thus of the user) to within the accuracy limits of the particular GPS chip set. For differential GPS systems intended for military use, the horizontal position information is typically accurate to less than three meters. RF data communication chip set 32 provides two-way data communication (using antenna 34) betweensystem 10 and other elements in a tactical network. Those other elements may include additional systems 10 (e.g., worn by other soldiers in a squad), a remotely located commander, etc. The communication link(s) serviced by chip set 32 may include a Land Warrior radio local net, single channel ground and airborne radio system (SINCGARS), Force XXI Battle Command Brigade-and-Below (FBCB2) tracking system, etc. -
Computer 30 includes abus 35 connected to GPS chip set 31 and local communication chip set 32 viaGPS interface 36 andcommunication interface 37, respectively. Aprocessor 38 communicates withinterfaces memory 40 viabus 35.Memory 40 includes a combination of volatile (e.g., random access memory, or RAM) and non-volatile memory (e.g., removable and/or non-removable FLASH, EEPROM, hard drive, etc.). Stored inmemory 40 are instructions for performing operations ofsystem 10 as described herein and a tactical database (described below). Additional components communicating withprocessor 38 viabus 35 include avideo input interface 41,video output interface 42 and sensor interfaces 43-45.Video input interface 41 receives video input fromscope 17 andgoggles 11 and makes that input available toprocessor 38. Similarly, sensor interfaces 43 and 44 receive input fromsensors processor 38. Other sensors (e.g., barometric sensors) may also provide input throughinterface 45.Processor 38 provides video output, viainterface 42, togoggles 11 for presentation in a display generated withingoggles 11. -
FIG. 2 is merely one example of a control unit. In other embodiments, the components and/or functions shown may be combined and/or divided in various ways. As but one example, a single input interface could receive all sensor and video feed data. As but another example, some or all of the operations performed byprocessor 38 and other elements ofcontrol unit 16 could be performed by one or more application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. In some such embodiments, some or all of the operations described herein as being performed byprocessor 38 are hardwired into one or more ASICs as gates and other logic dedicated to the calculations and other operations described herein. -
FIG. 3 is a block diagram showing data flows insystem 10.Scope 17 receives electromagnetic input (e.g., visible light, infrared) from within its field of view. The field of view forscope 17 is shown schematically inFIG. 3 with dotted lines and labeled “scope FOV.” In at least some embodiments, the scope FOV is 18°. Also shown inFIG. 3 is an optical centerline (or “line of sight”) for scope 17 (also referred to herein as the “scope FOV centerline”). For ease of explanation, it is assumed in certain of the following examples that the scope FOV is a cone symmetrically centered on the scope optical centerline. This need not be the case, however, and the invention includes embodiments in which the scope FOV is asymmetric and/or differently shaped and/or not centered on an optical centerline. An example of an image from a scope with a non-conical FOV is described below in connection withFIG. 13B .Scope 17 outputs a video feed corresponding to an image of external objects within the scope FOV. For simplicity, a video image output fromscope 17 will be referred to as a “scope image.”Sensor 18 is coupled to scope 17 (because of both components being attached to rifle 19) and outputs data (PS, RS, YS) reflecting the rotational orientation ofscope 17 about pitch (P), roll (R) and yaw (Y) axes (as shown inFIG. 1 ). In a similar manner,sensor 13 coupled togoggles 11 outputs data (PG, RG, YG) reflecting the rotational orientation ofgoggles 11 about pitch (P), roll (R) and yaw (Y) axes. Although one example of pitch, roll and yaw axes is shown inFIG. 1 , other axes may be used. In at least some embodiments,sensors sensor 13 also includes a magnetometer. The magnetometer senses the earth's magnetic field and provides a signal (MagNorth) indicative of magnetic north. - In addition to sensor data and a
scope 17 video signal,control unit 16 also receives periodic position data updates viaantenna 33 and tactical data viaantenna 34. As described above, position data viaantenna 33 includes data from GPS satellites. Tactical data includes data providing positions and descriptions of various elements (e.g., friendly or hostile forces) in the environment in whichsystem 10 is operating. Other types of data may be received from one or moreadditional sensors 55. In at least some embodiments,sensor 55 is a barometer providing data indicating changes in the elevation ofsystem 10, which data can be used to confirm and/or adjust GPS-based elevation data. -
Goggles 11 includes anarray 56 of electromagnetic receptors (e.g., charge-coupled devices, photodiodes or other photo-sensitive elements, microbolometers) that receive electromagnetic radiation (e.g., visible light, IR) from within the receptor array field of view (“goggles FOV”). As with the scope FOV, some of the examples herein assume that the goggles FOV is a cone symmetrically centered about an optical centerline/line of sight (also referred to herein as the “goggles FOV centerline”) extending fromgoggles 11. This configuration is not required, however. An example of an image from a goggles array with a non-conical FOV is described below in connection withFIG. 13A . Visible light from the goggles FOV also passes througheyepiece 12 and can be seen by the user. Althougheyepiece 12 andarray 56 are offset slightly from one another, the eyepiece FOV (not indicated inFIG. 3 ) is the same or larger than the array field of view (i.e., the goggles FOV inFIG. 3 ) and generally has the same center line.Array 56 generates image data (e.g., a night vision view of the environment within the goggles FOV). This image data is then provided to animage generator 57 that processes the image data into an image of objects in the goggles FOV. For convenience, an image corresponding to the data fromarray 56 will be referred to as a “goggles image.” As explained in more detail below, the goggles image may be modified bycomputer 30 prior to being displayed to a user. Accordingly, the goggles image is also output vialine 58 tocomputer 30. A video feed from computer 30 (which may contain the goggles image modified to include the scope image and other graphics) is received byprojector 59 vialine 61.Projector 59, throughbeam splitter 60, combines the goggles image with the visible light passing directly througheyepiece 12. The goggles image can also be provided directly toprojector 59 from image generator 57 (e.g., ifcomputer 30 orscope 17 becomes inoperative but night vision capability is still desired). A heads-up display (HUD) field of view (HUD FOV) is also shown inFIG. 3 , and is further discussed below. - For simplicity,
FIG. 3 only shows one array, one eyepiece, one projector and one beam splitter. In at least some embodiments, there is a separate arrangement of array, eyepiece, projector and beam splitter for each eye.Goggles 11 in some embodiments is a modified version of the AN/PVS-21 low-profile night vision goggles with M-2755 Enhanced Heads Up Display available from Sensor Technology Systems, Inc. of Beavercreek, Ohio. The modifications to a standard AN/PVS-21 (or other night vision goggles or enhanced vision system) required to implement various embodiments of the invention will be apparent to persons of skill in the art once such persons are provided with the information included herein. -
FIG. 4 shows an example of auser display 70 provided bygoggles 11. For convenience, “user display” is used to refer to the image projected by projector 59 (FIG. 3 ) throughbeam splitter 60 intoeyepiece 12, and which is visible by a wearer ofgoggles 11. A user display may include the goggles image (or a portion thereof), the scope image (or a portion thereof) and other graphical objects. The example ofFIG. 4 assumessystem 10 is being used in a forest setting, though such a setting is merely for purposes of illustration. Located within the goggles FOV (and thus in goggles image 82) are numerous trees and bushes such as might be present in a forest environment, as well as soldiers 71 (partially behind a tree in the lower left) and 72 (partially covered by foliage in the upper right).User display 70 is also slightly shaded so as to indicate that the goggles image is a night vision display. The outer circle ofFIG. 4 generally corresponds to the limit of the goggles FOV. The HUD portion ofuser display 70 is shown as arectangular region 73 in the center portion of the goggles FOV. As explained in further detail below, various graphical indicia are overlaid withinHUD 73. Also overlaid onHUD 73 is aweapon view 74 corresponding to (and generated from) the scope image. For convenience, some of the examples herein assume that the weapon view is the same size as the scope image. This is not a requirement, however. The scope image may be cropped to provide a weapon view that is actually smaller than the scope image (or that corresponds to reduced scope FOV). AlthoughHUD 73 is shown as a rectangle andweapon view 74 is shown as a circle inFIG. 4 , other shapes could be used. In some embodiments, for example,weapon view 74 is rectangular. Portions of the scope image at the outer edges of the scope FOV may similarly be cropped so as to give a weapon view a desired shape. - As discussed below, the location and rotation of
weapon view 74 withinuser display 70 is determined bycomputer 30 based on output fromsensors rifle 19 is moved, scope images (or portions thereof) are dynamically positioned withinuser display 70 so as to indicate where scope 17 (and thus rifle 19) is pointing. In this manner, a user ofsystem 10 is able to simultaneously survey a setting, acquire a target and point a weapon at the target without having to remove his or her goggles and adjust to a weapon sight. - An aiming
reticle 76 is located at the center ofweapon view 74, and corresponds to the rifle aim point (as zeroed to the rifle at a specified range) and lies near the scope FOV centerline. Various other graphical objects are also included in the HUD. For example, “friendly”icons soldiers system 10 is aware thatsoldiers display 70 is anicon 79 indicating the location of (or direction to) a rally point (i.e., a position determined in advance of a tactical operation and at which persons involved in the operation may regroup).Icon 80 shows a reported location of an enemy machine gun position.Colored path 81 indicates a preplanned route. Additional details of generating the graphical objects indisplay 70, as well as examples of other graphical objects that could be included indisplay 70, are provided below. -
FIGS. 5A and 5B are a flow chart explaining the operation ofsystem 10. The steps shown inFIGS. 5A and 5B (and in other flow charts described below) can be reordered, combined, split, replaced, etc. For example,FIGS. 5A and 5B are not intended to indicate a requirement that all steps be performed as part of a single algorithm. Many aspects ofsystem 10 operation could be accommodated on multiple independently-running parallel program threads or as processes running in parallel on an ASIC, an FPGA, multiple processors, etc. Moreover, all of the operations shown in the flow charts ofFIGS. 5A through 5B would not necessarily be performed the same number of times as is suggested by the flow chart. As but one example, refreshing and/or updating of icons (or of certain icons) might occur at a different rate than updating and/or repositioning ofweapon view 74. - Beginning in
block 101, a user activatessystem 10, andgoggles 11,control unit 16 andscope 17 are energized.Sensors system 10 remains motionless for a period of time so as to allow rate sensors and accelerometers insensors IMU sensors sensors sensor 18 onrifle 19 might arbitrarily choose a zero-yaw position that corresponds to rifle 19 pointing along a 37 degree compass heading, andsensor 13 ongoggles 11 might arbitrarily choose a zero-yaw position that corresponds to the optical centerline of the goggles FOV pointing along a 246 degree compass heading. In at least some embodiments, this difference in zero-yaw orientations is determined by the user looking at a distant (e.g., over 300 meters away) object through goggles. A calibration reticle corresponding to the goggles FOV centerline is shown in the goggles display. The user then movesrifle 19 so that a reticle corresponding to the centerline ofscope 17 is aligned with the goggles reticle and presses a “calibration” button. - After initial calibration,
computer 30 receives position data forsystem 10 from GPS chipset 31 (FIG. 2 ) and/or fromdata communication chipset 32 inblock 105. Inblock 107,computer 30 receives data for a video frame (i.e., a scope image) fromscope 17 and data for a goggles image fromimage generator 57. Operation then proceeds to block 109, wherecomputer 30 receives angular pitch (PG), roll (RG) and yaw (YG) orientation data forgoggles 11 fromsensor 13.Computer 30 also receives pitch (PS), roll (RS) and yaw (YS) data fromsensor 18. Data values fromsensors goggles 11 andscope 17, which because of calibration share the same coordinate system. - In
block 111, the pitch, roll and yaw angle values from (goggles)IMU 13 and the pitch, roll and yaw angle values from (scope)IMU 18 are used to create an IMU-based rotation matrix. The IMU-based rotation matrix is then used inblock 113 to calculate a location (i.e., horizontal and vertical position) and an image rotation for the scope image within the goggles image (and thus, the correct placement ofweapon view 74 in user display 70). Rotation matrices and the use of same to place and orient one image within another are known in the art and/or would be readily apparent to persons of ordinary skill in the art once such persons are provided with the information contained herein. As part of the IMU-based location and image rotation calculation,computer 30 adjusts for differences between scope 17 (and a scope image) and array 56 (and a goggles image). For example, the goggles FOV in some embodiments is 40° and the scope FOV is 18°. Moreover,array 56 and a similar sensor array inscope 17 may have different numbers of pixels and/or different pixel pitches. Adjusting for these and other differences betweenscope 17 andgoggles 11 so as to determine a linear distance up or down on user display 70 (and an image rotation) is within the ability of persons skilled in the art once such persons are supplied with information provided herein. - Turning briefly to
FIGS. 6A through 6H , positioning and rotating aweapon view 74 withindisplay 70 based on data fromsensors FIGS. 6A through 6H assume data fromsensors weapon view 74 is generated based on the IMU sensor data only. For convenience,HUD 73 and various icons and other indicia included inFIG. 4 are omitted fromFIGS. 6B, 6D, 6F and 6H . InFIG. 6A ,goggles 11 are looking in exactly the same direction thatrifle 19 is pointed. This results in a centered weapon view 74 (FIG. 6B ), andcomputer 30 places an image fromscope 17 directly over the center of goggles display 82. - In
FIG. 6C ,rifle 19 has yawed eight degrees to the left.FIG. 6D depicts the subsequent change inuser display 70. The pitch and roll values detected bysensors sensor 18 will change by minus eight degrees (or positive eight degrees, depending on the chosen convention). When this is compared to the values detected by sensor 13 (which have not changed in the present example), the proper location for the scope image will be −8 degrees from the HUD centerline. Accordingly, the placement ofweapon view 74 in the heads up display is shifted to the left as shown (the same result would have been obtained ifrifle 19 was held still andgoggles 11 yawed eight degrees to the right). In at least some embodiments, for scope/goggles movements that would otherwise moveweapon view 74 beyond the HUD width,weapon view 74 remains at the edge ofHUD 73. In such a case, however,computer 30 gives weapon view 74 a distinctive border to indicate to the user thatrifle 19 is pointing outside the visual field ofgoggles 11.Weapon view 74 remains visible in such embodiments so as to provide the user a view of where the weapon is pointing (e.g., allowing a user to aim and shoot around a corner). - In
FIG. 6E ,rifle 19 has been pitched upward six degrees. The yaw and roll values detected bysensors sensor 13 and fromsensor 18 differ by six degrees. This results inweapon view 74 being shifted upward withindisplay 70 as shown inFIG. 6F . - For ease of explanation, parallax is ignored in the explanation of
FIGS. 6A through 6F . Because video sources withingoggles 11 andscope 17 are separated by some distance (e.g. 0.5 meters, typically mostly vertical), the devices will in fact be looking at slightly different points even if they are in perfect alignment. As a result, in processing a video frame fromscope 17, the location where the frame is placed may be slightly off, and a displayed frame of video will not be aligned as perfectly as possible. However, this problem diminishes as the distance to a target increases. For example, a target at 10 meters with 0.5 meters betweenscope 17 andgoggles 11 produces an error of about 2.9 degrees in the placement of a weapon view. At 100 meters, with the same 0.5 meters betweenscope 17 andgoggles 11, the error is only 0.29 degrees. For larger distances, image comparison position calculations (described below) compensate for errors caused by parallax. - At shorter distances, parallax is in many applications a non-issue. At least some implementations of
system 10 are predominantly intended for use against targets at distances greater than 10 meters. Moreover, when targeting aweapon using system 10,weapon view 74 will ultimately be the source of assurance that the weapon is pointed at a proper target. Even ifweapon view 74 is slightly misaligned with the surrounding portions of the goggles view, the soldier will be primarily concerned that the weapon is pointed at the correct target, andweapon view 74 will still accurately show the intended target. Additionally, if it is known a priori that targets will be at a short range (e.g., during a room-clearing operation), calibration can be performed using a target at the presumed average range of real targets to be encountered (rather than at 300+ meters), thus minimizing the effects of parallax. -
FIGS. 6G and 6H show rotation ofweapon view 74 indisplay 70. Based on the data received fromsensors computer 30 calculates an amount by whichweapon view 74 should be rotated withindisplay 70 so as to be properly registered with the remainder ofdisplay 70. Various algorithms for rotating an image by a desired amount are known in the art. In the example ofFIGS. 6G and 6H the image rotation ofweapon view 74 is ten degrees. - Returning to
FIG. 5A , flow proceeds fromblock 113 to block 117 ofFIG. 5B . Inblock 117, the IMU-based calculation for position and rotation ofweapon view 74 withindisplay 70 is checked using an image-based method. This permitscomputer 30 to adjust the manner in which IMU-based positions and rotations are calculated, thereby correcting for bias drift and helping to maintain proper registration of the scope image within the goggles image. As used herein, “registration” refers to positioning of a scope image (or a portion of that scope image) within a goggles image so that the two images are properly aligned and positioned, and one image coincides with the other. - Although IMU-based orientation determination is relatively simple from a computational standpoint and is generally quite robust over short periods of time, IMUs can have certain disadvantages. For example, IMUs have bias drift rates and may be susceptible to significant overshoot if moved rapidly. To address these concerns, the relative orientation of
goggles 11 andscope 17 can be independently deduced by processing image data fromimage generator 57 andscope 17 if there is sufficient image content and contrast and if similar imaging technologies (e.g., microbolometers, CCD, etc.) are used. By checking the IMU-based position and image rotation calculations using an image comparison, the manner in which data from IMU sensors is used can be periodically recalibrated and the effects of bias drift, etc. reduced. - If certain conditions are satisfied, the image from
scope 17 can be considered a dilated, translated and rotated distortion of the image generated fromimage generator 57. The angle between the viewing vector of array 56 (i.e., the goggles FOV centerline) and the viewing vector of scope 17 (the scope FOV centerline) should not be too great (i.e., so the image fromscope 17 is contained within the goggles image). If the angle is too great as determined by IMU readings (such as when an operator looks away from where the weapon is pointed), the image comparison algorithm is (in some embodiments) not invoked until the IMU data indicates that the angle is within acceptable limits for image comparison. The distance betweengoggles 11 andscope 17 should also be small compared to the distance fromgoggles 11 andscope 17 to objects being viewed. This ensures that parallax effects do not disrupt a simplifying assumption that the goggles FOV centerline and the scope FOV centerline are co-located. - In at least some embodiments, image-based position calculation relies on Brouwer's Fixed Point Theorem. This theorem holds that a continuous mapping from a planar region onto itself is guaranteed to have one point whose coordinates are left unchanged by the mapping. If the scope image is a dilated, translated and rotated distortion of the goggles image, then there will be at least one point in the goggles image that has the same position in both the goggles and scope images. For example, assume the position of the fixed point is at (u,v) in local coordinates of the goggles image and at (u′,v′) in local coordinates of the scope image. Brouwer's theorem holds that u=u′ and v=v′. The location of the fixed point constrains the solution of the proper orientation of the scope image relative to the goggles image up to a twist rotation. The twist rotation is about an axis corresponding to a line defined by the center of projection of the scope camera and the point u′,v′ (i.e., the fixed point) on the scope camera focal plane. If the Brouwer fixed point for the goggles and scope images is at the center of the scope image, the twist rotation corresponds to the difference between the roll axis orientations of
goggles 11 andscope 17. - The Brouwer fixed point can be found by scanning and comparing a goggles image and a scope image and determining a location where a rotationally invariant similarity metric is maximized. One such metric is the peak to sidelobe ratio (PSR) of a cross correlation of two linear signals. One of those signals is from a small annular region centered about a location in a goggles image and the other is from an annular region (of the same angular size) centered about a location in the scope image. If the PSR for those two annular regions is the maximum, the centers of those annular regions represent the Brouwer fixed point. Numerous definitions of PSR are known in the art and can be used in different embodiments. For example, PSR can be defined as (PEAK−mean)/σ, where PEAK is a peak cross correlation value for the centers of the two annular regions, and where mean and σ are the respective mean and standard deviations of cross correlation values for the annular regions.
- Once the Brouwer fixed point is found, and assuming the scope image and goggles image are within a sufficiently close wavelength range (e.g., if the
scope 17 andarray 56 are both sensitive to IR at wavelengths between 8 and 12 microns) the twist rotation about the fixed point can be deduced from the phase shift of the two signals. The phase shift is proportional to the displacement of the cross correlation peak used in the PSR calculation between the two signals. That is, starting from the zero point of each annulus (as defined by image orientation) and taking signal values around the annulus, determine the angular displacement from zero for each peak. The difference between these two angles is the phase shift. -
FIG. 7 is a flow chart providing additional details about the operations performed withinblock 117 ofFIG. 5B in some embodiments. Inblock 202,computer 30 performs an image-based check on a point of interest in the portion of the goggles image identified (using IMU data fromsensors 13 and 18) as corresponding to the center of the scope image. “Point of interest” refers to a point in an image (a center of a portion of the goggles image in the current example) that is being evaluated as possibly corresponding to a point on the other image (the center of the scope image in the current example). If the image-based check passes, the values for the IMU-based rotation matrix (and thus, the center forweapon view 74 predicted by that IMU-based rotation matrix) are verified, andcomputer 30 proceeds to block 204 on the “yes” branch. If the image-based check fails,computer 30 proceeds on the “no” branch to block 206. As described in more detail below,computer 30 will inblock 206 and in subsequent blocks expand the search (primarily in the global yaw plane) to perform image-based checks on additional points of interest in the goggles image. - Turning briefly to
FIGS. 8A through 8G , the performance of an image-based check according to at least some embodiments is explained.FIGS. 8A through 8G conceptually illustrate a case in which the image-based check verifies the position for the scope image calculated with IMU data fromsensors FIGS. 6B, 6D, 6F and 6H ,HUD 73 and other elements shown inFIG. 4 have been omitted fromFIGS. 8A through 8G .FIG. 8A shows agoggles image 82, andFIG. 8B shows ascope image 302. In the example ofFIGS. 8A and 8B , the user is holding his head level but has tiltedrifle 19 twenty degrees (20°) to the left. Shown inFIG. 8A (with broken lines) is theportion 303 ofgoggles image 82 wherecomputer 30 intends (relying on the IMU-based rotation matrix) to placescope image 302. Thecenter 304 ofportion 303 is also indicated; so as to avoid confusion with the aiming reticle of the weapon view, the center is shown as a “+” inside of a circle. Thebroken line 303 inFIG. 8A only shows the portion ofgoggles image 82 over whichscope image 302 ofFIG. 8B would be overlaid. The intended rotation ofscope image 302 within goggles image 82 (according to the IMU-based rotation matrix) is shown inFIG. 8A as the arcuate dimension “20°.” - In
block 202 ofFIG. 7 ,computer 30 performs an image-based check ofpoint 304 so as to evaluate the validity ofpoint 304 as the center of a weapon view. Stated differently,computer 30 initially assumes thatpoints point 304, several PSRs are calculated using a region ofgoggles image portion 303 and regions ofscope image 302. As seen inFIG. 8C , a first PSR calculation is based on a cross correlation using the pixel intensity values of the equally-sizedannular regions goggles image portion 303 andscope image 302. The PSR value from this calculation (PSR[304:305]) is then compared to PSRs calculated for annular regions centered on locations aroundcenter 305 ofscope image 302. For example, and as shown inFIG. 8D , another PSR (PSR[304:305-1]) is calculated based on a cross correlation using the pixel intensity values of the equally-sizedannular regions 308 and 309-1 respectively centered onpoints 304 and 305-1 ofgoggles image portion 303 andscope image 302. Similarly, PSR[304:305-2], PSR[304:305-3] and PSR[304:305-4] are calculated based on cross correlations using the pixel intensity values of equally-sizedannular regions 308 and 309-2 (centered on point 305-2, as shown inFIG. 8E ), equally-sizedannular regions 308 and 309-3 (centered on point 305-3, as shown inFIG. 8F ) and equally-sizedannular regions 308 and 309-4 (centered on point 305-4, as shown inFIG. 8G ). - Point 304 passes the image-based check (i.e., the assumption that points 304 and 305 are the Brouwer fixed point is validated) when
computer 30 determines that PSR[304:305], PSR[304:305-1], PSR[304:305-2], PSR[304:305-3] and PSR[304:305-4] are within an acceptable range of one another. An acceptable PSR range will depend on the characteristic performance of the sensor technology (microbolometer, etc.) being used in the imaging sources (scope 17 andarray 56 in the present example), as well as on peculiarities of a given manufacturer's implementation of a particular class of sensor. The acceptable PSR range for a given combination of a goggles sensor type and a scope sensor type can be determined by comparing a series of exemplar images obtained using samples of those two sensor types. Determining an acceptable range is within the routine ability of a person of ordinary skill in the art once such a person is provided with the information contained herein. - Although five PSR values were obtained in the image-based check of the preceding example, this need not be the case. In other embodiments, an image-based check of a point of interest calculates PSR values using regions centered on more or fewer points around the
center 305 ofscope image 302. Those points aroundcenter 305 could also have locations (relative to center 305) other than as shown for points 305-1 through 305-4. In still other embodiments, PSRs for an image-based check of a goggles image point of interest are based on annular regions centered on points around that goggles image point of interest instead of annular regions centered on points surrounding the scope image. - Returning to
FIG. 7 , and as mentioned above,computer 30 moves to block 204 from the “yes” branch ofblock 202 if the image-based check for the IMU-based center of the weapon view passes. Inblock 204,computer 30 determines if there is a roll difference between the two images. In particular,computer 30 calculates the phase shift of the scope image and goggles image and deduces a twist rotation from the phase shift. If the difference between the twist rotation and the IMU-based roll difference is within acceptable limits (e.g., less than five degrees), the IMU-based value (and thus, the rotation of the scope image within the goggles image predicted by IMU data) is verified, and operation proceeds to block 119 ofFIG. 5B . If the difference between the twist rotation and IMU-based rotation is not within acceptable limits,computer 30 proceeds on the “no” branch fromblock 204 to block 218.Block 218 is discussed below. - Returning to block 202 of
FIG. 7 , a failed image-based check inblock 202 indicates that the IMU-based location forscope image 302 withingoggles image 82 may be incorrect. In other words, the scope image and the part of the goggles image over whichcomputer 30 intends (based on IMU data) to overlay the scope image do not sufficiently represent the same part of the external environment. In such a case,computer 30 proceeds on the “no” branch fromblock 202 to block 206. Inblock 206,computer 30 attempts to find the proper location forscope image center 305 withingoggles image 82 by comparingscope image 302 with different parts ofgoggles image 82. In particular,computer 30 performs image-based checks for points of interest ingoggles image 82 that are slightly offset from a center calculated from IMU data. This is illustrated conceptually in the example ofFIGS. 8H through 8K , which assumes thesame goggles image 82 andscope image 302 ofFIGS. 8A and 8B , but in which the IMU-based position and image rotation are not correct.FIG. 8H showsgoggles image 82 and aportion 311 ofgoggles image 82 wherecomputer 30 now intends (relying on inaccurate IMU-based information) to placescope image 302. Thecenter 313 ofportion 311 and the IMU-based image rotation angle (17°) are also shown. For convenience, the correct position ofscope image 302 withingoggles image 82 is shown with a different type of broken line. -
FIG. 8I shows an enlarged area ofgoggles image 82 aroundcenter 313. Inblock 206,computer 30 performs image-based checks for each ofpoints 315 a through 315 h. As seen inFIG. 8I , these points are slightly offset fromcenter 313. For example, the image-based check ofpoint 315 g is based on cross correlations of the pixel intensity values for annular region 316 (centered onpoint 315 g, as seen inFIG. 8J ) and each of equally-sizedannular regions 309 through 309-4 shown inFIGS. 8C through 8G . Similar image-based checks are performed for thepoints 315 a-315 f and 315 h. - Returning to
FIG. 7 ,computer 30 then proceeds to block 208. Inblock 208,computer 30 determines if any of the image-based checks ofblock 206 passed. If so,computer 30 proceeds to block 216, which block is explained below. If not,computer 30 proceeds on the “no” branch to block 210. Inblock 210,computer 30 expands its search and performs image-based checks on points in the goggles image that are even further offset from IMU-basedcenter 313. This is shown inFIG. 8K , where further offset locations 315 i through 315 hh are indicated. In at least some embodiments, the expanded search is biased in the global yaw plane. As shown inFIG. 8K , this results in a greater number of expanded search points of interest in the horizontal direction than in the vertical direction. The search is biased in the global yaw plane because bias drift in the pitch and roll planes is an order of magnitude less than in the yaw plane due to the steadying influence of a constant one-g gravitational signal in the accelerometers. Although not shown, the arrangement of points of interest checked duringblock 206 could also be biased in the global yaw plane. - From block 210 (
FIG. 7 ),computer 30 proceeds to block 212 and determines if any of the image-based checks ofblock 210 passed. If so,computer 30 selects the location (e.g., one of points 315 i through 315 hh) for which the highest PSR was obtained (e.g., the highest PSR for a point of interest that passed) and proceeds to block 216 along the “yes” branch. If none of the image-based checks inblock 210 resulted in a pass,computer 30 proceeds on the “no” branch to block 214. Inblock 214,computer 30 determines if there are additional regions ofgoggles image 82 with which a portion ofscope image 302 can be compared. If so,computer 30 returns on the “yes” branch to block 210 and further expands its search to include locations that are farther away fromcenter 313. Ifcomputer 30 determines inblock 214 that there are no more regions ofgoggles image 82 with whichscope image 302 can be compared,computer 30 proceeds on the “no” branch to block 119 (FIG. 5B ). In at least some embodiments,computer 30 treats a failure of all points of interest to pass an image-based check as evidence that there is temporarily insufficient contrast and/or content in the scope or goggles image to perform a PSR calculation, or that the scope FOV is not within the goggles FOV. Under such circumstances, any adjustment of IMU-based position calculation using image comparison results will wait until a better set of scope and goggles images are available. Alternatively, calibration can be performed manually (typically in less than ten seconds). Adjustment of IMU-based position calculation is described in more detail below. - When
computer 30 does determine that a point of interest in the goggles image has passed an image-based check,computer 30 proceeds to block 216 ofFIG. 7 (either from the “yes” branch ofblock 208 or from the “yes” branch of block 212). Inblock 216,computer 30 identifies a portion of the goggles image centered on the point for which the highest PSR was found.Computer 30 then proceeds to block 218 and calculates the twist rotation ofscope image 302. The calculations ofblocks 206 through 216 determine the amount by which the scope image is translated relative to the goggles image. The relative rotation of the two images (if any) is calculated separately inblock 218. As indicated above, this calculation is in some embodiments based on the phase shift ofscope image 302 and the portion ofgoggles image 82 identified inblock 216. -
Computer 30 then proceeds to block 219, where the IMU-based goggles portion (311 in the example ofFIG. 8H ) and rotation matrix are replaced with the portion identified inblock 216 and with the twist rotation (from block 218). Ifblock 219 is reached from the “no” branch ofblock 204, only the (IMU-based) roll is replaced.Computer 30 also adjusts the manner of IMU-based position calculation. This is illustrated conceptually inFIGS. 9A through 9C .FIG. 9A shows agoggles image 350, thecenter point 351 for an IMU-based location for a scope image and thecenter point 352 for an image-comparison-based location for the scope image. The value H|IMU is the horizontal location of the scope image in the goggles image calculated with data fromIMU sensors FIG. 9B , value V|IMU is the vertical location of the scope image in the goggles image calculated with data fromIMU sensors FIGS. 9A and 9B , persons ordinarily skilled in the art will appreciate that angular differences between the orientations ofgoggles 11 andscope 17 directly correspond to (and can be used to readily calculate) a rotation matrix that can be used to correct a calibration matrix that has drifted. - As shown in
FIGS. 9A and 9B , the difference between H|IMU and H|Image (ΔH) and the difference between V|IMU and V|IMAGE (ΔV) are used to update the rotation correction matrix derived from the initial calibration and subsequent updates until a new correction is available (e.g., as a result of a subsequent pass through blocks 202-219 ofFIG. 7 ). Finally,FIG. 9C shows values R|IMU (representing the difference between the roll orientations ofgoggles 11 andscope 17 calculated with data fromIMU sensors 13 and 18) and R|Image (the twist rotation calculated using the image comparison technique described above). R|IMU and R|Image are used (in a manner readily appreciable by persons of ordinary skill in the art) to correct IMU-based roll calculations until a new correction is available. In general, corrections to pitch and roll are much smaller than those to yaw due to the presence of three-axis accelerometers in both IMUs that yield a consistent gravity vector, thus correcting much of the drift in these directions. In some embodiments, pitch and roll corrections from image registration will not be applied until they exceed a pre-determined threshold because this effect due to gravity usually will keep these axes in control. - From
block 219 ofFIG. 7 ,computer 30 proceeds to block 119 ofFIG. 5B . Inblock 119, the scope image is cropped, resized and/or rotated. The resulting image is then overlaid on the goggles image as the weapon view (block 121). A reticle (which corresponds to the scope FOV centerline) may be generated, but in some embodiments an integral scope image reticle will be displayed as part of the image. As mentioned above, the weapon view may be given a desired shape (e.g., rectangular or circular) and angular dimension(s), and portions of the scope view outside of the desired shape may be cropped. -
Computer 30 proceeds fromblock 121 to block 123, where a tactical object database is updated. As discussed in connection withFIG. 4 , and as described in more detail below,user display 70 graphically indicates locations of various real-world objects of interest to the user. Data for such objects is stored in a database, accessible bycomputer 30, located in memory 40 (seeFIG. 2 ), and periodically updated by messages received from other friendly units by means of the communications chip set 32.FIG. 10 shows an example of adatabase 401, according to at least some embodiments, storing information about real-world objects. In the present example, the real-world objects are items of tactical interest to a soldier; accordingly, such items may also be referred to herein as “tactical objects.”FIG. 10 shows database 401 in tabular form for purposes of explanation.Database 401 need not be stored or organized as a table, however, and need not have other features shown inFIG. 10 . A database such asdatabase 401 could be stored in numerous other formats, could include information other than that shown inFIG. 10 , and/or could be stored as multiple separate databases. - The first column in
FIG. 10 contains an identifier for an object of interest. For convenience, relatively simple identifiers are shown inFIG. 10 . The identifiers could take any form, however. In some cases, an identifier may be a name (e.g., of another soldier) or other characters that a human user would recognize, and that might be displayed inuser display 70 under some circumstances. The second column ofFIG. 10 indicates an object type. Although only four object types are shown indatabase 401, numerous other types of objects could be included. For example, mission objectives, friendly outposts or emplacements, obstacles, contaminated areas, magnetic headings, stadia marks and other items can be included withindatabase 401 and shown in a user display with an icon or other indicator. The third column ofdatabase 401 gives location information for each object. The location information provides the three-dimensional location of an object within a particular tactical region. In some cases, the tactical region may be relatively small (e.g., several square miles that a military unit will patrol). In other cases, the tactical region may be much larger. For simplicity, the location data inFIG. 10 is represented generically as <x coord>, coord> (with a lower-case “y” in this case referring to a linear distance axis and not yaw) and <z coord>. However, actual location information may be in the form geographical coordinates, military grid reference system coordinates, or in any other desired format. The fourth column ofFIG. 10 shows the type of icon or other graphical object used to mark or otherwise indicate a tactical object inuser display 70. Although the icon type is shown graphically inFIG. 10 for convenience, an actual database may contain a pointer or other data object corresponding to a particular icon. - Although not shown in
FIG. 10 , terrain and other geographic information is also stored indatabase 401. This geographic information includes elevations of points in the tactical region, true north corrections for magnetic north values obtained from a magnetometer insensor 13, magnetic dip angle to determine if there is interference to the compass, etc. - Data can be input into
database 401 in various ways. Some data might be downloaded into database 401 (e.g., via an RS232 or USB port on computer 30) from a laptop computer as part of planning for a mission. Examples of such data could include planned routes, known enemy positions, etc. Other data is added todatabase 401, whilesystem 10 is in use, based on periodic updates over the radio communications link.Block 123 ofFIG. 5B represents such updating ofdatabase 401 via radio communications. For example, changing locations of friendly personnel can be relayed fromother systems 10 used by such personnel. As another example, an enemy unit location may be reported by asoldier using system 10, and that enemy unit location is then broadcast to other soldiers (each of whom is also equipped with a system 10). Radio updates may also come from sources other than (or in addition to) soldiers equipped withsystems 10. As but one example, a remotely located commander or an airborne observation unit may provide tactical information about enemy units, changed mission objectives, etc. - After updating
database 401,computer 30 proceeds to block 125 ofFIG. 5B . Inblock 125,computer 30 identifies tactical objects for which an icon or other graphical indicator is to be shown onuser display 70.Computer 30 then generates a display icon (or other visual indicator) for each of the identified objects.FIG. 11 is a flow chart providing additional details about the operations performed withinblock 125 ofFIG. 5B .FIGS. 12A through 12D are used to conceptually explain various operations ofFIG. 11 .Computer 30 would not necessarily generate the diagrams shown inFIGS. 12A through 12D as part of the operations described in connection withFIG. 11 . - In
block 501 ofFIG. 11 ,computer 30 first determines the location ofsystem 10 based on GPS coordinates and/or other data received inblock 105 ofFIG. 5A . This is shown inFIG. 12A , a plan view of a portion of the tactical region for which information is stored indatabase 401. Various friendly soldiers, enemy positions and other tactical objects are shown using the same types of icons shown inFIGS. 4 and 10 . Each of the tactical objects has one or more entries indatabase 401. The position of system 10 (and thus of goggles 11) is also indicated. In block 503 (FIG. 11 ),computer 30 then uses magnetometer output fromsensor 13, or (in case of magnetic interference) the yaw Euler angle calculated from the IMU movements subsequent to the last acceptable magnetic fix, to determine the direction of the goggles FOV (i.e., the direction in which the user is looking). In particular, the value of MagNorth (seeFIG. 3 ) is corrected (using geographically-specific magnetic deviation and true-north correction data in database 401) so as to determine the direction in whichgoggles 11 are pointed. This is shown inFIG. 12B . Also shown inFIG. 12B is the goggles FOV and HUD FOV. Flow then proceeds to block 505 ofFIG. 11 , where pitch axis data fromgoggles sensor 13 is used to determine the degree to which the user is looking up or down. This is shown inFIG. 12C . -
Computer 30 then proceeds to block 507. After determining the direction in whichgoggles 11 are pointed,computer 30 calculates a spatial volume corresponding to the boundary ofHUD 73 inuser display 70. This HUD volume is a pyramid centered on the goggles FOV centerline and extending outward. As seen inFIGS. 2, 4 and 12B , the HUD FOV (e.g., 32°) is narrower than and centered within the goggles FOV (e.g.,) 40°. The HUD pyramid is bounded by a plane located at a range (d) fromgoggles 11. This is shown inFIG. 12D .Computer 30searches database 401 for all objects within the just-calculated HUD pyramid. The range (d) is in some embodiments selected by the user. For example, a user might be more concerned about objects that are relatively close (e.g., within 500 meters) than about those that are further away. Moreover, providing icons or other display indicia for distant objects may unnecessarily clutteruser display 70. In certain embodiments, range (d) may be changed whilesystem 10 is operating and the user display can be de-cluttered in other ways. As but one example, route information, control measures, etc. may be removed at the push of a button during a firefight so as to minimize potential distracters to the soldier, while friendly icons would be retained to help limit fratricide. Removed information could be restored by the operator when again needed. - In
block 511 ofFIG. 11 ,computer 30 then generates icons or other indicia inuser display 70 corresponding to the objects, identified in the database search ofblock 509, within the HUD pyramid. In some embodiments, the size of the icons or other indicia is logarithmically scaled so that icons or other indicia for more distant objects are smaller. -
Computer 30 then returns to block 127 (FIG. 5B ). If more video frames fromgoggles 11 andscope 17 are received (i.e.,goggles 11 andscope 17 remain ON), thencomputer 30 returns to block 105 ofFIG. 5A . Otherwise, operation ofsystem 10 terminates. - Other embodiments include numerous variations and/or modifications of the above-described devices and operations. As but one example, the weapon view can be a magnified version of the scope view (either at all times or selectively at the request of the user). In such embodiments, the image-based comparison techniques described above can first be performed using an unmagnified scope image. Once the proper location and orientation for the scope image are deduced, the scope image can be magnified and cropped prior to overlay as the weapon view. Other types of information (e.g., a battery life indicator, navigational headings, etc.) can also be selectively shown in a user display. Indeed, a user can be provided with a great deal of flexibility in selecting the types of information to be shown on the display. Such selection can take place during set-up or while the system is in use. For example, a soldier on patrol may choose to see display indicators for routes, rally points, indirect fire targets, preplanned control measures, etc. When the soldier is attacked or is about to enter a firelight, he or she might then place the weapon in a close-combat mode (e.g., by pressing a mode control button located on a weapon). In such a close-combat mode, the user could avoid information overload by suppressing all information except icons for friendly forces. In some cases, the soldier might even choose to suppress “friendly” icons and opt for an audible warning if the weapon is pointed at friendly personnel. As yet another variation, a soldier might configure his system to display icons for friendly forces within a first range (e.g., 300 meters) and for enemy positions within a second range (e.g., 1000 meters), to display graphics for patrol routes within a third range (e.g., 500 meters), to plot control measures with routes, etc.
- As discussed above, location information can be provided via GPS transmissions. Location data can also be augmented by local information such as timing and ranges from nearby soldiers also equipped with a system such as described herein. Other types of navigational data (e.g., WAAS, or wide area augmentation system) can also be used. Barometric pressure can be used to provide a relative elevation error and used as a check against a GPS-based elevation, as well as during system initialization. A terrain database (e.g., within a database such as database 401) can include values for magnetic deviation, gravitational acceleration, dip angle, elevation (e.g., DTED, or digital terrain elevation data) and/or other quantities as a function of longitude and latitude.
- Data from various sources can be updated at various rates (e.g., video frames at 30 Hz, IMU data at 100 Hz, GPS or other navigational updates at 1 Hz, range data from other soldiers at 8 Hz). IMU accelerometers can have bandwidths greater than 1000 Hz, resolution of 1.25 milligrams, and 2% sensitivity. A magnetometer can provide outputs at 200 Hz and have a sensitivity of 1 mGauss (1 part in 300-600 Earth field). Rate sensors can have a 2% error and 100 Hz bandwidth, and barometric pressure data provided at 100 Hz and within 1-2 meters of resolution. Range data from other soldiers may be within 1.3 meters of error. Of course, all of these values are only examples, and the invention is not limited to systems having specific values for any of these quantities.
- Other embodiments also include numerous variations on the above-described methods for using a comparison of a scope image and a goggles image to check an IMU-based location and rotation for that scope image within that goggles image. For example, the operations described in connection with
FIG. 7 initially assume that the center for a scope image within a goggles image calculated from IMU data is correct. If the image comparison suggests that the IMU-based calculation is not correct, a search of other possible center locations moves outward from the IMU-based center. Assuming an IMU drift rate of approximately 1°/hour, a procedure such as that ofFIG. 7 works well (and minimizes computational load on a processor) if the scope and goggles point in the same direction at least once every 10 to 15 minutes. However, the procedure could be modified for IMUs having different drift rates or if it is expected that the scope and goggles will not be looking in the same direction for an extended period. As but one example, a system could be configured so that a user can manually initiate a search (e.g., beginning at the goggles image center) across the entire goggles image for a scope image location. This could be used for initially calibrating the IMUs, or as a re-calibration if the user has not pointed his weapon forward for an extended period (e.g., if the weapon was slung over a shoulder). The search patterns shown inFIGS. 8J and 8K are only examples. Numerous other search patterns could be implemented. - In some embodiments, an alternate image comparison algorithm is used instead of (or in addition to) the operations described in connection with
FIGS. 7-8K .FIGS. 13A and 13B show a goggles images and scope image, respectively, that will be used to describe this alternate algorithm. Unlike the previous discussion, which assumed the fields of view for bothgoggles array 56 andscope 17 were conical, the images ofFIGS. 13A and 13B instead assume thatarray 56 and a similar array inside ofscope 17 are rectangular. In particular,FIG. 13A is agoggles image 601 generated when array 56 (FIG. 3 ) is a 320×240 pixel microbolometer array (8-12 micron range) having a 14.4° horizontal FOV, a 10.8° vertical FOV, and an 18° diagonal FOV. The data fromarray 56 is converted into a 640×480 pixel NTSC format image for transmission to controlunit 16 and digitized for processing.FIG. 13B is ascope image 603 generated when an array inside scope 17 (FIG. 3 ) is also 320×240 pixel microbolometer array (8-12 micron range) having a 14.4° horizontal FOV, a 10.8° vertical FOV, and an 18° diagonal FOV. The data from the scope array is also converted into a 640×480 pixel NTSC format image for transmission to controlunit 16 and digitized for processing. In the present example, the horizontal, vertical and diagonal goggles fields of view are the same as the horizontal, vertical and diagonal scope fields of view. However, the below-described alternative image comparison algorithm can also be used in connection with a goggles and/or scope image created from different arrays having different fields of view (including the conical fields of view described previously), as well as in circumstances when the fields of view for the goggles and the scope are different in horizontal, vertical and/or diagonal planes. Adaptation of the below described alternative algorithm to such other array configurations is within the routine ability of persons of ordinary skill in the art once such persons are provided with the information contained herein. - In both
images images images array 56 outputting a gray-scale value of G corresponds to a 2×2 pixel block inimage 601, with all four pixels in that 2×2 block also having a gray-scale value of G. The significance of this will be apparent below. - Returning to
FIG. 13B , acenter 604 is also indicated.Surrounding center 604 is a circle 605 having a diameter corresponding to 4° of the FOV. The region withincircle 604 will be overlaid ongoggles image 601 as a weapon view (similar toweapon view 74 ofFIG. 4 ). -
FIG. 14 is a flow chart for an alternate image comparison algorithm according to at least some embodiments. Various blocks in theFIG. 14 flow chart will be further explained using diagrams and plots inFIGS. 15-22 . In some implementations, the alternative image comparison algorithm ofFIG. 14 assumes that current IMU data is available fromsensors 13 and 18 (FIG. 3 ), that IMU drift since the last calibration (or image registration) has not been significant, and that IMU data is thus usable as a starting point in the algorithm (as described below). In at least some embodiments, random noise in IMU output rarely exceeds 1 degree in yaw or ½ degree in pitch or roll. TheFIG. 14 algorithm also assumes that center area 605 is wholly contained within goggles image 601 (i.e., that the portion of the external environment corresponding to center area 605 is also represented by a region wholly contained in goggles image 601) at time intervals such that the bias drift will be less than 2 degrees. - Similar to the algorithm previously described in connection with
FIG. 7 , the image comparison algorithm ofFIG. 14 is entered fromblock 115 ofFIG. 5B . As with the algorithm ofFIG. 7 , the algorithm ofFIG. 14 is also performed by computer 30 (FIG. 2 ). In the first block ofFIG. 14 ,computer 30 resets a counter (N). As described in more detail below, the counter N is used to prevent excessive looping through the algorithm. Fromblock 701,computer 30 proceeds to block 703 and locates a region of high contrast, withinscope image 603, that is relativelynear center 604. This is shown in more detail byFIG. 15 . -
FIG. 15 is a partially schematic diagram in which the various image elements (e.g., trees, foliage, etc.) have been omitted for simplicity, but in which the boundary ofscope image 603 andcenter 604 are shown. Inblock 703,computer 30searches image 603 for asquare region 610 in which the pixels inside ofregion 610 have a high degree of contrast. In particular,computer 30 searches for thesquare region 610 in which the standard deviation of pixel gray-scale values is maximized. In the present example, and as indicated above,image 603 comprises numerous 2×2 pixel blocks in which all pixels in a 2×2 block have the same gray-scale value. Accordingly, when calculating a standard deviation for a region ofimage 603, only every other pixel in the horizontal and vertical directions need be used. In at least some embodiments,region 610 corresponds to a 3° horizontal FOV by 3° vertical FOV portion ofimage 603. In the present example of example ofimage 603,region 610 is approximately 133 pixels by 133 pixels. In some such embodiments, thecenter 606 ofregion 610 should also be within 2° ofcenter 604 in the horizontal plane and within 1° ofcenter 604 in the vertical plane. In the present example,computer 30 thus searches for a 133×133 pixel region contained within a 311×222 pixel region ofimage 603 that is centered aboutcenter 604. - Returning to
FIG. 14 ,computer 30 proceeds fromblock 703 to block 705 and determines if theregion 610 identified inblock 703 is of sufficiently high contrast. If, for example, the maximum standard deviation for gray-scale values within any 133×133 pixel block is not sufficiently high, there may be insufficient contrast to continue with an image comparison. A sufficiently high standard deviation value could be determined for a given combination of goggles and scope imaging array types based on a series of test images. If the standard deviation for gray scale values in theregion 610 identified inblock 703 is not sufficiently high,computer 30 returns to block 119 ofFIG. 5B via the “yes” branch. If the standard deviation of gray-scale values for theregion 610 identified inblock 703 is sufficiently high, computer continues with the image comparison and proceeds on the “no” branch to block 707. - In
block 707,computer 30 locates the point ingoggles image 601 that IMU data (fromsensors 13 and 18) indicates is a representation of the same part of external environment represented by point 606 (FIG. 15 ). As indicated above,point 606 is the center of theregion 610 found bycomputer 30 inblock 703. The operation ofblock 707 is shown in more detail inFIG. 16 .FIG. 16 is a partially schematic diagram in which the various image elements (e.g., trees, foliage, etc.) have been omitted for simplicity, but in which the boundary ofgoggles image 601 is shown.Point 607 is the point which IMU data indicates corresponds to point 606 in the scope image. Also shown inFIG. 601 is a 1° by 1° (e.g., 44 pixels by 44 pixels)region 613 centered onpoint 607. The significance ofregion 613 is explained below. - Returning to
FIG. 14 ,computer 30 proceeds fromblock 707 to block 709. Inblock 709,computer 30 obtains pixel gray-scale values from 128 locations on a circle aroundcenter 606 of region 610 (FIG. 15 ). This is shown in more detail inFIG. 17A , a schematic diagram ofpixels surrounding center 606 in a portion ofscope image 603. Each of the squares inFIG. 17A represents one pixel ofscope image 603.Surrounding center 606 are 128 locations (shown as diamonds) that are equally spaced on a circle having a diameter of approximately 1°. For each of the diamonds inFIG. 17A ,computer 30 obtains the gray-scale value for the pixel in which the diamond's center is located. Unlike the calculation discussed in connection with block 703 (FIG. 14 ), however, values are not limited to those for every other pixel. In other words, gray-scale values may be obtained for multiple pixels within a 2×2 pixel block corresponding to a single pixel of the microbolometer array inscope 17. Indeed, there might be instances in which two of the points on the circle are located within the same pixel ofimage 603. - Returning to
FIG. 14 ,computer 30 proceeds fromblock 709 to block 711. Inblock 711,computer 30 performs an operation similar to that ofblock 709, but instead usinggoggles image 601. Specifically,computer 30 obtains pixel gray-scale values from 128 locations on a circle aroundpoint 607 in goggles image 601 (FIG. 16 ). This is shown in more detail inFIG. 17B , a schematic diagram ofpixels surrounding point 607 in a portion ofgoggles image 601. Similar to the calculation ofblock 709, gray-scale values may be obtained for multiple pixels within a 2×2 pixel block corresponding to a single pixel of themicrobolometer array 56 ingoggles 11. - In the present example,
goggles image 601 andscope image 603 have the same horizontal, vertical and diagonal fields of view and are of the same pixel dimensions. This need not be the case, however. If one ofimages blocks FIG. 17C . InFIG. 17C ,point 607′ is a point (determined using IMU data) corresponding to the center of a high contrast region found in block 703 (FIG. 14 ). Each square inFIG. 17C represents one pixel of a goggles image having the larger FOV values. - Returning to
FIG. 14 ,computer 30 then proceeds to block 713. Inblock 713,computer 30 performs various calculations using the gray-scale values obtained inblocks FIG. 18 is a plot showing an example of raw data that might be obtained inblocks FIG. 18 and of subsequentFIGS. 19 and 20 are provided for purposes of explanation, and would not necessarily be generated bycomputer 30. InFIG. 18 , the solid line plot represents the 128 gray-scale values obtained in one ofblocks blocks FIG. 14 ),computer 30 first performs a discrete Fast Fourier Transform (FFT) on each data set (i.e., a discrete FFT on the 128 scope image values and a discrete FFT on the 128 goggles image values).Computer 30 then multiplies the complex conjugate of one of those discrete FFTs by the other of the discrete FFTs.Computer 30 next obtains an inverse FFT of the product.Computer 30 then subtracts the DC component of the inverse FFT and obtains data that allows peak-to-sidelobe ratio (PSR) comparisons. -
Computer 30 then evaluates the result ofblock 713 inblock 715. If the center points of the scope image region and of the goggles image region to which it was compared (e.g., points 606 and 607 ofFIGS. 15 through 17B ) represent the same part of the external environment, the PSR will be relatively large (e.g., 3 or higher).FIG. 19 is a plot of the results of a calculation in block 713 (FIG. 14 ) when the center points do coincide. The sidelobes inFIG. 19 can be seen as three regions (on either side of the peak) that are higher than their surroundings. The location of the peak can also be used to determine the rotation of one image relative to the other. In the example ofFIG. 19 , the peak is displaced 8 pixel increments from the left of the center of the plot. This corresponds to a rotation of (8/128)*360°, or 22.5°. The direction of rotation is determined based on whether the displacement is to the left or to the right of the center.FIG. 20 is an example of a plot resulting from the calculations ofblock 713 using random data. - If the result of the evaluation in
FIG. 14 block 715 is favorable (e.g., if the PSR 3),computer 30 proceeds to block 717 via the “Fay.” branch and stores the PSR, rotation and the pixel coordinates of the point in thegoggles image 601 that was used in block 711 (e.g., point 607) as the circle center.Computer 30 then continues to block 719. If the result inblock 715 is not favorable (e.g., PSR<3),computer 30 proceeds on the “Unfav.” branch to block 719 without storing a PSR, center location or rotation. - In
block 719,computer 30 determines if there are additional points inregion 613 of goggles image 601 (FIG. 16 ) that should be checked as possibly coinciding withpoint 606 ofscope image 603.FIGS. 21 and 22 show a pattern by which additional points inregion 613 are selected in at least some embodiments.FIG. 21 schematically showsregion 613 and location 607 (found inFIG. 14 block 707), with each of the squares representing a single pixel.FIG. 22 is an enlargement of the portion ofregion 613 indicated inFIG. 21 . In the first pass throughblock 721 after checkingpoint 607,computer 30 selects the pixel at position {circle around (2)}.Computer 30 then returns to block 711 and repeats block 711 using pixel position {circle around (2)} instead ofpoint 607, and then repeatsblocks 713 through 719 (possibly omitting block 715). In the next pass throughblock 721, pixel position {circle around (3)} is selected. This continues until some stop condition is reached. In some embodiments, a stop condition is reached when the pattern shown generally inFIG. 22 reaches the sides ofregion 613 along the horizontal plane. The pattern ofFIG. 22 weights the search based on a higher probability that any random noise or bias drift will be greater along the yaw axis than along the pitch or roll axes. In other embodiments, the stop condition is reached after every other point in both the horizontal and vertical directions has been checked. - Once a stop condition is reached and there are no more points to select within
region 613,computer 30 proceeds to block 723 and determines if any of the data sets (with each set containing a PSR, rotation and center point location) stored inblock 717 is unambiguously the best. In at least some embodiments, the “best” data set is one in which the PSR is a predetermined percentage above the other stored PSRs. The percentage by which a “best” point should exceed others depends on the characteristics of the imaging technology involved, but one example value is an excess of 10%. - If an unambiguously best data set is identified in
block 723,computer 30 proceeds to block 725. In block 725 (which is substantiallysimilar block 219 ofFIG. 7 ),computer 30 updates the IMU-based position forweapon view 74 based on the position and rotation of scope image region 605 withingoggles image 601 resulting from the unambiguously best data set.Computer 30 also uses that data set to adjust the manner in which subsequent IMU-based position and image rotation calculations are made. This adjustment is performed as previously explained in connection withFIGS. 9A-9C . Fromblock 725,computer 30 proceeds to block 119 ofFIG. 5B . - If an unambiguously best stored data set is not found in
block 723,computer 30 proceeds to block 727. If counter N has reached a maximum vale (NMAX),computer 30 returns (via the “yes” branch) to block 119 ofFIG. 5B . Appropriate values of NMAX would depend of various design preferences (e.g., processor speed, acceptable latency, etc.). If counter N has not reached NMAX,computer 30 proceeds on the “no” branch to block 729 and increments counter N, and then proceeds to block 731. In block 731 a new high contrast region is selected in the scope image. In some embodiments, the selection ofblock 731 identifies the 3°×3° pixel block having the next highest gray-scale value standard deviation, and which does not overlap a previously selected 3°×3° pixel block inimage 603 by more than a predetermined amount.Computer 30 then returns to block 707. - As with the algorithm described in connection with
FIG. 7 , numerous variations on the algorithm ofFIG. 14 are within the scope of the invention. The search pattern ofblock 721, the sizes of image regions evaluated, and various other parameters can be varied. As but one example, the above-described algorithm employscenter 604 as one reference point andpoint 607 as a corresponding reference point. In other embodiments, a point other than the center of the scope image can be used as the initial reference point. As another example, individual pixel gray-scale values are the image element data points used in the FFT analyses. Other image element data points could be used (e.g., an average of gray-scale values for several pixels clustered around a point, wavelength (color) information if sensors provide color information). As yet another example, the diamonds inFIGS. 17A through 17C have a particular dimensioned arrangement relative to the reference point (center 604) and the corresponding reference point (point 607). In other embodiments, different arrangements (e.g., oval, square, concentric rings, etc.) and/or different dimensions (e.g., closer to or farther from the reference point and corresponding reference point) can be used. In still some other embodiments, data sets are stored inblock 717 for only the five highest (or some other predetermined number of highest) PSR values. In some embodiments, the scope image has a small number of possible FOVs (e.g., 9° diagonal or 18° diagonal). - In some embodiments using the algorithm of
FIG. 14 , and in circumstances where an image-based placement of the weapon view cannot be performed for a significant period (e.g., if there is a long period in which there are no sufficiently high contrast images), drift may become obvious to the user. In such a case, the user can manually recalibrate the system. - Although embodiments of the invention have been described by example of a system intended for use by a soldier or other combat personnel, the invention is not limited in this regard. In other embodiments, a heads up display need not be associated with a pair of goggles. For example, the heads up display could appear before a windshield in a vehicle. A weapon mounted on the vehicle includes a video gun sight producing images processed and projected onto the heads up display. In this embodiment, an orientation sensor may be placed to sense the orientation of the vehicle rather than a pair of goggles worn by the observer. This embodiment may be particularly useful for remotely controlled weapon systems, for example a robot carrying a weapon.
- It should also be noted that the techniques described herein are not limited to weapon targeting or other combat uses. Other embodiments could be used in a myriad of settings, including law enforcement, medicine, astronomy, etc.
- Although examples of carrying out the invention have been described, those skilled in the art will appreciate that there are numerous other variations, combinations and permutations of the above described devices and techniques that fall within the spirit and scope of the invention as set forth in the appended claims. As used in the claims, “controller” generically refers to any of one or more FPGAs, microprocessors, ASICs, other types of computational devices, or combinations thereof. It is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. In the claims, various portions are prefaced with letter or number references for convenience. However, use of such references does not imply a temporal relationship not otherwise required by the language of the claims.
Claims (1)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/165,210 US20190049734A1 (en) | 2007-02-28 | 2018-10-19 | System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display |
US17/723,730 US11965714B2 (en) | 2007-02-28 | 2022-04-19 | System and method for video image registration and/or providing supplemental data in a heads up display |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/680,207 US9229230B2 (en) | 2007-02-28 | 2007-02-28 | System and method for video image registration and/or providing supplemental data in a heads up display |
US14/950,643 US9618752B2 (en) | 2007-02-28 | 2015-11-24 | System and method for video image registration and/or providing supplemental data in a heads up display |
US15/483,086 US10139629B2 (en) | 2007-02-28 | 2017-04-10 | System and method for video image registration and/or providing supplemental data in a heads up display |
US16/165,210 US20190049734A1 (en) | 2007-02-28 | 2018-10-19 | System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/483,086 Continuation US10139629B2 (en) | 2007-02-28 | 2017-04-10 | System and method for video image registration and/or providing supplemental data in a heads up display |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/723,730 Continuation US11965714B2 (en) | 2007-02-28 | 2022-04-19 | System and method for video image registration and/or providing supplemental data in a heads up display |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190049734A1 true US20190049734A1 (en) | 2019-02-14 |
Family
ID=39715307
Family Applications (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/680,207 Active 2034-11-06 US9229230B2 (en) | 2007-02-28 | 2007-02-28 | System and method for video image registration and/or providing supplemental data in a heads up display |
US14/950,643 Active US9618752B2 (en) | 2007-02-28 | 2015-11-24 | System and method for video image registration and/or providing supplemental data in a heads up display |
US15/483,086 Active US10139629B2 (en) | 2007-02-28 | 2017-04-10 | System and method for video image registration and/or providing supplemental data in a heads up display |
US16/165,210 Abandoned US20190049734A1 (en) | 2007-02-28 | 2018-10-19 | System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display |
US17/723,730 Active US11965714B2 (en) | 2007-02-28 | 2022-04-19 | System and method for video image registration and/or providing supplemental data in a heads up display |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/680,207 Active 2034-11-06 US9229230B2 (en) | 2007-02-28 | 2007-02-28 | System and method for video image registration and/or providing supplemental data in a heads up display |
US14/950,643 Active US9618752B2 (en) | 2007-02-28 | 2015-11-24 | System and method for video image registration and/or providing supplemental data in a heads up display |
US15/483,086 Active US10139629B2 (en) | 2007-02-28 | 2017-04-10 | System and method for video image registration and/or providing supplemental data in a heads up display |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/723,730 Active US11965714B2 (en) | 2007-02-28 | 2022-04-19 | System and method for video image registration and/or providing supplemental data in a heads up display |
Country Status (1)
Country | Link |
---|---|
US (5) | US9229230B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11460270B1 (en) * | 2021-11-23 | 2022-10-04 | George Carter | System and method utilizing a smart camera to locate enemy and friendly forces |
WO2023057654A1 (en) * | 2021-10-08 | 2023-04-13 | Uab "Yukon Advanced Optics Worldwide" | Observation device |
Families Citing this family (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9323055B2 (en) * | 2006-05-26 | 2016-04-26 | Exelis, Inc. | System and method to display maintenance and operational instructions of an apparatus using augmented reality |
JP5119636B2 (en) * | 2006-09-27 | 2013-01-16 | ソニー株式会社 | Display device and display method |
US9229230B2 (en) | 2007-02-28 | 2016-01-05 | Science Applications International Corporation | System and method for video image registration and/or providing supplemental data in a heads up display |
US20080218331A1 (en) | 2007-03-08 | 2008-09-11 | Itt Manufacturing Enterprises, Inc. | Augmented reality-based system and method to show the location of personnel and sensors inside occluded structures and provide increased situation awareness |
US20090214436A1 (en) | 2008-02-18 | 2009-08-27 | Washington University | Dichromic fluorescent compounds |
JP2010033367A (en) * | 2008-07-29 | 2010-02-12 | Canon Inc | Information processor and information processing method |
WO2010071531A1 (en) * | 2008-12-19 | 2010-06-24 | Saab Ab | System and method for mixing a scene with a virtual scenario |
US8336777B1 (en) * | 2008-12-22 | 2012-12-25 | Pantuso Francis P | Covert aiming and imaging devices |
US20100196859A1 (en) * | 2009-02-01 | 2010-08-05 | John David Saugen | Combat Information System |
TWI400467B (en) * | 2009-05-11 | 2013-07-01 | Acer Inc | Electronic device with object guidance function and its object guidance method |
EP2502410B1 (en) * | 2009-11-19 | 2019-05-01 | eSight Corporation | A method for augmenting sight |
US10470010B2 (en) | 2010-01-15 | 2019-11-05 | Colt Canada Ip Holding Partnership | Networked battle system or firearm |
US10477619B2 (en) | 2010-01-15 | 2019-11-12 | Colt Canada Ip Holding Partnership | Networked battle system or firearm |
US10337834B2 (en) | 2010-01-15 | 2019-07-02 | Colt Canada Ip Holding Partnership | Networked battle system or firearm |
US10477618B2 (en) | 2010-01-15 | 2019-11-12 | Colt Canada Ip Holding Partnership | Networked battle system or firearm |
US20110250962A1 (en) * | 2010-04-09 | 2011-10-13 | Feiner Steven K | System and method for a 3d computer game with true vector of gravity |
US9615147B2 (en) * | 2010-05-17 | 2017-04-04 | Flir Systems, Inc. | Multisensory meter system |
US20110301813A1 (en) * | 2010-06-07 | 2011-12-08 | Denso International America, Inc. | Customizable virtual lane mark display |
US8854220B1 (en) * | 2010-08-30 | 2014-10-07 | Exelis, Inc. | Indicating desiccant in night vision goggles |
US20120157204A1 (en) * | 2010-12-20 | 2012-06-21 | Lai Games Australia Pty Ltd. | User-controlled projector-based games |
GB2499776A (en) * | 2011-11-17 | 2013-09-04 | Thermoteknix Systems Ltd | Projecting secondary information into an optical system |
ITFI20110266A1 (en) * | 2011-12-09 | 2013-06-10 | Selex Galileo Spa | "MIRA SYSTEM" |
WO2013112554A1 (en) | 2012-01-23 | 2013-08-01 | Washington University | Goggle imaging systems and methods |
US9829715B2 (en) * | 2012-01-23 | 2017-11-28 | Nvidia Corporation | Eyewear device for transmitting signal and communication method thereof |
US9625720B2 (en) * | 2012-01-24 | 2017-04-18 | Accipiter Radar Technologies Inc. | Personal electronic target vision system, device and method |
US9132342B2 (en) * | 2012-10-31 | 2015-09-15 | Sulon Technologies Inc. | Dynamic environment and location based augmented reality (AR) systems |
US20140125870A1 (en) * | 2012-11-05 | 2014-05-08 | Exelis Inc. | Image Display Utilizing Programmable and Multipurpose Processors |
WO2014085605A2 (en) * | 2012-11-28 | 2014-06-05 | Intrepid Networks, Llc | Integrated systems and methods providing situational awareness of operations in an organization |
JP6123365B2 (en) * | 2013-03-11 | 2017-05-10 | セイコーエプソン株式会社 | Image display system and head-mounted display device |
KR102098277B1 (en) * | 2013-06-11 | 2020-04-07 | 삼성전자주식회사 | Visibility improvement method based on eye tracking, machine-readable storage medium and electronic device |
US10228242B2 (en) | 2013-07-12 | 2019-03-12 | Magic Leap, Inc. | Method and system for determining user input based on gesture |
CN115031581A (en) * | 2013-10-31 | 2022-09-09 | 威罗门飞行公司 | Interactive weapon aiming system displaying remote sensing image of target area |
US10104241B2 (en) * | 2013-11-14 | 2018-10-16 | Drs Network & Imaging Systems, Llc | Method for integrated optical systems |
US10338776B2 (en) | 2013-12-06 | 2019-07-02 | Telefonaktiebolaget Lm Ericsson (Publ) | Optical head mounted display, television portal module and methods for controlling graphical user interface |
WO2015199780A2 (en) * | 2014-04-01 | 2015-12-30 | Baker Joe D | Mobile ballistics processing and targeting display system |
EP2950035A1 (en) * | 2014-05-28 | 2015-12-02 | Patents Factory Ltd. Sp. z o.o. | Method and system for a tactical visualization |
US10330440B2 (en) * | 2014-11-26 | 2019-06-25 | Philip Lyren | Target analysis and recommendation |
US9852547B2 (en) * | 2015-03-23 | 2017-12-26 | International Business Machines Corporation | Path visualization for augmented reality display device based on received data and probabilistic analysis |
US10806804B2 (en) | 2015-05-06 | 2020-10-20 | Washington University | Compounds having RD targeting motifs and methods of use thereof |
AU2016268788A1 (en) | 2015-05-26 | 2017-12-07 | Colt Canada Ip Holding Partnership | A networked battle system with heads up display |
JP6555804B2 (en) * | 2015-06-22 | 2019-08-07 | 株式会社日立国際電気 | Shooting training system |
US10378858B2 (en) * | 2016-01-21 | 2019-08-13 | Kai W. Gull | Systems, devices and methods for visualizing a target |
CA3013763C (en) * | 2016-02-03 | 2021-02-16 | Vk Integrated Systems | Firearm electronic system |
DE102017101118A1 (en) * | 2017-01-20 | 2018-07-26 | Steiner-Optik Gmbh | Communication system for transmitting captured object information between at least two communication partners |
CN108346398A (en) * | 2017-01-24 | 2018-07-31 | 上海珏芯光电科技有限公司 | Its driving method of display driving board device |
US10976136B2 (en) * | 2017-09-06 | 2021-04-13 | Mehmet Ali GUZELDERE | Wireless vision equipment for weapons |
SG11202002344XA (en) * | 2017-09-15 | 2020-04-29 | Tactacam LLC | Weapon sighted camera system |
CN107886487B (en) * | 2017-12-03 | 2021-04-27 | 南京理工大学 | FPGA-based real-time implementation method of low-light-level video interframe accumulation noise reduction algorithm |
DE102018112779A1 (en) * | 2018-04-16 | 2019-10-17 | Indivi Optics Gmbh | Night vision sighting system of a handgun, especially a hunting rifle |
US11354787B2 (en) * | 2018-11-05 | 2022-06-07 | Ultrahaptics IP Two Limited | Method and apparatus for correcting geometric and optical aberrations in augmented reality |
US20200296462A1 (en) | 2019-03-11 | 2020-09-17 | Wci One, Llc | Media content presentation |
US20200296316A1 (en) | 2019-03-11 | 2020-09-17 | Quibi Holdings, LLC | Media content presentation |
IT201900005110A1 (en) * | 2019-04-04 | 2020-10-04 | Eye Tech Lab S R L | MONOCULAR DIGITAL MAGNIFIER AND RELATED GLASSES. |
US10997418B2 (en) * | 2019-09-09 | 2021-05-04 | Ar, Llc | Augmented, virtual and mixed-reality content selection and display |
US11961294B2 (en) | 2019-09-09 | 2024-04-16 | Techinvest Company Limited | Augmented, virtual and mixed-reality content selection and display |
WO2021119423A1 (en) | 2019-12-13 | 2021-06-17 | Washington University | Near infrared fluorescent dyes, formulations and related methods |
US11409360B1 (en) * | 2020-01-28 | 2022-08-09 | Meta Platforms Technologies, Llc | Biologically-constrained drift correction of an inertial measurement unit |
US11410270B2 (en) | 2020-06-03 | 2022-08-09 | University Of Central Florida Research Foundation, Inc. | Intelligent object magnification for augmented reality displays |
US11651502B2 (en) * | 2020-07-22 | 2023-05-16 | Microsoft Technology Licensing, Llc | Systems and methods for updating continuous image alignment of separate cameras |
US12026235B2 (en) * | 2021-06-04 | 2024-07-02 | Morphix, Inc. | Target classification system |
US12014514B2 (en) | 2021-06-04 | 2024-06-18 | Morphix, Inc. | Target classification system |
US11644277B2 (en) * | 2021-07-01 | 2023-05-09 | Raytheon Canada Limited | Digital booster for sights |
US20230333450A1 (en) * | 2022-04-18 | 2023-10-19 | Thales Defense & Security, Inc. | Methods and systems for automatic parallax compensation for helmet mounted imaging sensors used for augmented or extended reality displays |
Family Cites Families (99)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4091412A (en) | 1967-12-01 | 1978-05-23 | The United States Of America As Represented By The Secretary Of The Army | Target designation system |
US3674925A (en) | 1970-12-03 | 1972-07-04 | Us Navy | Cable-less television system |
DE2231666C3 (en) | 1972-06-28 | 1975-11-20 | Societe Nationale Industrielle Aerospatiale, Paris | Device for training on optical aiming devices |
US3798795A (en) | 1972-07-03 | 1974-03-26 | Rmc Res Corp | Weapon aim evaluation system |
US3843969A (en) | 1973-11-05 | 1974-10-29 | Us Air Force | Personnel armor suspension system |
DE2436169C3 (en) | 1974-07-26 | 1980-06-12 | Precitronic Gesellschaft Fuer Feinmechanik Und Electronic Mbh, 2000 Hamburg | Shot simulation device |
US4020324A (en) | 1974-08-26 | 1977-04-26 | Lear Siegler, Inc. | Weapon delivery system |
US3964178A (en) | 1975-07-03 | 1976-06-22 | The United States Of America As Represented By The Secretary Of The Navy | Universal infantry weapons trainer |
JPS52105768A (en) | 1976-03-01 | 1977-09-05 | Ise Electronics Corp | Cathode ray display panel |
US4028725A (en) | 1976-04-21 | 1977-06-07 | Grumman Aerospace Corporation | High-resolution vision system |
DE2742899C3 (en) | 1977-09-23 | 1980-06-12 | Siemens Ag, 1000 Berlin Und 8000 Muenchen | Arrangement for the optical transmission of messages |
SE428329B (en) | 1979-03-28 | 1983-06-20 | John Lorenz Weibull | DIRECTIONAL AND PUSHING OVENING DEVICE |
US4280125A (en) | 1979-07-27 | 1981-07-21 | Xerox Corporation | Thin vacuum panel display device |
US4398799A (en) * | 1980-03-04 | 1983-08-16 | Pilkington P.E. Limited | Head-up displays |
US4290757A (en) | 1980-06-09 | 1981-09-22 | The United States Of America As Represented By The Secretary Of The Navy | Burst on target simulation device for training with rockets |
DE3024247A1 (en) | 1980-06-27 | 1982-01-21 | Fried. Krupp Gmbh, 4300 Essen | DEVICE FOR LIMITING THE SHOOTING AREA OF A PISTON WEAPON, IN PARTICULAR A TANK CANNON, WHILE TRAINING SHOOTING |
DE3133889C2 (en) | 1981-08-27 | 1984-04-26 | Honeywell Gmbh, 6050 Offenbach | Process for visual simulation |
US4516157A (en) | 1982-11-23 | 1985-05-07 | Campbell Malcolm G | Portable electronic camera |
US4605959A (en) | 1984-08-23 | 1986-08-12 | Westinghouse Electric Corp. | Portable communications terminal |
US4703879A (en) | 1985-12-12 | 1987-11-03 | Varo, Inc. | Night vision goggle headgear |
GB2189365A (en) | 1986-03-20 | 1987-10-21 | Rank Xerox Ltd | Imaging apparatus |
US4786966A (en) | 1986-07-10 | 1988-11-22 | Varo, Inc. | Head mounted video display and remote camera system |
US5005213A (en) | 1986-07-10 | 1991-04-02 | Varo, Inc. | Head mounted video display and remote camera system |
US5200827A (en) | 1986-07-10 | 1993-04-06 | Varo, Inc. | Head mounted video display and remote camera system |
US4884137A (en) | 1986-07-10 | 1989-11-28 | Varo, Inc. | Head mounted video display and remote camera system |
US4970589A (en) | 1986-07-10 | 1990-11-13 | Varo, Inc. | Head mounted video display and remote camera system |
WO1988002841A1 (en) | 1986-10-17 | 1988-04-21 | Hughes Aircraft Company | Weapon automatic alerting and cueing system |
US4963006A (en) | 1986-12-26 | 1990-10-16 | Nikon Corporation | Zoom lens with inner or rear focusing system |
US4923401A (en) | 1988-11-25 | 1990-05-08 | The United States Of America As Represented By The Secretary Of The Navy | Long range light pen |
GB8908375D0 (en) | 1989-04-13 | 1989-06-01 | Measurement Devices Ltd | Hand-held laser rangefinder |
GB8921268D0 (en) | 1989-09-20 | 1990-04-25 | Marconi Gec Ltd | Orientation monitoring apparatus |
US5194006A (en) | 1991-05-15 | 1993-03-16 | Zaenglein Jr William | Shooting simulating process and training device |
FR2683918B1 (en) | 1991-11-19 | 1994-09-09 | Thomson Csf | MATERIAL CONSTITUTING A RIFLE SCOPE AND WEAPON USING THE SAME. |
US5272514A (en) | 1991-12-06 | 1993-12-21 | Litton Systems, Inc. | Modular day/night weapon aiming system |
KR960010914B1 (en) | 1992-03-24 | 1996-08-13 | 휴우즈 트레이닝, 인크. | Portable exploitation and control system |
US5325449A (en) | 1992-05-15 | 1994-06-28 | David Sarnoff Research Center, Inc. | Method for fusing images and apparatus therefor |
US5274236A (en) | 1992-12-16 | 1993-12-28 | Westinghouse Electric Corp. | Method and apparatus for registering two images from different sensors |
US5625765A (en) | 1993-09-03 | 1997-04-29 | Criticom Corp. | Vision systems including devices and methods for combining images for extended magnification schemes |
US5572401A (en) | 1993-12-13 | 1996-11-05 | Key Idea Development L.L.C. | Wearable personal computer system having flexible battery forming casing of the system |
FR2718519B1 (en) | 1994-04-12 | 1996-04-26 | Thomson Csf | Aiming device for weapon, and equipped weapon. |
US5555324A (en) | 1994-11-01 | 1996-09-10 | Massachusetts Institute Of Technology | Method and apparatus for generating a synthetic image by the fusion of signals representative of different views of the same scene |
JP3599828B2 (en) | 1995-05-18 | 2004-12-08 | オリンパス株式会社 | Optical device |
DE19546017C1 (en) | 1995-12-09 | 1997-04-24 | Daimler Benz Aerospace Ag | Missile weapon system |
SE506468C2 (en) | 1996-01-08 | 1997-12-22 | Tommy Andersson | Hit position marker for shotgun shooting |
US5864481A (en) | 1996-01-22 | 1999-01-26 | Raytheon Company | Integrated, reconfigurable man-portable modular system |
US6181302B1 (en) | 1996-04-24 | 2001-01-30 | C. Macgill Lynde | Marine navigation binoculars with virtual display superimposing real world image |
US5834676A (en) | 1996-08-12 | 1998-11-10 | Sight Unseen | Weapon-mounted location-monitoring apparatus |
JP3851384B2 (en) | 1996-09-18 | 2006-11-29 | シャープ株式会社 | Image composition apparatus and method |
US5711104A (en) | 1996-12-19 | 1998-01-27 | Schmitz; Geoffrey W. | Small arms visual aiming system, a method for aiming a firearm, and headgear for use therewith |
IL120840A (en) | 1997-05-16 | 2000-09-28 | Planop Planar Optics Ltd | Distant aiming apparatus particularly for a firearm |
US5806229A (en) | 1997-06-24 | 1998-09-15 | Raytheon Ti Systems, Inc. | Aiming aid for use with electronic weapon sights |
US7832137B2 (en) | 1997-12-08 | 2010-11-16 | Horus Vision, Llc | Apparatus and method for calculating aiming point information |
US6237462B1 (en) | 1998-05-21 | 2001-05-29 | Tactical Telepresent Technolgies, Inc. | Portable telepresent aiming system |
US6123006A (en) | 1998-07-13 | 2000-09-26 | Recon/Optical, Inc. | Retrofit extended vision module for weapon system optical sight |
FR2787566B1 (en) | 1998-12-18 | 2001-03-16 | Sextant Avionique | SIGHT ASSISTING AID METHOD AND SYSTEM |
US7015954B1 (en) | 1999-08-09 | 2006-03-21 | Fuji Xerox Co., Ltd. | Automatic video system using multiple cameras |
US7037258B2 (en) | 1999-09-24 | 2006-05-02 | Karl Storz Imaging, Inc. | Image orientation for endoscopic video displays |
JP3632563B2 (en) | 1999-10-19 | 2005-03-23 | 株式会社豊田自動織機 | Image positional relationship correction device, steering assist device including the image positional relationship correction device, and image positional relationship correction method |
US6899539B1 (en) * | 2000-02-17 | 2005-05-31 | Exponent, Inc. | Infantry wearable information and weapon system |
US20030210228A1 (en) | 2000-02-25 | 2003-11-13 | Ebersole John Franklin | Augmented reality situational awareness system and method |
US20020015536A1 (en) | 2000-04-24 | 2002-02-07 | Warren Penny G. | Apparatus and method for color image fusion |
DE10027657B4 (en) * | 2000-06-03 | 2004-09-30 | Robert Bosch Gmbh | Method for determining reference points in a fingerprint image |
US6781127B1 (en) | 2000-06-08 | 2004-08-24 | Equinox Corporation | Common aperture fused reflective/thermal emitted sensor and system |
US6449419B1 (en) | 2000-09-05 | 2002-09-10 | Richard Brough | Optical viewing system and clamping device therefor |
US6765569B2 (en) | 2001-03-07 | 2004-07-20 | University Of Southern California | Augmented-reality tool employing scene-feature autocalibration during camera motion |
JP4878083B2 (en) | 2001-03-13 | 2012-02-15 | キヤノン株式会社 | Image composition apparatus and method, and program |
US20040257441A1 (en) | 2001-08-29 | 2004-12-23 | Geovantage, Inc. | Digital imaging system for airborne applications |
WO2003025854A2 (en) | 2001-09-17 | 2003-03-27 | Bae Systems Information & Electronic Systems Integration Inc. | Co-registered acoustical and optical cameras for underwater imaging |
US6560029B1 (en) | 2001-12-21 | 2003-05-06 | Itt Manufacturing Enterprises, Inc. | Video enhanced night vision goggle |
US7369174B2 (en) | 2001-12-27 | 2008-05-06 | Sage Technologies Ltd. | Helmet-mounted thermal imaging system |
US6963800B1 (en) * | 2002-05-10 | 2005-11-08 | Solider Vision | Routing soldiers around enemy attacks and battlefield obstructions |
JP2004163413A (en) | 2002-08-28 | 2004-06-10 | Carl-Zeiss-Stiftung Trading As Carl Zeiss | Microscope system and microscope inspection method |
US7002551B2 (en) * | 2002-09-25 | 2006-02-21 | Hrl Laboratories, Llc | Optical see-through augmented reality modified-scale display |
US6836320B2 (en) | 2002-10-23 | 2004-12-28 | Ae Systems Information And Electronic Systems Integration Inc. | Method and apparatus for active boresight correction |
US7050078B2 (en) | 2002-12-19 | 2006-05-23 | Accenture Global Services Gmbh | Arbitrary object tracking augmented reality applications |
ITMI20030121A1 (en) | 2003-01-27 | 2004-07-28 | Giuseppe Donato | MODULAR SURVEILLANCE SYSTEM FOR MONITORING OF CRITICAL ENVIRONMENTS. |
US7756635B2 (en) * | 2003-04-09 | 2010-07-13 | Primordial, Inc. | Method and system for generating and presenting off-road travel routes |
KR100562414B1 (en) | 2003-04-23 | 2006-03-17 | 한창의 | Cyber pointer system and it's control process |
US7401920B1 (en) | 2003-05-20 | 2008-07-22 | Elbit Systems Ltd. | Head mounted eye tracking and display system |
JP2005108108A (en) | 2003-10-01 | 2005-04-21 | Canon Inc | Operating device and method for three-dimensional cg and calibration device for position/attitude sensor |
US20080048931A1 (en) | 2003-11-26 | 2008-02-28 | Rafael - Armament Development Authority Ltd. | Helmet System for Information or Weapon Systems |
CN1624414A (en) | 2003-12-02 | 2005-06-08 | 郑忠田 | Electronic image display and sighting device of gun |
US20050179799A1 (en) | 2004-02-14 | 2005-08-18 | Umanskiy Yuriy K. | Firearm mounted video camera |
US20060055786A1 (en) | 2004-03-09 | 2006-03-16 | Viosport | Portable camera and wiring harness |
US7634139B2 (en) * | 2004-03-16 | 2009-12-15 | Sony Corporation | System and method for efficiently performing a pattern matching procedure |
US20060215935A1 (en) | 2004-04-02 | 2006-09-28 | The Boeing Company | System and architecture for automatic image registration |
US7620265B1 (en) | 2004-04-12 | 2009-11-17 | Equinox Corporation | Color invariant image fusion of visible and thermal infrared video |
CN1727836A (en) | 2004-07-29 | 2006-02-01 | 白世忠 | Tactical operation platform for handgum aiming fire with line changed |
US7787012B2 (en) | 2004-12-02 | 2010-08-31 | Science Applications International Corporation | System and method for video image registration in a heads up display |
DE102005013117A1 (en) * | 2005-03-18 | 2006-10-05 | Rudolf Koch | Rifle with a aiming device |
US20100258000A1 (en) | 2005-05-05 | 2010-10-14 | Matthew Charles Hagerty | Wireless, waterproof remote video weapon mounted sighting system, SmartSight |
AU2006244201B2 (en) | 2005-05-09 | 2012-09-06 | Lockheed Martin Corporation | Continuous extended range image processing |
EP1889113B1 (en) | 2005-05-30 | 2013-05-01 | Durner, Andreas | System having day and night vision spectacles |
US7286062B2 (en) * | 2005-06-29 | 2007-10-23 | Honeywell International, Inc. | Perspective view conformal traffic targets display |
IL174412A0 (en) | 2006-03-20 | 2006-12-31 | Israel Rom | A device for orientation, navigation, and target acquisition and a method of use thereof |
US7805020B2 (en) | 2006-07-25 | 2010-09-28 | Itt Manufacturing Enterprises, Inc. | Motion compensated image registration for overlaid/fused video |
US20090040308A1 (en) | 2007-01-15 | 2009-02-12 | Igor Temovskiy | Image orientation correction method and system |
US9229230B2 (en) | 2007-02-28 | 2016-01-05 | Science Applications International Corporation | System and method for video image registration and/or providing supplemental data in a heads up display |
US8587659B1 (en) | 2007-05-07 | 2013-11-19 | Equinox Corporation | Method and apparatus for dynamic image registration |
-
2007
- 2007-02-28 US US11/680,207 patent/US9229230B2/en active Active
-
2015
- 2015-11-24 US US14/950,643 patent/US9618752B2/en active Active
-
2017
- 2017-04-10 US US15/483,086 patent/US10139629B2/en active Active
-
2018
- 2018-10-19 US US16/165,210 patent/US20190049734A1/en not_active Abandoned
-
2022
- 2022-04-19 US US17/723,730 patent/US11965714B2/en active Active
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2023057654A1 (en) * | 2021-10-08 | 2023-04-13 | Uab "Yukon Advanced Optics Worldwide" | Observation device |
GB2614032A (en) * | 2021-10-08 | 2023-06-28 | Uab Yukon Advanced Optics Worldwide | Observation device |
US11460270B1 (en) * | 2021-11-23 | 2022-10-04 | George Carter | System and method utilizing a smart camera to locate enemy and friendly forces |
Also Published As
Publication number | Publication date |
---|---|
US9229230B2 (en) | 2016-01-05 |
US20080204361A1 (en) | 2008-08-28 |
US20170212353A1 (en) | 2017-07-27 |
US10139629B2 (en) | 2018-11-27 |
US9618752B2 (en) | 2017-04-11 |
US20160077343A1 (en) | 2016-03-17 |
US11965714B2 (en) | 2024-04-23 |
US20230042217A1 (en) | 2023-02-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11965714B2 (en) | System and method for video image registration and/or providing supplemental data in a heads up display | |
US8817103B2 (en) | System and method for video image registration in a heads up display | |
CN111417952B (en) | Device with network-connected scope to allow multiple devices to track a target simultaneously | |
JP3490706B2 (en) | Head tracker system | |
US8209140B2 (en) | Cooperative calibration of heading sensors | |
US20080048931A1 (en) | Helmet System for Information or Weapon Systems | |
US10425540B2 (en) | Method and system for integrated optical systems | |
US20070103671A1 (en) | Passive-optical locator | |
US10510137B1 (en) | Head mounted display (HMD) apparatus with a synthetic targeting system and method of use | |
US7518713B2 (en) | Passive-optical locator | |
US10704864B1 (en) | System for tracking a presumed target using scopes that are remotely located from each other | |
US20180039061A1 (en) | Apparatus and methods to generate images and display data using optical device | |
KR20210082432A (en) | direct view optics | |
KR20210133972A (en) | Vehicle-mounted device with networked scopes for simultaneous tracking of targets from multiple different devices | |
US9476676B1 (en) | Weapon-sight system with wireless target acquisition | |
US20230145665A1 (en) | Multi-platform integrated display | |
US20240230279A1 (en) | System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display | |
US20220042769A1 (en) | Autonomous optronic module for geolocated target pointing for a portable system, and corresponding system | |
US20210081037A1 (en) | Large Area Tracker with Milliwave Boresight Capability | |
EP3361213B1 (en) | A system for the determination of the position of an observed target |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
AS | Assignment |
Owner name: SCIENCE APPLICATIONS INTERNATIONAL CORPORATION, VIRGINIA Free format text: CHANGE OF NAME;ASSIGNOR:SAIC GEMINI, INC.;REEL/FRAME:058244/0214 Effective date: 20130927 Owner name: SAIC GEMINI, INC., VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SCIENCE APPLICATIONS INTERNATIONAL CORPORATION;REEL/FRAME:058001/0936 Effective date: 20130927 Owner name: SCIENCE APPLICATIONS INTERNATIONAL CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SCALES, JOHN RICHARD;RODGERS, MICHAEL HARRIS;SIGNING DATES FROM 20070219 TO 20070228;REEL/FRAME:058001/0884 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |