US20230368577A1 - Automotive and industrial motion sensory device - Google Patents
Automotive and industrial motion sensory device Download PDFInfo
- Publication number
- US20230368577A1 US20230368577A1 US18/225,598 US202318225598A US2023368577A1 US 20230368577 A1 US20230368577 A1 US 20230368577A1 US 202318225598 A US202318225598 A US 202318225598A US 2023368577 A1 US2023368577 A1 US 2023368577A1
- Authority
- US
- United States
- Prior art keywords
- information
- display
- augmented reality
- gesture
- imaging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000033001 locomotion Effects 0.000 title claims abstract description 36
- 230000001953 sensory effect Effects 0.000 title claims abstract description 25
- 238000003384 imaging method Methods 0.000 claims description 29
- 230000003190 augmentative effect Effects 0.000 claims description 26
- 238000005286 illumination Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 13
- 238000000034 method Methods 0.000 claims description 10
- 238000005516 engineering process Methods 0.000 abstract description 11
- 238000004458 analytical method Methods 0.000 description 24
- 238000010191 image analysis Methods 0.000 description 15
- 239000000872 buffer Substances 0.000 description 13
- 230000003287 optical effect Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000001429 visible spectrum Methods 0.000 description 2
- 101100264195 Caenorhabditis elegans app-1 gene Proteins 0.000 description 1
- 238000012952 Resampling Methods 0.000 description 1
- 241000270295 Serpentes Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000002775 capsule Substances 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000005670 electromagnetic radiation Effects 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004313 glare Effects 0.000 description 1
- 210000000003 hoof Anatomy 0.000 description 1
- 238000002513 implantation Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 230000031893 sensory processing Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000007474 system interaction Effects 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01P—MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT
- G01P13/00—Indicating or recording presence, absence, or direction, of movement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/002—Specific input/output arrangements not covered by G06F3/01 - G06F3/16
- G06F3/005—Input arrangements through a video camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
- H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30268—Vehicle interior
Definitions
- the technology disclosed relates to highly functional/highly accurate motion sensory control devices capable of capturing and providing images to motion capture systems that detect gestures in a three dimensional (3D) sensory space for use in automotive and industrial control systems.
- the first dashboard consisted of a board placed in front of the driver of a carriage to shield from debris cast off the horses' hooves.
- controls for various systems environmental, safety, entertainment and so forth proliferated.
- the dashboard was retained as a convenient place for various controls. The operator's attention must be removed from the road (or runway, rail or sea-lane) to “hunt” for the knob or switch, hopefully labelled in his or her own language.
- replacing English language labels with international symbols made the dashboard equally unintelligible to everyone everywhere.
- the need for a more simplified interface became apparent and joysticks, keyboards or keypads, glass cockpits, and so forth were pressed into service. But complexity—and confusion—proliferated.
- Implementations of the technology disclosed address these and other problems by providing an embeddable motion sensory control device capable of acquiring imaging information of a scene and providing at least a near real time (i.e., sufficiently fast that any residual lag between the scene change and the system's response is unnoticeable or practically insignificant) stream of imaging information to a motion capture or image analyzer that detects gestures in a three dimensional (3D) sensory space, interprets the gesture as a command to a system or machine under control, and issuing the command when appropriate.
- the device can be embedded in a wide variety of machines or systems.
- an embeddable motion sensory control device in a representative implementation, includes a plurality of imaging sensors arranged on a first portion that provide stereoscopic imaging information for a scene being viewed.
- One or more illumination sources arranged on a second portion are also included.
- a controller is coupled to the imaging sensors and illumination sources to control operation thereof, acquiring imaging information of a scene, and providing at least a near real time stream of the imaging information to a system or device under control.
- some implementations can provide improved user experience, greater safety and improved functionality. Some implementations can enable motion capture or image analysis systems to recognize gestures, thereby enabling an operator to control a device or system, such as a vehicle or vehicle subsystem, by intuitive gesture sets. Some implementations can provide improved interfacing and/or control with a variety of machines (e.g., aircraft or automobiles, trains, planes, forklifts, ships and so forth) for example.
- machines e.g., aircraft or automobiles, trains, planes, forklifts, ships and so forth
- Devices can be embedded within the machine under control and can work cooperatively with a proxy or supporting device (smart telephones, portable computing systems, including laptop, tablet computing devices, personal data assistants, special purpose visualization computing machinery, including heads up displays (HUD), wearable virtual and/or augmented reality systems, including Google Glass, and others, graphics processors, embedded microcontrollers, gaming consoles, or the like; wired or wirelessly coupled networks of one or more of the foregoing, and/or combinations thereof).
- Device implementation can obviate or reduce the need for contact-based input devices such as a mouse, joystick, touch pad, or touch screen.
- Some implementations can provide for improved interface with computing and/or other machinery than would be possible with heretofore known techniques.
- a richer human—machine interface experience can be provided.
- FIG. 1 shows an example motion sensory and imaging device.
- FIG. 2 shows an example sensory system based upon a motion sensory and imaging device.
- FIG. 3 shows a simplified block diagram of a computer system.
- FIG. 4 shows basic operations and functional units involved in motion capture and image analysis.
- FIG. 5 shows an example augmented reality presented by a motion sensory and imaging capable device.
- FIG. 1 shows an example motion sensory device 100 , which includes a main board 182 (first portion) couplable to an illumination board 172 (second portion) with threaded fasteners or otherwise. Electrical interconnection between illumination board 172 and main board 182 permits exchange of signals and power flow and can be made by board mounted connectors, cabling or combinations thereof.
- One or more fasteners can fasten the device 100 to a mounting surface A of a host device (e.g., a vehicle (with or without a heads-up display (HUD), an appliance or portable electronic device (such as a smartphone or tablet computer) or a wearable device (such as a head mounted display (HMD), headsets, or the like) that interfaces with a vehicle or otherwise).
- a host device e.g., a vehicle (with or without a heads-up display (HUD), an appliance or portable electronic device (such as a smartphone or tablet computer) or a wearable device (such as a head mounted display (HMD), headsets, or the like) that interface
- the mounting surface A can be any surface (internal or external) of the host device.
- the device can be disposed within a cavity or receptacle of a host device using a friction fit, fasteners or any combination thereof.
- Device 100 can be embedded in any of a variety of equipment to meet design requirements of a wide variety of applications.
- the illumination board 172 has a number of individually controllable illumination sources 108 , 110 , which can be LEDs or other sources, embedded thereon.
- Two cameras 102 , 104 provide stereoscopic image-based sensing and reside on the main board 182 of device 100 in the illustrated implementation.
- the main board 182 may also include a processor conducting basic image processing, control of the cameras 102 , 104 and the sources 108 , 110 .
- Stereoscopic imaging information provided by cameras 102 , 104 can be provided selectively or continuously to a user by means of a presentation device (HUD, dashboard/console mounted display device, wireless transmission to a display associated with a portable device, or a wearable appliance (HMD).
- the device 100 can provide live real time or near real time image information from the cameras, real time or near real time imaging information augmented by computer generated graphics, information, icons or other virtualized presentations, virtualized representations of the scene being viewed, and/or time varying combinations selected therefrom.
- Gestures made by a user are sensed by the cameras 102 , 104 of the sensory device 100 , and the resulting imaging information can be provided to a motion capture or image analysis system to identify and determine commands to a system.
- integrating scanning with imaging capabilities into a single motion sensory device 100 provides a highly functional, flexible, yet compact device suited to installation in machines with limited space, such as, e.g., vehicles, appliances, portable or wearable electronic devices, and so forth.
- Some of the illumination sources 108 , 110 can have associated focusing optics.
- six LEDs 108 (four of which are arranged at the center and two of which flank the board 172 at the sides) have focusing lenses, and ten additional LEDs 110 (which are arranged in columns of two, three, three, and two LEDs, respectively) are without focusing lenses.
- the board 172 may also include a socket 178 for coupling a photo-detector (or other sensor). Information from a photo-detector sensing changes in reflectance indicating presence or absence of objects within a region of space into which the illumination sources 108 , 110 emit light during a “scanning” of the region of space.
- FIG. 1 Various modifications of the design shown in FIG. 1 are possible; for example the number and arrangement of LEDs, photo-detectors, and cameras may vary, the illumination and imaging hardware may be integrated on a single board, other modifications or combinations thereof depending upon the requirements of a particular application.
- System 200 deployed in an automobile cabin to recognize gestures is shown.
- System 200 is preferably implemented within a vehicle cabin 201 either in the front seat area, rear seat area, other placements or combinations.
- all or portions of sensory system 200 can be affixed to or embedded within dashboard 216 , console between front seat passengers, e.g., sensor 200 - 1 , the cabin roof, e.g., sensor 200 - 2 , the backs of the front seats, e.g., sensor 200 - 3 , other locations, or combinations thereof.
- System 200 includes any number of cameras 102 , 104 coupled to a sensory-analysis system 206 .
- Cameras 102 , 104 can be any type of camera, including cameras sensitive across the visible spectrum (e.g., red-green-blue or RGB) or, more typically, with enhanced sensitivity to a confined wavelength band (e.g., the infrared (IR) or ultraviolet (UV) bands) or combinations thereof; more generally, the term “camera” herein refers to any device (or combination of devices) capable of capturing an image of an object and representing that image in the form of digital data. While illustrated using an example of a two-camera implementation, other implementations are readily achievable using different numbers of cameras or non-camera light sensitive image sensors or combinations thereof. For example, line sensors or line cameras rather than conventional devices that capture a two-dimensional (2D) image can be employed.
- the term “light” is used generally to connote any electromagnetic radiation, which may or may not be within the visible spectrum, and may be broadband (e.g., white light) or narrowband (e.g., a single wavelength or narrow band of wavelengths).
- sensory system 200 is capable of separating information received from pixels of cameras 102 , 104 sensitive to IR light from information received from pixels sensitive to visible light, e.g., RGB (red, green, and blue) and processing these two types of image information separately.
- IR infrared
- RGB visible light
- a video stream including a sequence of images of a scene in the real world can be captured using cameras having a set of RGB pixels and a set of IR pixels. Information from the IR sensitive pixels is separated out for processing to recognize gestures.
- Information from the RGB sensitive pixels is provided to a presentation interface (HUD, HMD, etc.) of a host device as a live video feed to a presentation output.
- the presentation output is displayed to a user.
- One or more virtual objects can be integrated with the video stream images to form the presentation output.
- the sensory system 200 can provide any of gesture recognition, a real world presentation of real world objects via pass through video feed, and/or an augmented reality including virtual objects integrated with a real world view.
- Cameras 102 , 104 are preferably capable of capturing video images (i.e., successive image frames at a constant rate of at least 15 frames per second); although no particular frame rate is required.
- the capabilities of cameras 102 , 104 are not critical to the technology disclosed, and the cameras can vary as to frame rate, image resolution (e.g., pixels per image), color or intensity resolution (e.g., number of bits of intensity data per pixel), focal length of lenses, depth of field, etc.
- image resolution e.g., pixels per image
- color or intensity resolution e.g., number of bits of intensity data per pixel
- focal length of lenses e.g., depth of field, etc.
- any cameras capable of focusing on objects within a spatial volume of interest can be used.
- the volume of interest can be defined as a cube approximately one meter on a side.
- the cameras 102 , 104 are disposed opposite the motion to be detected, e.g., where the hand 214 is expected to move.
- the amount of information recorded about the hand is proportional to the number of pixels it occupies in the camera images, and the hand will occupy more pixels when the camera's angle with respect to the hand's “pointing direction” is as close to perpendicular as possible.
- the sensor is disposed along the motion detected, e.g., where the hand 214 is expected to move.
- the one or more sources 108 , 110 can be disposed to illuminate region of interest 212 in which one or more portions of the operator (or occupant's) body in this example a hand 214 ) that may optionally hold a tool or other object of interest and cameras 102 , 104 are oriented toward the region 212 to capture video images of the hand 214 .
- the operation of light sources 108 , 110 and cameras 102 , 104 is controlled by sensory-analysis system 206 which can be a computer system, control logic implemented in hardware and/or software or combinations thereof. Based on the captured images, sensory-analysis system 206 determines the position and/or motion of object 214 .
- the sources 108 , 110 are infrared light sources.
- the light sources can be, e.g., infrared light-emitting diodes (LEDs), and cameras 102 , 104 can be sensitive to infrared light.
- LEDs infrared light-emitting diodes
- Use of infrared light can allow the system 200 to operate under a broad range of lighting conditions and can avoid various inconveniences or distractions that may be associated with directing visible light into the region where the person is moving.
- a particular wavelength or region of the electromagnetic spectrum need not be required.
- filters 221 , 222 are placed in front of cameras 102 , 104 to filter out extraneous light so that only the light provided by sources 108 , 110 is registered in the images captured by cameras 102 , 104 .
- the system selectively chooses to process visible (RGB) information or infrared (IR) information from cameras 102 , 104 differently, separately or in conjunction with one another to adjust operation of system 200 to varying ambient conditions.
- one or more sonic transducers 215 , 217 are sonic sources sending sonic energy and detecting reflected sonic energy used in conjunction with, or instead of, cameras 102 , 104 and light sources 108 , 110 .
- the sonic sources transmit sound waves to the user; the user either blocks (or “sonic shadowing”) or alters the sound waves (or “sonic deflections”) that impinge upon her.
- Such sonic shadows and/or deflections can also be used to detect the user's gestures and/or provide presence information and/or distance information using ranging techniques known in the art.
- the sound waves are, for example, ultrasound, that is not audible to humans.
- lasers or other radiation emitting devices can be used to detect position, presence or both of hand 214 .
- the illustrated system 200 can include any of various other sensors not shown in FIG. 2 for clarity, alone or in various combinations, to enhance the experience provided to the operator of vehicle 201 .
- system 206 may switch to a touch mode in which touch gestures are recognized based on acoustic or vibrational sensors.
- system 206 may switch to the touch mode, or supplement image capture and processing with touch sensing, when signals from acoustic or vibrational sensors are sensed.
- a tap or touch gesture may act as a “wake up” signal to bring the image and audio analysis system 206 from a standby mode to an operational mode.
- the system 206 may enter the standby mode if optical signals from the cameras 102 , 104 are absent for longer than a threshold interval.
- FIG. 2 is representative and not limiting.
- lasers or other light sources can be used instead of LEDs.
- Visible light (RGB) can be used instead of infrared (IR) light.
- information from one type of light source can be used to clarify, supplement, confirm or reject results from analyzing information gathered using a second type of light source.
- additional optics e.g., a lens or diffuser
- Useful arrangements can also include short- and wide-angle illuminators for different ranges.
- Light sources are typically diffuse rather than specular point sources; for example, packaged LEDs with light-spreading encapsulation are suitable.
- FIG. 3 is a simplified block diagram of a computer system 300 , for implementing all or portions of sensory-analysis system 206 (also referred to as an image analyzer) according to an implementation of the technology disclosed.
- Sensory-analysis system 206 can include or consist of any device or device component that is capable of capturing and processing image data.
- computer system 300 includes a processor 342 , memory 334 , a sensor interface 336 , a presentation I/F 238 which can include a display or other presentation mechanism(s), (e.g.
- Memory 334 can be used to store instructions to be executed by processor 342 as well as input and/or output data associated with execution of the instructions.
- memory 334 contains instructions, conceptually illustrated as a group of modules described in greater detail below, that control the operation of processor 342 and its interaction with the other hardware components.
- An operating system directs the execution of low-level, basic system functions such as memory allocation, file management and operation of mass storage devices.
- the operating system may be or include a variety of operating systems such as Microsoft WINDOWS operating system, the Unix operating system, the Linux operating system, the Xenix operating system, the IBM AIX operating system, the Hewlett Packard UX operating system, the Novell NETWARE operating system, the Sun Microsystems SOLARIS operating system, the OS/2 operating system, the BeOS operating system, the MAC OS operating system, the APACHE operating system, an OPENACTION operating system, iOS, Android or other mobile operating systems, or another operating system or platform.
- operating systems such as Microsoft WINDOWS operating system, the Unix operating system, the Linux operating system, the Xenix operating system, the IBM AIX operating system, the Hewlett Packard UX operating system, the Novell NETWARE operating system, the Sun Microsystems SOLARIS operating system, the OS/2 operating system, the BeOS operating system, the MAC OS operating system, the APACHE operating system, an OPENACTION operating system, iOS, Android or other mobile operating systems, or another operating system or platform.
- the computing environment can also include other removable/non-removable, volatile/nonvolatile computer storage media.
- a hard disk drive can read or write to non-removable, nonvolatile magnetic media.
- a magnetic disk drive can read from or write to a removable, nonvolatile magnetic disk
- an optical disk drive can read from or write to a removable, nonvolatile optical disk such as a CD-ROM or other optical media.
- Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
- the storage media are typically connected to the system bus through a removable or non-removable memory interface.
- Processor 342 can be a general-purpose microprocessor, but depending on implementation can alternatively be a microcontroller, peripheral integrated circuit element, a CSIC (customer-specific integrated circuit), an ASIC (application-specific integrated circuit), a logic circuit, a digital signal processor, a programmable logic device such as an FPGA (field-programmable gate array), a PLD (programmable logic device), a PLA (programmable logic array), an RFID processor, smart chip, or any other device or arrangement of devices that is capable of implementing the actions of the processes of the technology disclosed.
- a programmable logic device such as an FPGA (field-programmable gate array), a PLD (programmable logic device), a PLA (programmable logic array), an RFID processor, smart chip, or any other device or arrangement of devices that is capable of implementing the actions of the processes of the technology disclosed.
- Sensor interface 336 can include hardware, firmware and/or software that enables communication between computer system 300 and cameras 102 , 104 shown in FIG. 1 , as well as associated light sources 108 , 110 .
- sensor interface 336 can include one or more data ports 346 , 348 to which cameras 102 , 104 can be connected, as well as hardware, firmware and/or software signal processors to modify data signals received from the cameras (e.g., to reduce noise or reformat data) prior to providing the signals as inputs to a motion-capture (“mocap”) program 344 executing on processor 342 .
- micap motion-capture
- sensor interface 336 can also transmit signals to the cameras, e.g., to activate or deactivate the cameras, to control camera settings (frame rate, image quality, sensitivity, etc.), and/or other optional sensors to control sensor settings (calibration, sensitivity levels, etc.), or the like. Such signals can be transmitted, e.g., in response to control signals from processor 342 , which can in turn be generated in response to user input or other detected events.
- Sensor interface 336 can also include controllers 347 , 349 , to which light sources (e.g., light sources 108 , 110 ) can be connected.
- controllers 347 , 349 provide operating current to the light sources, e.g., in response to instructions from processor 342 executing mocap program 344 .
- the light sources can draw operating current from an external power supply, and controllers 347 , 349 can generate control signals for the light sources, e.g., instructing the light sources to be turned on or off or changing the brightness.
- a single controller can be used to control multiple light sources.
- mocap program 344 Instructions defining mocap program 344 are stored in memory 334 , and these instructions, when executed, perform motion-capture analysis on images supplied from cameras connected to sensor interface 336 .
- mocap program 344 includes various modules, such as an object detection module 352 , an object analysis module 354 , and a gesture-recognition module 356 .
- Object detection module 352 can analyze images (e.g., images captured via sensor interface 336 ) to detect edges of an object therein and/or other information about the object's location.
- Object analysis module 354 can analyze the object information provided by object detection module 352 to determine the 3D position and/or motion of the object (e.g., a user's hand).
- object analysis module 354 can also analyze audio signals (e.g., audio signals captured via interface 336 ) to localize the object by, for example, time distance of arrival, multilateration or the like.
- audio signals e.g., audio signals captured via interface 336
- Memory 334 can also include other information and/or code modules used by mocap program 344 such as an application platform 366 that allows a user to interact with the mocap program 344 using different applications like application 1 (App1), application 2 (App2), and application N (AppN).
- application platform 366 that allows a user to interact with the mocap program 344 using different applications like application 1 (App1), application 2 (App2), and application N (AppN).
- results of gesture capture using sensor interface 336 and mocap program 344 can be interpreted as user input.
- a user can perform hand gestures that are analyzed using mocap program 344 , and the results of this analysis can be interpreted as an instruction to some other program executing on processor 342 (e.g., a web browser, GPS application, dictation program, or other application).
- a user might use upward or downward swiping gestures to “scroll” a webpage currently displayed via presentation I/F 238 , to use rotating gestures to increase or decrease the volume of audio output from speakers 339 , and so on.
- computer system 300 is illustrative and that variations and modifications are possible.
- a particular implementation can include other functionality not described herein, e.g., wired and/or wireless network interfaces, media playing and/or recording system interfaces, etc.
- one or more cameras can be built into the vehicle or equipment into which the sensor 200 is imbedded rather than being supplied as separate components.
- an image analyzer can be implemented using only a subset of computer system components (e.g., as a processor executing program code, an ASIC, or a fixed-function digital signal processor, with suitable I/O interfaces to receive image data and output analysis results).
- the user e.g., vehicle operator, occupant(s) performs a gesture that is captured by the cameras 102 , 104 as a series of temporally sequential images.
- cameras 102 , 104 can capture any observable pose or portion of a user. For instance, if a user places a large portion of their body into the field of view near the cameras 102 , 104 , cameras 102 , 104 can capture not only the intended hand gestures, but the whole body of the user can be within the field of view of the cameras.
- Motions of the hand 214 can be distinguished by object analysis module 354 and analyzed by a gesture-recognition module 356 , of the mocap 344 .
- Gesture-recognition module 356 provides input to one or more systems or subsystems of the vehicle or other controlled equipment, allowing a user to remotely control those systems or subsystems and/or virtually manipulate objects, such as virtualized gauges, indicators, buttons, levers, or other controls, in a virtual environment displayed via presentation I/F 238 .
- the user can perform the gesture using any part of her body, such as a finger, a hand, or an arm or combinations thereof.
- the sensory-analysis system 206 can determine the shapes and positions of the user's hand in 3D space and in real time using algorithms described below as well as, in more detail, in, e.g., U.S. Ser. No. 61/587,554, Ser. Nos.
- the sensory-analysis system 206 can not only recognize gestures for purposes of providing input to the electronic device, but can also capture the position and shape of the user's hand in consecutive video images in order to characterize the hand gesture in 3D space and reproduce it on a display screen, for example via presentation I/F 238 , as a rigged hand 99 for example.
- Rigged hand 99 is determined from model hand 98 that includes a rigged hand overlay 94 covering one or more capsule elements 97 built from the images by the object detection module 352 .
- a rigged hand representation 599 can be projected as part of an augmented vehicle environment onto a (semi-)transparent or other surface through which one or more real objects can be viewed also, e.g., a windshield 516 according to one implementation of the technology disclosed.
- a windshield 516 e.g., a windshield 516
- a video projector 504 coupled to presentation I/F 238 , can project an image of an arrow (e.g., virtual device 523 ) from a GPS application superimposed upon a (semi-) transparent surface, e.g., windshield 516 , being viewed directly by the user (or displayed to a user via live video feed in the event the operator is remote (e.g., operating a drone or a driverless vehicle) from the vehicle or the vehicle lacks a windshield); thereby creating an augmented environment experience when operating the vehicle.
- Optional haptic projector 506 can project the feeling of the pressure of clicking or pressing a physical control.
- Optional audio projector 502 can project the sounds of virtual “speed bumps” indicating the driver of a car should pay attention to the change in route, an oncoming obstacle or the speed of the vehicle.
- the gesture-recognition module 356 compares one or more primitives of the detected gesture to a library of gesture primitives electronically stored as records in a database, which is implemented in the sensory-analysis system 206 , the electronic device, or on an external storage system.
- gestures can be stored as vectors, i.e., mathematically specified spatial trajectories, other primitives or combinations thereof, and the gesture record can have a field specifying the relevant part of the user's body making the gesture; thus, similar trajectories executed by a user's hand and head can be stored in the database as different gestures so that an application can interpret them differently.
- FIG. 4 depicts the basic operations and functional units 400 involved in motion capture and image analysis in accordance with implementations of the technology disclosed.
- the camera(s) 402 , 404 record digital images 410 of a scene.
- Each digital image is captured as an array of pixel values by the associated camera's image sensor, and the digital images are transferred—either in “raw” format or following conventional preprocessing—to one or more frame buffers 415 .
- a frame buffer is a partition or dedicated segment of volatile memory that stores a “bitmapped” image frame 420 corresponding to the pixel values of an image as output by the camera that recorded it.
- the bitmap is generally organized conceptually as a grid, with each pixel mapped one-to-one or otherwise to output elements of a display. It should be stressed, however, that the topology of how memory cells are physically organized within the frame buffers 415 does not matter and need not conform directly to the conceptual organization.
- the number of frame buffers included in a system generally reflects the number of images simultaneously analyzed by the analysis system or module 430 , which is described in greater detail below.
- analysis module 430 analyzes the pixel data in each of a sequence of image frames 420 to locate objects therein and track their movement over time (as indicated at 440 ). This analysis can take various forms, and the algorithm performing the analysis dictates how pixels in the image frames 420 are handled. For example, the algorithm implemented by analysis module 430 can process the pixels of each frame buffer on a line-by-line basis—i.e., each row of the pixel grid is successively analyzed. Other algorithms can analyze pixels in columns, tiled areas, or other organizational formats.
- the motion captured in a series of camera images is used to compute a corresponding series of output images for display via the presentation I/F 238 .
- camera images of a moving hand can be translated into a wire-frame or other graphic depiction of the hand by the processor 342 .
- hand gestures can be interpreted as input used to control a separate visual output; by way of illustration, a user can be able to use upward or downward swiping gestures to “scroll” a webpage or other document currently displayed, or open and close her hand to zoom in and out of the page.
- the output images are generally stored in the form of pixel data in a frame buffer, e.g., one of the frame buffers 415 .
- a video display controller reads out the frame buffer to generate a data stream and associated control signals to output the images via the presentation I/F 238 .
- the video display controller can be provided along with the processor 342 and memory 334 on-board the motherboard of the computer 300 , and can be integrated with the processor 342 or implemented as a co-processor that manipulates a separate video memory.
- the computer 300 can be equipped with a separate graphics or video card that aids with generating the feed of output images for the presentation I/F 238 .
- the video card generally includes a graphics processing unit (GPU) and video memory, and is useful, in particular, for complex and computationally expensive image processing and rendering.
- the graphics card can include the frame buffer and the functionality of the video display controller (and the on-board video display controller can be disabled). In general, the image-processing and motion-capture functionality of the system can be distributed between the GPU and the main processor 342 in various ways.
- Suitable algorithms for motion-capture program 344 are described below as well as, in more detail, in U.S. Ser. No. 61/587,554, Ser. Nos. 13/414,485, 61/724,091, 13/724,357, and 13/742,953, filed on Jan. 17, 2012, Mar. 7, 2012, Nov. 8, 2012, Dec. 21, 2012 and Jan. 16, 2013, respectively, which are hereby incorporated herein by reference in their entirety.
- the various modules can be programmed in any suitable programming language, including, without limitation high-level languages such as C, C++, C #, OpenGL, Ada, Basic, Cobra, FORTRAN, Java, Lisp, Perl, Python, Ruby, or Object Pascal, or low-level assembly languages.
- the modes of operation of a device equipped with a motion sensory control device can determine the coarseness of the data provided to the image-analysis module 430 , the coarseness of its analysis, or both in accordance with entries in a performance database.
- the image-analysis module 430 can operate on every image frame and on all data within a frame, capacity limitations can dictate analysis of a reduced amount of image data per frame (i.e., resolution) or discarding of some frames altogether if the data in each of the frame buffers 415 are organized as a sequence of data lines.
- the manner in which data is dropped from the analysis can depend on the image-analysis algorithm or the uses to which the motion-capture output is put.
- data is dropped in a symmetric or uniform fashion—e.g., every other line, every third line, etc. is discarded up to a tolerance limit of the image-analysis algorithm or an application utilizing its output.
- the frequency of line dropping can increase toward the edges of the frame.
- Still other image-acquisition parameters that can be varied include the frame size, the frame resolution, and the number of frames acquired per second.
- the frame size can be reduced by, e.g., discarding edge pixels or by resampling to a lower resolution (and utilizing only a portion of the frame buffer capacity).
- acquisition parameters parameters relevant to acquisition of image data
- image-analysis parameters parameters relevant to operation of the image-analysis module 430
- acquisition parameters parameters relevant to operation of the image-analysis module 430
- image-analysis parameters parameters relevant to operation of the image-analysis module 430
- acquisition parameters and image-analysis parameters are representative only, and not limiting.
- Acquisition parameters can be applied to the cameras 402 , 404 and/or to the frame buffers 415 .
- the camera 402 , 404 for example, can be responsive to acquisition parameters in operating the cameras 402 , 404 to acquire images at a commanded rate, or can instead limit the number of acquired frames passed (per unit time) to the frame buffers 415 .
- Image-analysis parameters can be applied to the image-analysis module 430 as numerical quantities that affect the operation of the contour-defining algorithm.
- the desirable values for acquisition parameters and image-analysis parameters appropriate to a given level of available resources can depend, for example, on the characteristics of the image-analysis module 430 , the nature of the application utilizing the mocap output, and design preferences. Whereas some image-processing algorithms can be able to trade off a resolution of contour approximation against input frame resolution over a wide range, other algorithms may not exhibit much tolerance at all—requiring, for example, a minimal image resolution below which the algorithm fails altogether.
- System 500 includes a sensory processing system 206 controlling a variety of sensors and projectors, such as for example one or more cameras 102 , 104 (or other image sensors) and optionally some illumination sources 108 , 110 comprising an imaging system.
- a plurality of vibrational (or acoustical) sensors 508 , 510 positioned for sensing contacts with surface (e.g., a touch screen display or the like not shown in FIG.
- Optionally projectors under control of system 206 can render virtual objects to provide the augmented vehicle environment 501 , such as an optional audio projector 502 to provide audio feedback (e.g., “clicks”, chimes, speech feedback when virtual buttons are pressed or virtual controls are manipulated), optional video projector 504 , an optional haptic projector 506 to provide haptic feedback (e.g., button pressure, pressure indicating a physical object has been grasped, etc.) to a user of augmented vehicle environment 501 .
- audio feedback e.g., “clicks”, chimes, speech feedback when virtual buttons are pressed or virtual controls are manipulated
- optional video projector 504 e.g., an optional haptic projector 506 to provide haptic feedback (e.g., button pressure, pressure indicating a physical object has been grasped, etc.) to a user of augmented vehicle environment 501 .
- haptic feedback e.g., button pressure, pressure indicating a physical object has been graspe
- sensors and projectors of system 500 can be incorporated wholly or partly into sensor device 500 - 1 , or otherwise conveniently disposed toward a region of interest 112 in order to sense motion and presence of hand 114 , that moves along the indicated path 118 to give commands to the system 500 by gesturing.
- One or more applications 522 , 523 can be provided as virtual objects integrated into the display of the augmented reality 113 .
- user e.g., owner of hand 114
- view real objects e.g., through windshield 516
- outputs of applications 521 , 522 virtualized into the augmented reality 113 in the same environment 501 .
- Projection can include an image or other visual representation of the user's hand 599 and/or one or more optional objects.
- Objects can include: objects associated with an application 522 , 523 , 524 , objects representing an operational parameter of the vehicle 521 , advertising objects 517 , objects representing more abstract things, other types of objects, and combination objects.
- visual projection mechanism 504 of FIG. 5 can project an icon 522 of a person from the operator's contact list.
- a GPS application can plan route to the selected location for the person in the contact list and provide guidance information 523 , also projected as virtualized objects into augmented vehicle environment 501 , to the operator.
- Various objects 521 associated with status monitors provide information about the vehicle's systems.
- An advertising application searches previous destinations of trips made in close proximity to previous trips to visit contact 522 and displays an appropriate advertising object 517 . For example, if the operator visits a friend (contact 522 ) and they've gone to Burger King the last three times, an advertising icon for McDonalds may appear.
- a vehicle “happiness” icon 524 displays an overall level of happiness of the occupants determined from a variety of parameters (e.g., frequency of changes in environmental controls, frequency of change in entertainment selections, amounts of ambient light, temperature, humidity, etc.).
- surface 516 can be a wearable computing device such as Google GlassTM or equivalent connectable wirelessly or by wire to sensory system 500 .
- Projections for augmented vehicle environments can be differentiated for front and rear seat passengers in an automobile for example.
- Front seat passengers can experience clicks, chimes and/or speech feedback responsive to the occupant's gestures.
- Rear seat passengers can experience clicks, chimes and/or speech feedback on separate audio channel to headphones or HMDs used by the rear seat passengers (to avoid distracting driver).
- gestures can be expanded for all front seat passengers to control vehicle (sub)systems.
- Driverless vehicles can include a larger more interactive HUD (up to whole windshield).
- Gestures control non-safety related navigation decisions (e.g., override determined routing, waypoints on moving map display, choosing rest stops for purposes of rerouting (e.g., bathroom breaks), and so forth).
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The technology disclosed relates to highly functional/highly accurate motion sensory control devices for use in automotive and industrial control systems capable of capturing and providing images to motion capture systems that detect gestures in a three dimensional (3D) sensory space.
Description
- This application is a continuation of U.S. application Ser. No. 17/848,181, entitled “AUTOMOTIVE AND INDUSTRIAL MOTION SENSORY DEVICE”, filed Jun. 23, 2022, which is a continuation of U.S. Pat. No. 11,386,711, which issued on Jul. 12, 2022 and is entitled “AUTOMOTIVE AND INDUSTRIAL MOTION SENSORY DEVICE, which claims the benefit of U.S. Provisional Patent Application No. 62/038,112, entitled “AUTOMOTIVE AND INDUSTRIAL MOTION SENSORY DEVICE”, filed Aug. 15, 2014, the disclosures of which are incorporated by reference.
- The technology disclosed relates to highly functional/highly accurate motion sensory control devices capable of capturing and providing images to motion capture systems that detect gestures in a three dimensional (3D) sensory space for use in automotive and industrial control systems.
- The first dashboard consisted of a board placed in front of the driver of a carriage to shield from debris cast off the horses' hooves. As vehicles became more complex, and mechanical motive power supplanted the horse, controls for various systems (environmental, safety, entertainment and so forth) proliferated. The dashboard was retained as a convenient place for various controls. The operator's attention must be removed from the road (or runway, rail or sea-lane) to “hunt” for the knob or switch, hopefully labelled in his or her own language. In the 1970s, replacing English language labels with international symbols made the dashboard equally unintelligible to everyone everywhere. The need for a more simplified interface became apparent and joysticks, keyboards or keypads, glass cockpits, and so forth were pressed into service. But complexity—and confusion—proliferated.
- Some have looked to capturing motions of the operator's hands (gestures) and interpreting the motions to provide commands. Some rudimentary efforts by SoftKinetic and others include inferring motion from shadows passing over simple photo-detector sensors. Unfortunately, such systems tend to be prone to false positives. The sensor cannot discriminate between the operator's hand and the wagging of the tail of the family dog. Changing and uncontrollable lighting situations, background objects, glare, reflections and so forth pose further challenges to the use of optical components. To date, such considerations have limited the deployment and use of motion capture technology in the vehicle cabin to little more than non-functional pipe dreams.
- Implementations of the technology disclosed address these and other problems by providing an embeddable motion sensory control device capable of acquiring imaging information of a scene and providing at least a near real time (i.e., sufficiently fast that any residual lag between the scene change and the system's response is unnoticeable or practically insignificant) stream of imaging information to a motion capture or image analyzer that detects gestures in a three dimensional (3D) sensory space, interprets the gesture as a command to a system or machine under control, and issuing the command when appropriate. The device can be embedded in a wide variety of machines or systems.
- In a representative implementation, an embeddable motion sensory control device is provided that includes a plurality of imaging sensors arranged on a first portion that provide stereoscopic imaging information for a scene being viewed. One or more illumination sources arranged on a second portion are also included. A controller is coupled to the imaging sensors and illumination sources to control operation thereof, acquiring imaging information of a scene, and providing at least a near real time stream of the imaging information to a system or device under control.
- Advantageously, some implementations can provide improved user experience, greater safety and improved functionality. Some implementations can enable motion capture or image analysis systems to recognize gestures, thereby enabling an operator to control a device or system, such as a vehicle or vehicle subsystem, by intuitive gesture sets. Some implementations can provide improved interfacing and/or control with a variety of machines (e.g., aircraft or automobiles, trains, planes, forklifts, ships and so forth) for example. Devices can be embedded within the machine under control and can work cooperatively with a proxy or supporting device (smart telephones, portable computing systems, including laptop, tablet computing devices, personal data assistants, special purpose visualization computing machinery, including heads up displays (HUD), wearable virtual and/or augmented reality systems, including Google Glass, and others, graphics processors, embedded microcontrollers, gaming consoles, or the like; wired or wirelessly coupled networks of one or more of the foregoing, and/or combinations thereof). Device implementation can obviate or reduce the need for contact-based input devices such as a mouse, joystick, touch pad, or touch screen. Some implementations can provide for improved interface with computing and/or other machinery than would be possible with heretofore known techniques. In some implementations, a richer human—machine interface experience can be provided.
- Other aspects and advantages of the present technology can be seen on review of the drawings, the detailed description and the claims, which follow.
-
FIG. 1 shows an example motion sensory and imaging device. -
FIG. 2 shows an example sensory system based upon a motion sensory and imaging device. -
FIG. 3 shows a simplified block diagram of a computer system. -
FIG. 4 shows basic operations and functional units involved in motion capture and image analysis. -
FIG. 5 shows an example augmented reality presented by a motion sensory and imaging capable device. -
FIG. 1 shows an example motionsensory device 100, which includes a main board 182 (first portion) couplable to an illumination board 172 (second portion) with threaded fasteners or otherwise. Electrical interconnection betweenillumination board 172 andmain board 182 permits exchange of signals and power flow and can be made by board mounted connectors, cabling or combinations thereof. One or more fasteners can fasten thedevice 100 to a mounting surface A of a host device (e.g., a vehicle (with or without a heads-up display (HUD), an appliance or portable electronic device (such as a smartphone or tablet computer) or a wearable device (such as a head mounted display (HMD), headsets, or the like) that interfaces with a vehicle or otherwise). The mounting surface A can be any surface (internal or external) of the host device. Alternatively, the device can be disposed within a cavity or receptacle of a host device using a friction fit, fasteners or any combination thereof.Device 100 can be embedded in any of a variety of equipment to meet design requirements of a wide variety of applications. - The
illumination board 172 has a number of individuallycontrollable illumination sources cameras main board 182 ofdevice 100 in the illustrated implementation. Themain board 182 may also include a processor conducting basic image processing, control of thecameras sources - Stereoscopic imaging information provided by
cameras device 100 can provide live real time or near real time image information from the cameras, real time or near real time imaging information augmented by computer generated graphics, information, icons or other virtualized presentations, virtualized representations of the scene being viewed, and/or time varying combinations selected therefrom. Gestures made by a user are sensed by thecameras sensory device 100, and the resulting imaging information can be provided to a motion capture or image analysis system to identify and determine commands to a system. Advantageously, integrating scanning with imaging capabilities into a single motionsensory device 100 provides a highly functional, flexible, yet compact device suited to installation in machines with limited space, such as, e.g., vehicles, appliances, portable or wearable electronic devices, and so forth. - Some of the
illumination sources board 172 at the sides) have focusing lenses, and ten additional LEDs 110 (which are arranged in columns of two, three, three, and two LEDs, respectively) are without focusing lenses. Theboard 172 may also include asocket 178 for coupling a photo-detector (or other sensor). Information from a photo-detector sensing changes in reflectance indicating presence or absence of objects within a region of space into which the illumination sources 108, 110 emit light during a “scanning” of the region of space. - Various modifications of the design shown in
FIG. 1 are possible; for example the number and arrangement of LEDs, photo-detectors, and cameras may vary, the illumination and imaging hardware may be integrated on a single board, other modifications or combinations thereof depending upon the requirements of a particular application. - Now with reference to
FIG. 2 , an examplesensory system 200 deployed in an automobile cabin to recognize gestures is shown.System 200 is preferably implemented within avehicle cabin 201 either in the front seat area, rear seat area, other placements or combinations. As shown inFIG. 2 , all or portions ofsensory system 200 can be affixed to or embedded withindashboard 216, console between front seat passengers, e.g., sensor 200-1, the cabin roof, e.g., sensor 200-2, the backs of the front seats, e.g., sensor 200-3, other locations, or combinations thereof.System 200 includes any number ofcameras analysis system 206.Cameras - In some implementations,
sensory system 200 is capable of separating information received from pixels ofcameras sensory system 200 can provide any of gesture recognition, a real world presentation of real world objects via pass through video feed, and/or an augmented reality including virtual objects integrated with a real world view. -
Cameras cameras cameras hand 214 is expected to move. In this location, the amount of information recorded about the hand is proportional to the number of pixels it occupies in the camera images, and the hand will occupy more pixels when the camera's angle with respect to the hand's “pointing direction” is as close to perpendicular as possible. In an alternative implementation, shown by sensor 200-3, the sensor is disposed along the motion detected, e.g., where thehand 214 is expected to move. - In some implementations, the one or
more sources interest 212 in which one or more portions of the operator (or occupant's) body in this example a hand 214) that may optionally hold a tool or other object of interest andcameras region 212 to capture video images of thehand 214. The operation oflight sources cameras analysis system 206 which can be a computer system, control logic implemented in hardware and/or software or combinations thereof. Based on the captured images, sensory-analysis system 206 determines the position and/or motion ofobject 214. - In one implementation, the
sources cameras system 200 to operate under a broad range of lighting conditions and can avoid various inconveniences or distractions that may be associated with directing visible light into the region where the person is moving. However, a particular wavelength or region of the electromagnetic spectrum need not be required. In one implementation, filters 221, 222 are placed in front ofcameras sources cameras cameras system 200 to varying ambient conditions. - In another implementation, one or more
sonic transducers cameras light sources hand 214. - The illustrated
system 200 can include any of various other sensors not shown inFIG. 2 for clarity, alone or in various combinations, to enhance the experience provided to the operator ofvehicle 201. For example, in low-light situations where free-form gestures cannot be recognized optically with a sufficient degree of reliability,system 206 may switch to a touch mode in which touch gestures are recognized based on acoustic or vibrational sensors. Alternatively,system 206 may switch to the touch mode, or supplement image capture and processing with touch sensing, when signals from acoustic or vibrational sensors are sensed. In still another operational mode, a tap or touch gesture may act as a “wake up” signal to bring the image andaudio analysis system 206 from a standby mode to an operational mode. For example, thesystem 206 may enter the standby mode if optical signals from thecameras - It should be stressed that the arrangement shown in
FIG. 2 is representative and not limiting. For example, lasers or other light sources can be used instead of LEDs. Visible light (RGB) can be used instead of infrared (IR) light. Further, information from one type of light source can be used to clarify, supplement, confirm or reject results from analyzing information gathered using a second type of light source. In implementations that include laser(s), additional optics (e.g., a lens or diffuser) may be employed to widen the laser beam (and make its field of view similar to that of the cameras). Useful arrangements can also include short- and wide-angle illuminators for different ranges. Light sources are typically diffuse rather than specular point sources; for example, packaged LEDs with light-spreading encapsulation are suitable. -
FIG. 3 is a simplified block diagram of acomputer system 300, for implementing all or portions of sensory-analysis system 206 (also referred to as an image analyzer) according to an implementation of the technology disclosed. Sensory-analysis system 206 can include or consist of any device or device component that is capable of capturing and processing image data. In some implementations,computer system 300 includes aprocessor 342,memory 334, asensor interface 336, a presentation I/F 238 which can include a display or other presentation mechanism(s), (e.g. holographic projection systems, heads up displays (HUDs), wearable googles or other head mounted displays (HMDs) for occupant/rear seat implantations, other visual presentation mechanisms or combinations thereof),speakers 339, anmicrophones 340, and anoptional wireless interface 341.Memory 334 can be used to store instructions to be executed byprocessor 342 as well as input and/or output data associated with execution of the instructions. In particular,memory 334 contains instructions, conceptually illustrated as a group of modules described in greater detail below, that control the operation ofprocessor 342 and its interaction with the other hardware components. An operating system directs the execution of low-level, basic system functions such as memory allocation, file management and operation of mass storage devices. The operating system may be or include a variety of operating systems such as Microsoft WINDOWS operating system, the Unix operating system, the Linux operating system, the Xenix operating system, the IBM AIX operating system, the Hewlett Packard UX operating system, the Novell NETWARE operating system, the Sun Microsystems SOLARIS operating system, the OS/2 operating system, the BeOS operating system, the MAC OS operating system, the APACHE operating system, an OPENACTION operating system, iOS, Android or other mobile operating systems, or another operating system or platform. - The computing environment can also include other removable/non-removable, volatile/nonvolatile computer storage media. For example, a hard disk drive can read or write to non-removable, nonvolatile magnetic media. A magnetic disk drive can read from or write to a removable, nonvolatile magnetic disk, and an optical disk drive can read from or write to a removable, nonvolatile optical disk such as a CD-ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The storage media are typically connected to the system bus through a removable or non-removable memory interface.
-
Processor 342 can be a general-purpose microprocessor, but depending on implementation can alternatively be a microcontroller, peripheral integrated circuit element, a CSIC (customer-specific integrated circuit), an ASIC (application-specific integrated circuit), a logic circuit, a digital signal processor, a programmable logic device such as an FPGA (field-programmable gate array), a PLD (programmable logic device), a PLA (programmable logic array), an RFID processor, smart chip, or any other device or arrangement of devices that is capable of implementing the actions of the processes of the technology disclosed. -
Sensor interface 336 can include hardware, firmware and/or software that enables communication betweencomputer system 300 andcameras FIG. 1 , as well as associatedlight sources sensor interface 336 can include one ormore data ports cameras program 344 executing onprocessor 342. In some implementations,sensor interface 336 can also transmit signals to the cameras, e.g., to activate or deactivate the cameras, to control camera settings (frame rate, image quality, sensitivity, etc.), and/or other optional sensors to control sensor settings (calibration, sensitivity levels, etc.), or the like. Such signals can be transmitted, e.g., in response to control signals fromprocessor 342, which can in turn be generated in response to user input or other detected events. -
Sensor interface 336 can also includecontrollers light sources 108, 110) can be connected. In some implementations,controllers processor 342 executingmocap program 344. In other implementations, the light sources can draw operating current from an external power supply, andcontrollers - Instructions defining
mocap program 344 are stored inmemory 334, and these instructions, when executed, perform motion-capture analysis on images supplied from cameras connected tosensor interface 336. In one implementation,mocap program 344 includes various modules, such as anobject detection module 352, anobject analysis module 354, and a gesture-recognition module 356.Object detection module 352 can analyze images (e.g., images captured via sensor interface 336) to detect edges of an object therein and/or other information about the object's location.Object analysis module 354 can analyze the object information provided byobject detection module 352 to determine the 3D position and/or motion of the object (e.g., a user's hand). In some implementations,object analysis module 354 can also analyze audio signals (e.g., audio signals captured via interface 336) to localize the object by, for example, time distance of arrival, multilateration or the like. (“Multilateration is a navigation technique based on the measurement of the difference in distance to two or more stations at known locations that broadcast signals at known times. See Wikipedia, at http://en.wikipedia.org/w/index.php?title=Multilateration&oldid=523281858, on Nov. 16, 2012, 06:07 UTC). Examples of operations that can be implemented in code modules ofmocap program 344 are described below.Memory 334 can also include other information and/or code modules used bymocap program 344 such as anapplication platform 366 that allows a user to interact with themocap program 344 using different applications like application 1 (App1), application 2 (App2), and application N (AppN). - Presentation I/
F 238,speakers 339,microphones 340, andoptional wireless interface 341 can be used to facilitate user or system interaction withcomputer system 300. In some implementations, results of gesture capture usingsensor interface 336 andmocap program 344 can be interpreted as user input. For example, a user can perform hand gestures that are analyzed usingmocap program 344, and the results of this analysis can be interpreted as an instruction to some other program executing on processor 342 (e.g., a web browser, GPS application, dictation program, or other application). Thus, by way of illustration, a user might use upward or downward swiping gestures to “scroll” a webpage currently displayed via presentation I/F 238, to use rotating gestures to increase or decrease the volume of audio output fromspeakers 339, and so on. - It will be appreciated that
computer system 300 is illustrative and that variations and modifications are possible. A particular implementation can include other functionality not described herein, e.g., wired and/or wireless network interfaces, media playing and/or recording system interfaces, etc. In some implementations, one or more cameras can be built into the vehicle or equipment into which thesensor 200 is imbedded rather than being supplied as separate components. Further, an image analyzer can be implemented using only a subset of computer system components (e.g., as a processor executing program code, an ASIC, or a fixed-function digital signal processor, with suitable I/O interfaces to receive image data and output analysis results). - While
computer system 300 is described herein with reference to particular blocks, it is to be understood that the blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of component parts. Further, the blocks need not correspond to physically distinct components. To the extent that physically distinct components are used, connections between components (e.g., for data communication) can be wired and/or wireless as desired. - With reference to
FIGS. 2 and 3 , the user (e.g., vehicle operator, occupant(s)) performs a gesture that is captured by thecameras cameras cameras cameras hand 214 can be distinguished byobject analysis module 354 and analyzed by a gesture-recognition module 356, of themocap 344. Gesture-recognition module 356 provides input to one or more systems or subsystems of the vehicle or other controlled equipment, allowing a user to remotely control those systems or subsystems and/or virtually manipulate objects, such as virtualized gauges, indicators, buttons, levers, or other controls, in a virtual environment displayed via presentation I/F 238. The user can perform the gesture using any part of her body, such as a finger, a hand, or an arm or combinations thereof. As part of gesture recognition or independently, the sensory-analysis system 206 can determine the shapes and positions of the user's hand in 3D space and in real time using algorithms described below as well as, in more detail, in, e.g., U.S. Ser. No. 61/587,554, Ser. Nos. 13/414,485, 61/724,091, 13/724,357, and 13/742,953, filed on Jan. 17, 2012, Mar. 7, 2012, Nov. 8, 2012, Dec. 21, 2012 and Jan. 16, 2013, respectively, the entire disclosures of which are hereby incorporated by reference. - As a result, the sensory-
analysis system 206 can not only recognize gestures for purposes of providing input to the electronic device, but can also capture the position and shape of the user's hand in consecutive video images in order to characterize the hand gesture in 3D space and reproduce it on a display screen, for example via presentation I/F 238, as arigged hand 99 for example.Rigged hand 99 is determined frommodel hand 98 that includes arigged hand overlay 94 covering one or more capsule elements 97 built from the images by theobject detection module 352. - In one implementation, and with reference to
FIG. 5 , arigged hand representation 599 can be projected as part of an augmented vehicle environment onto a (semi-)transparent or other surface through which one or more real objects can be viewed also, e.g., awindshield 516 according to one implementation of the technology disclosed. For example, with continuing reference toFIG. 5 , avideo projector 504, coupled to presentation I/F 238, can project an image of an arrow (e.g., virtual device 523) from a GPS application superimposed upon a (semi-) transparent surface, e.g.,windshield 516, being viewed directly by the user (or displayed to a user via live video feed in the event the operator is remote (e.g., operating a drone or a driverless vehicle) from the vehicle or the vehicle lacks a windshield); thereby creating an augmented environment experience when operating the vehicle. Optionalhaptic projector 506 can project the feeling of the pressure of clicking or pressing a physical control. Optionalaudio projector 502 can project the sounds of virtual “speed bumps” indicating the driver of a car should pay attention to the change in route, an oncoming obstacle or the speed of the vehicle. - In one implementation, the gesture-
recognition module 356 compares one or more primitives of the detected gesture to a library of gesture primitives electronically stored as records in a database, which is implemented in the sensory-analysis system 206, the electronic device, or on an external storage system. (As used herein, the term “electronically stored” includes storage in volatile or non-volatile storage, the latter including disks, Flash memory, etc., and extends to any computationally addressable storage media (including, for example, optical storage).) For example, gestures can be stored as vectors, i.e., mathematically specified spatial trajectories, other primitives or combinations thereof, and the gesture record can have a field specifying the relevant part of the user's body making the gesture; thus, similar trajectories executed by a user's hand and head can be stored in the database as different gestures so that an application can interpret them differently. -
FIG. 4 depicts the basic operations andfunctional units 400 involved in motion capture and image analysis in accordance with implementations of the technology disclosed. As shown inFIG. 4 , the camera(s) 402, 404 recorddigital images 410 of a scene. Each digital image is captured as an array of pixel values by the associated camera's image sensor, and the digital images are transferred—either in “raw” format or following conventional preprocessing—to one ormore frame buffers 415. A frame buffer is a partition or dedicated segment of volatile memory that stores a “bitmapped”image frame 420 corresponding to the pixel values of an image as output by the camera that recorded it. The bitmap is generally organized conceptually as a grid, with each pixel mapped one-to-one or otherwise to output elements of a display. It should be stressed, however, that the topology of how memory cells are physically organized within the frame buffers 415 does not matter and need not conform directly to the conceptual organization. - The number of frame buffers included in a system generally reflects the number of images simultaneously analyzed by the analysis system or
module 430, which is described in greater detail below. Briefly,analysis module 430 analyzes the pixel data in each of a sequence of image frames 420 to locate objects therein and track their movement over time (as indicated at 440). This analysis can take various forms, and the algorithm performing the analysis dictates how pixels in the image frames 420 are handled. For example, the algorithm implemented byanalysis module 430 can process the pixels of each frame buffer on a line-by-line basis—i.e., each row of the pixel grid is successively analyzed. Other algorithms can analyze pixels in columns, tiled areas, or other organizational formats. - In various implementations, the motion captured in a series of camera images is used to compute a corresponding series of output images for display via the presentation I/
F 238. For example, camera images of a moving hand can be translated into a wire-frame or other graphic depiction of the hand by theprocessor 342. Alternatively, hand gestures can be interpreted as input used to control a separate visual output; by way of illustration, a user can be able to use upward or downward swiping gestures to “scroll” a webpage or other document currently displayed, or open and close her hand to zoom in and out of the page. In any case, the output images are generally stored in the form of pixel data in a frame buffer, e.g., one of the frame buffers 415. A video display controller reads out the frame buffer to generate a data stream and associated control signals to output the images via the presentation I/F 238. The video display controller can be provided along with theprocessor 342 andmemory 334 on-board the motherboard of thecomputer 300, and can be integrated with theprocessor 342 or implemented as a co-processor that manipulates a separate video memory. As noted, thecomputer 300 can be equipped with a separate graphics or video card that aids with generating the feed of output images for the presentation I/F 238. The video card generally includes a graphics processing unit (GPU) and video memory, and is useful, in particular, for complex and computationally expensive image processing and rendering. The graphics card can include the frame buffer and the functionality of the video display controller (and the on-board video display controller can be disabled). In general, the image-processing and motion-capture functionality of the system can be distributed between the GPU and themain processor 342 in various ways. - Suitable algorithms for motion-
capture program 344 are described below as well as, in more detail, in U.S. Ser. No. 61/587,554, Ser. Nos. 13/414,485, 61/724,091, 13/724,357, and 13/742,953, filed on Jan. 17, 2012, Mar. 7, 2012, Nov. 8, 2012, Dec. 21, 2012 and Jan. 16, 2013, respectively, which are hereby incorporated herein by reference in their entirety. The various modules can be programmed in any suitable programming language, including, without limitation high-level languages such as C, C++, C #, OpenGL, Ada, Basic, Cobra, FORTRAN, Java, Lisp, Perl, Python, Ruby, or Object Pascal, or low-level assembly languages. - Again with reference to
FIG. 4 , the modes of operation of a device equipped with a motion sensory control device can determine the coarseness of the data provided to the image-analysis module 430, the coarseness of its analysis, or both in accordance with entries in a performance database. For example, during a wide-area mode of operation, the image-analysis module 430 can operate on every image frame and on all data within a frame, capacity limitations can dictate analysis of a reduced amount of image data per frame (i.e., resolution) or discarding of some frames altogether if the data in each of theframe buffers 415 are organized as a sequence of data lines. The manner in which data is dropped from the analysis can depend on the image-analysis algorithm or the uses to which the motion-capture output is put. In some implementations, data is dropped in a symmetric or uniform fashion—e.g., every other line, every third line, etc. is discarded up to a tolerance limit of the image-analysis algorithm or an application utilizing its output. In other implementations, the frequency of line dropping can increase toward the edges of the frame. Still other image-acquisition parameters that can be varied include the frame size, the frame resolution, and the number of frames acquired per second. In particular, the frame size can be reduced by, e.g., discarding edge pixels or by resampling to a lower resolution (and utilizing only a portion of the frame buffer capacity). Parameters relevant to acquisition of image data (e.g., size and frame rate and characteristics) are collectively referred to as “acquisition parameters,” while parameters relevant to operation of the image-analysis module 430 (e.g., in defining the contour of an object) are collectively referred to as “image-analysis parameters.” The foregoing examples of acquisition parameters and image-analysis parameters are representative only, and not limiting. - Acquisition parameters can be applied to the
cameras camera cameras analysis module 430 as numerical quantities that affect the operation of the contour-defining algorithm. - The desirable values for acquisition parameters and image-analysis parameters appropriate to a given level of available resources can depend, for example, on the characteristics of the image-
analysis module 430, the nature of the application utilizing the mocap output, and design preferences. Whereas some image-processing algorithms can be able to trade off a resolution of contour approximation against input frame resolution over a wide range, other algorithms may not exhibit much tolerance at all—requiring, for example, a minimal image resolution below which the algorithm fails altogether. - In one implementation, and with reference to
FIG. 5 , which illustrates asystem 500 for projecting anaugmented vehicle environment 501 onto a (semi-)transparent or other surface through which one or more real objects can be viewed also, e.g., awindshield 516 according to one implementation of the technology disclosed.System 500 includes asensory processing system 206 controlling a variety of sensors and projectors, such as for example one ormore cameras 102, 104 (or other image sensors) and optionally someillumination sources sensors FIG. 5 for the sake of clarity) can be included. Optionally projectors under control ofsystem 206 can render virtual objects to provide the augmentedvehicle environment 501, such as anoptional audio projector 502 to provide audio feedback (e.g., “clicks”, chimes, speech feedback when virtual buttons are pressed or virtual controls are manipulated),optional video projector 504, an optionalhaptic projector 506 to provide haptic feedback (e.g., button pressure, pressure indicating a physical object has been grasped, etc.) to a user ofaugmented vehicle environment 501. For further information on projectors, reference may be had to “Visio-Tactile Projector” Youtube (https://www.youtube.com/watch?v=Bb0hNMxxewg) (accessed Jan. 15, 2014). In operation, sensors and projectors ofsystem 500 can be incorporated wholly or partly into sensor device 500-1, or otherwise conveniently disposed toward a region ofinterest 112 in order to sense motion and presence ofhand 114, that moves along the indicatedpath 118 to give commands to thesystem 500 by gesturing. One ormore applications augmented reality 113. Accordingly, user (e.g., owner of hand 114) is able to view real objects e.g., throughwindshield 516, outputs ofapplications augmented reality 113, in thesame environment 501. - Projection can include an image or other visual representation of the user's
hand 599 and/or one or more optional objects. Objects can include: objects associated with anapplication vehicle 521, advertising objects 517, objects representing more abstract things, other types of objects, and combination objects. For example,visual projection mechanism 504 ofFIG. 5 can project anicon 522 of a person from the operator's contact list. A GPS application can plan route to the selected location for the person in the contact list and provideguidance information 523, also projected as virtualized objects into augmentedvehicle environment 501, to the operator.Various objects 521 associated with status monitors provide information about the vehicle's systems. An advertising application searches previous destinations of trips made in close proximity to previous trips to visitcontact 522 and displays anappropriate advertising object 517. For example, if the operator visits a friend (contact 522) and they've gone to Burger King the last three times, an advertising icon for McDonalds may appear. A vehicle “happiness”icon 524 displays an overall level of happiness of the occupants determined from a variety of parameters (e.g., frequency of changes in environmental controls, frequency of change in entertainment selections, amounts of ambient light, temperature, humidity, etc.). - Alternatively,
surface 516 can be a wearable computing device such as Google Glass™ or equivalent connectable wirelessly or by wire tosensory system 500. - Projections for augmented vehicle environments can be differentiated for front and rear seat passengers in an automobile for example. Front seat passengers can experience clicks, chimes and/or speech feedback responsive to the occupant's gestures. Rear seat passengers can experience clicks, chimes and/or speech feedback on separate audio channel to headphones or HMDs used by the rear seat passengers (to avoid distracting driver).
- Alternatively, in a driverless automobile implementation, the “driver” no longer drives vehicle. There is not as significant a price for distracting the “driver”. In one such implementation, gestures can be expanded for all front seat passengers to control vehicle (sub)systems. Driverless vehicles can include a larger more interactive HUD (up to whole windshield). Gestures control non-safety related navigation decisions (e.g., override determined routing, waypoints on moving map display, choosing rest stops for purposes of rerouting (e.g., bathroom breaks), and so forth).
Claims (14)
1. A motion sensory control system, including:
a controller coupled to imaging sensors and one or more illumination sources, the controller being configured to acquire imaging information of a scene comprising a body part of an occupant of a vehicle;
wherein the controller is further configured to interact with an augmented reality system providing a display within the vehicle, such that a graphical object representing the body part of the occupant is displayed by the display, such that the graphical object appears to be interacting with other objects displayed by display, and
wherein the controller is further configured to recognize gestures executed by the body part by comparing a recognized gesture to one or more gestures stored in a reference library of gesture primitives and to interpret the recognized gesture as a stored command, and
wherein the controller is further configured to interact with the augmented reality system to display an augmented reality presentation by the display showing a second graphical object associated with the stored command.
2. The system of claim 1 , wherein
the controller is further configured to acquire imaging information for control objects within view of the imaging sensors, and the imaging information for control objects is used to determine gesture information indicating a command to a machine under control.
3. The system of claim 1 , wherein the controller further provides:
storage for the reference library of gesture primitives as electronic records in a database.
4. The system of claim 2 , wherein the acquiring of imaging information for control objects further includes:
separating information received from pixels sensitive to IR light from information received from pixels sensitive to visible light;
processing image information from IR sensors to be used for gesture recognition; and
processing image information from RGB sensors to be provided as a video feed via a presentation interface.
5. The system of claim 4 , wherein the processing of image information from RGB sensors further includes:
extracting features of a region of interest using RGB pixels that respectively capture red, green, and blue components of illumination in the scene.
6. The system of claim 4 , wherein the processing of image information from IR sensors further includes:
extracting features of a region of interest using IR pixels that capture infrared components of illumination in the scene.
7. The system of claim 1 , further including:
a port to receive information from a photodetector configured to sense changes in light intensity indicating presence or absence of objects within a region of space illuminated by the illumination sources.
8. The system of claim 1 ,
wherein the controller is configured to interact with the augmented reality system to present a presentation interface displaying an augmented reality presentation on the display, wherein the presentation interface is produced, in part, from images captured by the imaging sensors.
9. The system of claim 8 , wherein the augmented reality presentation further includes a sequence of images of a scene with computer generated information.
10. The system of claim 9 , wherein the controller is further configured to:
determine destination information from at least one image of the sequence of images and information entered or used by an operator;
select an icon representing an advertisement relevant to the destination information; and
interact with the augmented reality system to display the icon representing the advertisement in the augmented reality presentation by the display.
11. A method for sensing motion within a vehicle, comprising:
illuminating, with illumination sources, a portion of a vehicle, the vehicle being provided with a controller coupled to imaging sensors;
acquiring imaging information of a scene comprising a body part of an occupant of the vehicle;
interacting with an augmented reality system providing a display within the vehicle, such that a graphical object representing the body part of the occupant is displayed by the display, such that the graphical object appears to be interacting with other objects displayed by display;
recognizing gestures executed by the body part by comparing a recognized gesture to one or more gestures stored in a reference library of gesture primitives and to interpret the recognized gesture as a stored command; and
interacting with the augmented reality system to display an augmented reality presentation by the display showing a second graphical object associated with the stored command.
12. The method of claim 11 , further comprising:
acquiring imaging information for control objects within view of the imaging sensors; and
using the imaging information for control objects to determine gesture information indicating a command to a machine under control.
13. The method of claim 11 , further comprising:
receiving information from a photodetector configured to sense changes in light intensity indicating presence or absence of objects within a region of space illuminated by the illumination sources.
14. The method of claim 11 , further comprising:
determining destination information from the imaging information and information entered or used by an operator;
selecting an icon representing an advertisement relevant to the destination information; and
interacting with the augmented reality system to display the icon representing the advertisement in the augmented reality presentation by the display.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/225,598 US20230368577A1 (en) | 2014-08-15 | 2023-07-24 | Automotive and industrial motion sensory device |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201462038112P | 2014-08-15 | 2014-08-15 | |
US14/826,102 US11386711B2 (en) | 2014-08-15 | 2015-08-13 | Automotive and industrial motion sensory device |
US17/848,181 US11749026B2 (en) | 2014-08-15 | 2022-06-23 | Automotive and industrial motion sensory device |
US18/225,598 US20230368577A1 (en) | 2014-08-15 | 2023-07-24 | Automotive and industrial motion sensory device |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/848,181 Continuation US11749026B2 (en) | 2014-08-15 | 2022-06-23 | Automotive and industrial motion sensory device |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230368577A1 true US20230368577A1 (en) | 2023-11-16 |
Family
ID=51946162
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/826,102 Active US11386711B2 (en) | 2014-08-15 | 2015-08-13 | Automotive and industrial motion sensory device |
US17/848,181 Active US11749026B2 (en) | 2014-08-15 | 2022-06-23 | Automotive and industrial motion sensory device |
US18/225,598 Pending US20230368577A1 (en) | 2014-08-15 | 2023-07-24 | Automotive and industrial motion sensory device |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/826,102 Active US11386711B2 (en) | 2014-08-15 | 2015-08-13 | Automotive and industrial motion sensory device |
US17/848,181 Active US11749026B2 (en) | 2014-08-15 | 2022-06-23 | Automotive and industrial motion sensory device |
Country Status (4)
Country | Link |
---|---|
US (3) | US11386711B2 (en) |
JP (1) | JP3194297U (en) |
CN (1) | CN204537054U (en) |
DE (1) | DE202014104297U1 (en) |
Families Citing this family (74)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8908995B2 (en) | 2009-01-12 | 2014-12-09 | Intermec Ip Corp. | Semi-automatic dimensioning with imager on a portable device |
US9779546B2 (en) | 2012-05-04 | 2017-10-03 | Intermec Ip Corp. | Volume dimensioning systems and methods |
US10007858B2 (en) | 2012-05-15 | 2018-06-26 | Honeywell International Inc. | Terminals and methods for dimensioning objects |
US10321127B2 (en) | 2012-08-20 | 2019-06-11 | Intermec Ip Corp. | Volume dimensioning system calibration systems and methods |
US9939259B2 (en) | 2012-10-04 | 2018-04-10 | Hand Held Products, Inc. | Measuring object dimensions using mobile computer |
US9841311B2 (en) | 2012-10-16 | 2017-12-12 | Hand Held Products, Inc. | Dimensioning system |
US9080856B2 (en) | 2013-03-13 | 2015-07-14 | Intermec Ip Corp. | Systems and methods for enhancing dimensioning, for example volume dimensioning |
US10228452B2 (en) | 2013-06-07 | 2019-03-12 | Hand Held Products, Inc. | Method of error correction for 3D imaging device |
US9823059B2 (en) | 2014-08-06 | 2017-11-21 | Hand Held Products, Inc. | Dimensioning system with guided alignment |
JP3194297U (en) | 2014-08-15 | 2014-11-13 | リープ モーション, インコーポレーテッドLeap Motion, Inc. | Motion sensing control device for automobile and industrial use |
US10810715B2 (en) | 2014-10-10 | 2020-10-20 | Hand Held Products, Inc | System and method for picking validation |
US10775165B2 (en) | 2014-10-10 | 2020-09-15 | Hand Held Products, Inc. | Methods for improving the accuracy of dimensioning-system measurements |
US9779276B2 (en) | 2014-10-10 | 2017-10-03 | Hand Held Products, Inc. | Depth sensor based auto-focus system for an indicia scanner |
US10060729B2 (en) | 2014-10-21 | 2018-08-28 | Hand Held Products, Inc. | Handheld dimensioner with data-quality indication |
US9762793B2 (en) | 2014-10-21 | 2017-09-12 | Hand Held Products, Inc. | System and method for dimensioning |
US9897434B2 (en) | 2014-10-21 | 2018-02-20 | Hand Held Products, Inc. | Handheld dimensioning system with measurement-conformance feedback |
US9752864B2 (en) | 2014-10-21 | 2017-09-05 | Hand Held Products, Inc. | Handheld dimensioning system with feedback |
US9547373B2 (en) * | 2015-03-16 | 2017-01-17 | Thunder Power Hong Kong Ltd. | Vehicle operating system using motion capture |
US9539988B2 (en) | 2015-03-16 | 2017-01-10 | Thunder Power Hong Kong Ltd. | Vehicle camera cleaning system |
US10703211B2 (en) | 2015-03-16 | 2020-07-07 | Thunder Power New Energy Vehicle Development Company Limited | Battery pack, battery charging station, and charging method |
US10173687B2 (en) | 2015-03-16 | 2019-01-08 | Wellen Sham | Method for recognizing vehicle driver and determining whether driver can start vehicle |
US9550406B2 (en) * | 2015-03-16 | 2017-01-24 | Thunder Power Hong Kong Ltd. | Thermal dissipation system of an electric vehicle |
US9954260B2 (en) | 2015-03-16 | 2018-04-24 | Thunder Power New Energy Vehicle Development Company Limited | Battery system with heat exchange device |
US9533551B2 (en) | 2015-03-16 | 2017-01-03 | Thunder Power Hong Kong Ltd. | Electric vehicle thermal management system with series and parallel structure |
US9979890B2 (en) | 2015-04-23 | 2018-05-22 | Apple Inc. | Digital viewfinder user interface for multiple cameras |
EP3091510B1 (en) * | 2015-05-06 | 2021-07-07 | Reactive Reality AG | Method and system for producing output images |
US9786101B2 (en) | 2015-05-19 | 2017-10-10 | Hand Held Products, Inc. | Evaluating image values |
US9994317B2 (en) * | 2015-06-02 | 2018-06-12 | Airbus Group India Private Limited | Aviation mask |
US10066982B2 (en) | 2015-06-16 | 2018-09-04 | Hand Held Products, Inc. | Calibrating a volume dimensioner |
US20160377414A1 (en) | 2015-06-23 | 2016-12-29 | Hand Held Products, Inc. | Optical pattern projector |
US9835486B2 (en) | 2015-07-07 | 2017-12-05 | Hand Held Products, Inc. | Mobile dimensioner apparatus for use in commerce |
EP3396313B1 (en) | 2015-07-15 | 2020-10-21 | Hand Held Products, Inc. | Mobile dimensioning method and device with dynamic accuracy compatible with nist standard |
US10094650B2 (en) | 2015-07-16 | 2018-10-09 | Hand Held Products, Inc. | Dimensioning and imaging items |
US20170017301A1 (en) | 2015-07-16 | 2017-01-19 | Hand Held Products, Inc. | Adjusting dimensioning results using augmented reality |
US20180091733A1 (en) * | 2015-07-31 | 2018-03-29 | Hewlett-Packard Development Company, L.P. | Capturing images provided by users |
TWI578021B (en) * | 2015-08-19 | 2017-04-11 | 國立臺北科技大學 | Augmented reality interactive system and dynamic information interactive and display method thereof |
EP3138607A1 (en) * | 2015-09-04 | 2017-03-08 | Airbus Group India Private Limited | Aviation mask |
US10249030B2 (en) | 2015-10-30 | 2019-04-02 | Hand Held Products, Inc. | Image transformation for indicia reading |
US10225544B2 (en) | 2015-11-19 | 2019-03-05 | Hand Held Products, Inc. | High resolution dot pattern |
US10025314B2 (en) * | 2016-01-27 | 2018-07-17 | Hand Held Products, Inc. | Vehicle positioning and object avoidance |
DE102016204027A1 (en) | 2016-03-11 | 2017-09-14 | Robert Bosch Gmbh | Method and device for controlling a propulsion unit of an electric two-wheeler |
US10339352B2 (en) | 2016-06-03 | 2019-07-02 | Hand Held Products, Inc. | Wearable metrological apparatus |
US9940721B2 (en) | 2016-06-10 | 2018-04-10 | Hand Held Products, Inc. | Scene change detection in a dimensioner |
US10009536B2 (en) | 2016-06-12 | 2018-06-26 | Apple Inc. | Applying a simulated optical effect based on data received from multiple camera sensors |
US10163216B2 (en) | 2016-06-15 | 2018-12-25 | Hand Held Products, Inc. | Automatic mode switching in a volume dimensioner |
CN106249882B (en) | 2016-07-26 | 2022-07-12 | 华为技术有限公司 | Gesture control method and device applied to VR equipment |
US9996980B1 (en) * | 2016-11-18 | 2018-06-12 | Toyota Jidosha Kabushiki Kaisha | Augmented reality for providing vehicle functionality through virtual features |
CA3041958A1 (en) * | 2016-11-25 | 2018-05-31 | Sensoryx AG | Wearable motion tracking system |
US10909708B2 (en) | 2016-12-09 | 2021-02-02 | Hand Held Products, Inc. | Calibrating a dimensioner using ratios of measurable parameters of optic ally-perceptible geometric elements |
US11047672B2 (en) | 2017-03-28 | 2021-06-29 | Hand Held Products, Inc. | System for optically dimensioning |
CN107102731A (en) * | 2017-03-31 | 2017-08-29 | 斑马信息科技有限公司 | Gestural control method and its system for vehicle |
DK180859B1 (en) | 2017-06-04 | 2022-05-23 | Apple Inc | USER INTERFACE CAMERA EFFECTS |
US11587419B2 (en) * | 2017-08-04 | 2023-02-21 | Toyota Research Institute, Inc. | Methods and systems providing an intelligent camera system |
US20190073040A1 (en) * | 2017-09-05 | 2019-03-07 | Future Mobility Corporation Limited | Gesture and motion based control of user interfaces |
US11112964B2 (en) | 2018-02-09 | 2021-09-07 | Apple Inc. | Media capture lock affordance for graphical user interface |
US10584962B2 (en) | 2018-05-01 | 2020-03-10 | Hand Held Products, Inc | System and method for validating physical-item security |
US11722764B2 (en) | 2018-05-07 | 2023-08-08 | Apple Inc. | Creative camera |
US10375313B1 (en) | 2018-05-07 | 2019-08-06 | Apple Inc. | Creative camera |
DK201870623A1 (en) | 2018-09-11 | 2020-04-15 | Apple Inc. | User interfaces for simulated depth effects |
US10645294B1 (en) | 2019-05-06 | 2020-05-05 | Apple Inc. | User interfaces for capturing and managing visual media |
US11770601B2 (en) | 2019-05-06 | 2023-09-26 | Apple Inc. | User interfaces for capturing and managing visual media |
US11321857B2 (en) | 2018-09-28 | 2022-05-03 | Apple Inc. | Displaying and editing images with depth information |
US11128792B2 (en) | 2018-09-28 | 2021-09-21 | Apple Inc. | Capturing and displaying images with multiple focal planes |
TWM579743U (en) * | 2019-02-01 | 2019-06-21 | 帷享科技股份有限公司 | Multi-functional photography system |
DE102019106683B4 (en) * | 2019-03-15 | 2020-12-24 | Technische Universität Dresden | Method and system for establishing contact between a user's hand and an operating element of a device |
CN112887586B (en) * | 2019-05-06 | 2022-05-10 | 苹果公司 | User interface for capturing and managing visual media |
US11706521B2 (en) | 2019-05-06 | 2023-07-18 | Apple Inc. | User interfaces for capturing and managing visual media |
US11816757B1 (en) * | 2019-12-11 | 2023-11-14 | Meta Platforms Technologies, Llc | Device-side capture of data representative of an artificial reality environment |
US11354867B2 (en) * | 2020-03-04 | 2022-06-07 | Apple Inc. | Environment application model |
US11039074B1 (en) | 2020-06-01 | 2021-06-15 | Apple Inc. | User interfaces for managing media |
US11212449B1 (en) | 2020-09-25 | 2021-12-28 | Apple Inc. | User interfaces for media capture and management |
US11539876B2 (en) | 2021-04-30 | 2022-12-27 | Apple Inc. | User interfaces for altering visual media |
US11778339B2 (en) | 2021-04-30 | 2023-10-03 | Apple Inc. | User interfaces for altering visual media |
US12112024B2 (en) | 2021-06-01 | 2024-10-08 | Apple Inc. | User interfaces for managing media styles |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120007821A1 (en) * | 2010-07-11 | 2012-01-12 | Lester F. Ludwig | Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (hdtp) user interfaces |
US20130141375A1 (en) * | 2011-12-06 | 2013-06-06 | Lester F. Ludwig | Gesteme (gesture primitive) recognition for advanced touch user interfaces |
Family Cites Families (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7050606B2 (en) | 1999-08-10 | 2006-05-23 | Cybernet Systems Corporation | Tracking and gesture recognition system particularly suited to vehicular control applications |
US20020140633A1 (en) * | 2000-02-03 | 2002-10-03 | Canesta, Inc. | Method and system to present immersion virtual simulations using three-dimensional measurement |
KR100575906B1 (en) | 2002-10-25 | 2006-05-02 | 미츠비시 후소 트럭 앤드 버스 코포레이션 | Hand pattern switching apparatus |
JP3752246B2 (en) | 2003-08-11 | 2006-03-08 | 学校法人慶應義塾 | Hand pattern switch device |
US8964054B2 (en) | 2006-08-18 | 2015-02-24 | The Invention Science Fund I, Llc | Capturing selected image objects |
JP4115467B2 (en) | 2005-06-01 | 2008-07-09 | 富士フイルム株式会社 | Imaging device |
US7454136B2 (en) | 2005-07-28 | 2008-11-18 | Mitsubishi Electric Research Laboratories, Inc. | Method and apparatus for acquiring HDR flash images |
US8085243B2 (en) * | 2006-02-03 | 2011-12-27 | Panasonic Corporation | Input device and its method |
JP5243070B2 (en) * | 2008-03-14 | 2013-07-24 | ソニー株式会社 | Information providing apparatus, receiving terminal, information providing system, information providing method, and program |
CN102112945B (en) | 2008-06-18 | 2016-08-10 | 奥布隆工业有限公司 | Control system based on attitude for vehicle interface |
US20100013613A1 (en) * | 2008-07-08 | 2010-01-21 | Jonathan Samuel Weston | Haptic feedback projection system |
US8539359B2 (en) * | 2009-02-11 | 2013-09-17 | Jeffrey A. Rapaport | Social network driven indexing system for instantly clustering people with concurrent focus on same topic into on-topic chat rooms and/or for generating on-topic search results tailored to user preferences regarding topic |
US20120041344A1 (en) * | 2009-03-20 | 2012-02-16 | Carl-Erik Flodmark | Apparatus and method for estimating body fat mass |
US8666672B2 (en) * | 2009-11-21 | 2014-03-04 | Radial Comm Research L.L.C. | System and method for interpreting a user's psychological state from sensed biometric information and communicating that state to a social networking site |
US8396252B2 (en) | 2010-05-20 | 2013-03-12 | Edge 3 Technologies | Systems and related methods for three dimensional gesture recognition in vehicles |
US8766162B2 (en) | 2012-01-12 | 2014-07-01 | Maxim Integrated Products, Inc. | Ambient light based gesture detection |
US8693731B2 (en) | 2012-01-17 | 2014-04-08 | Leap Motion, Inc. | Enhanced contrast for object detection and characterization by optical imaging |
US8638989B2 (en) | 2012-01-17 | 2014-01-28 | Leap Motion, Inc. | Systems and methods for capturing motion in three-dimensional space |
US20130204457A1 (en) | 2012-02-06 | 2013-08-08 | Ford Global Technologies, Llc | Interacting with vehicle controls through gesture recognition |
US9389690B2 (en) * | 2012-03-01 | 2016-07-12 | Qualcomm Incorporated | Gesture detection based on information from multiple types of sensors |
US20140309813A1 (en) * | 2013-04-15 | 2014-10-16 | Flextronics Ap, Llc | Guest vehicle user reporting |
US9063574B1 (en) | 2012-03-14 | 2015-06-23 | Amazon Technologies, Inc. | Motion detection systems for electronic devices |
US8942881B2 (en) | 2012-04-02 | 2015-01-27 | Google Inc. | Gesture-based automotive controls |
US9239624B2 (en) | 2012-04-13 | 2016-01-19 | Nokia Technologies Oy | Free hand gesture control of automotive user interface |
US9571818B2 (en) | 2012-06-07 | 2017-02-14 | Nvidia Corporation | Techniques for generating robust stereo images from a pair of corresponding stereo images captured with and without the use of a flash device |
US9671566B2 (en) | 2012-06-11 | 2017-06-06 | Magic Leap, Inc. | Planar waveguide apparatus with diffraction element(s) and system employing same |
WO2014018836A1 (en) * | 2012-07-26 | 2014-01-30 | Leap Motion, Inc. | Object detection and tracking |
WO2014020604A1 (en) * | 2012-07-31 | 2014-02-06 | Inuitive Ltd. | Multiple sensors processing system for natural user interface applications |
US8819812B1 (en) | 2012-08-16 | 2014-08-26 | Amazon Technologies, Inc. | Gesture recognition for device input |
US8982261B2 (en) * | 2012-10-13 | 2015-03-17 | Hewlett-Packard Development Company, L.P. | Imaging with interleaved detection accumulations |
US8781171B2 (en) | 2012-10-24 | 2014-07-15 | Honda Motor Co., Ltd. | Object recognition in low-lux and high-lux conditions |
US20140118257A1 (en) | 2012-10-29 | 2014-05-01 | Amazon Technologies, Inc. | Gesture detection systems |
US9459697B2 (en) | 2013-01-15 | 2016-10-04 | Leap Motion, Inc. | Dynamic, free-space user interactions for machine control |
US8744645B1 (en) | 2013-02-26 | 2014-06-03 | Honda Motor Co., Ltd. | System and method for incorporating gesture and voice recognition into a single system |
US10037474B2 (en) * | 2013-03-15 | 2018-07-31 | Leap Motion, Inc. | Determining the relative locations of multiple motion-tracking devices |
CN105283899A (en) * | 2013-05-23 | 2016-01-27 | 艾弗诺泰普有限责任公司 | Method and system for maintaining or improving wellness |
US9477314B2 (en) | 2013-07-16 | 2016-10-25 | Google Technology Holdings LLC | Method and apparatus for selecting between multiple gesture recognition systems |
US9671826B2 (en) * | 2013-11-27 | 2017-06-06 | Immersion Corporation | Method and apparatus of body-mediated digital content transfer and haptic feedback |
US9613262B2 (en) | 2014-01-15 | 2017-04-04 | Leap Motion, Inc. | Object detection and tracking for providing a virtual device experience |
US9348419B2 (en) | 2014-01-24 | 2016-05-24 | Leap Motion, Inc. | Method for synchronizing operation of systems |
US10627904B2 (en) | 2014-02-07 | 2020-04-21 | Ultrahaptics IP Two Limited | Systems and methods of determining interaction intent in three-dimensional (3D) sensory space |
US9495762B2 (en) | 2014-05-14 | 2016-11-15 | Qualcomm Incorporated | Detecting and compensating for motion between a flash and a no-flash image |
US9868449B1 (en) | 2014-05-30 | 2018-01-16 | Leap Motion, Inc. | Recognizing in-air gestures of a control object to control a vehicular control system |
JP3194297U (en) | 2014-08-15 | 2014-11-13 | リープ モーション, インコーポレーテッドLeap Motion, Inc. | Motion sensing control device for automobile and industrial use |
US10120413B2 (en) * | 2014-09-11 | 2018-11-06 | Interaxon Inc. | System and method for enhanced training using a virtual reality environment and bio-signal data |
US10353532B1 (en) * | 2014-12-18 | 2019-07-16 | Leap Motion, Inc. | User interface for integrated gestural interaction and multi-user collaboration in immersive virtual reality environments |
US9696795B2 (en) | 2015-02-13 | 2017-07-04 | Leap Motion, Inc. | Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments |
CN110716759B (en) * | 2018-06-26 | 2023-06-30 | 深圳富泰宏精密工业有限公司 | Electronic device, computer-readable storage medium, and operation parameter configuration method |
-
2014
- 2014-09-03 JP JP2014004691U patent/JP3194297U/en not_active Expired - Fee Related
- 2014-09-11 CN CN201420521132.4U patent/CN204537054U/en not_active Expired - Lifetime
- 2014-09-11 DE DE202014104297.0U patent/DE202014104297U1/en not_active Expired - Lifetime
-
2015
- 2015-08-13 US US14/826,102 patent/US11386711B2/en active Active
-
2022
- 2022-06-23 US US17/848,181 patent/US11749026B2/en active Active
-
2023
- 2023-07-24 US US18/225,598 patent/US20230368577A1/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120007821A1 (en) * | 2010-07-11 | 2012-01-12 | Lester F. Ludwig | Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (hdtp) user interfaces |
US20130141375A1 (en) * | 2011-12-06 | 2013-06-06 | Lester F. Ludwig | Gesteme (gesture primitive) recognition for advanced touch user interfaces |
Also Published As
Publication number | Publication date |
---|---|
JP3194297U (en) | 2014-11-13 |
CN204537054U (en) | 2015-08-05 |
US11386711B2 (en) | 2022-07-12 |
US20160048725A1 (en) | 2016-02-18 |
US11749026B2 (en) | 2023-09-05 |
US20220319235A1 (en) | 2022-10-06 |
DE202014104297U1 (en) | 2015-01-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11749026B2 (en) | Automotive and industrial motion sensory device | |
US12095969B2 (en) | Augmented reality with motion sensing | |
US9868449B1 (en) | Recognizing in-air gestures of a control object to control a vehicular control system | |
US11676349B2 (en) | Wearable augmented reality devices with object detection and tracking | |
US12105889B2 (en) | Velocity field interaction for free space gesture interface and control | |
US11775033B2 (en) | Enhanced field of view to augment three-dimensional (3D) sensory space for free-space gesture interpretation | |
CN110546595B (en) | Navigation holographic image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |