DE112016006769T5 - Sign language input to a user interface of a vehicle - Google Patents

Abstract

Sensors in a vehicle recognize the execution of a sign language symbol. The sign language icon invokes a vehicle interface function. In some embodiments, the sign language icon is an ASL gesture. The sensors include three-dimensional optical, ultrasonic or other sensors. The locations of targets (fingertips, ankles, palm, etc.) are detected in the individual outputs of the sensors. The locations of targets determined from the outputs of multiple sensors can be matched and filtered using, for example, an RBMCDA algorithm and Kalman filtering. The system can be calibrated to recognize each symbol by measuring the target locations, with the user performing the respective symbol.

Description

GENERAL PRIOR ART

FIELD OF THE INVENTION

This invention relates to user interfaces for systems of a vehicle.

GENERAL PRIOR ART

A modern vehicle fulfills many functions beyond transport. For example, sound systems, climate controls, interfaces to a mobile telephone or other communication system, and other functions may be provided to an occupant of a vehicle. Accordingly, controls for these functions must be provided. To reduce the number of keys, interface functions may be implemented using a touch screen interface. The controls may be further simplified by providing a voice control system, such as the FORD SYNC system.

The systems and methods disclosed herein provide an improved approach to receiving user input from the hearing-impaired without requiring interaction with a touch screen or other physical interface element.

list of figures

• 1 ein schematisches Blockdiagramm eines Systems zum Implementieren von Ausführungsformen der Erfindung ist;
• 2 ein schematisches Blockdiagramm einer beispielhaften Rechenvorrichtung ist, die zum Implementieren von Verfahren gemäß Ausführungsformen der Erfindung geeignet ist;
• 3 ein schematisches Diagramm eines Fahrzeugs mit Sensoren zum Erkennen von visuellen Eingaben gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 4 ein Prozessablaufdiagramm eines Verfahrens zum Decodieren von visuellen Symbolen von mehreren Sensoren gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 5 ein Prozessablaufdiagramm eines Verfahrens zum Steuern eines Fahrzeugsystems unter Verwendung eines visuellen Symbols gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 6 ein Diagramm ist, das in einem Bild erkannte Ziele gemäß einer Ausführungsform der vorliegenden Erfindung veranschaulicht; und
• 7A und 7B ein Verfahren zum Bereitstellen einer visuellen Bestätigung der Eingabe eines visuellen Symbols gemäß einer Ausführungsform der vorliegenden Erfindung veranschaulichen.

In order to readily understand the advantages of the invention, a more particular description of the invention briefly described above is provided by reference to specific embodiments illustrated in the accompanying drawings. It being understood that these drawings illustrate only typical embodiments of the invention and are therefore not to be considered limiting of its scope, the invention will be described and explained with additional accuracy and in detail by the use of the accompanying drawings, in which:

• 1 Figure 3 is a schematic block diagram of a system for implementing embodiments of the invention;
• 2 FIG. 10 is a schematic block diagram of an example computing device suitable for implementing methods in accordance with embodiments of the invention; FIG.
• 3 Figure 3 is a schematic diagram of a vehicle having visual input sensors according to an embodiment of the present invention;
• 4 Fig. 10 is a process flow diagram of a method for decoding visual symbols of multiple sensors according to one embodiment of the present invention;
• 5 Fig. 10 is a process flow diagram of a method of controlling a vehicle system using a visual symbol according to an embodiment of the present invention;
• 6 Fig. 12 is a diagram illustrating targets detected in an image according to an embodiment of the present invention; and
• 7A and 7B illustrate a method of providing visual confirmation of the input of a visual symbol in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

It will be readily understood that the components of the present invention, as generally described herein and illustrated in the figures, may be arranged and configured in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the invention as illustrated in the figures is not to be taken as limiting the scope of the claimed invention but as merely representative of certain examples of presently contemplated embodiments according to the invention. The presently described embodiments are best understood with reference to the drawings, in which like parts are designated by like reference characters.

Embodiments according to the present invention may be embodied as a device, method, or computer program product. Accordingly, the present disclosure may take the form of a fully hardware embodied embodiment, an entirely software embodied embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware, all of which are generally incorporated herein by reference "Module" or "system". Furthermore, the present invention may take the form of a computer program product executed in a concrete expression medium having computer usable program code contained in the medium.

Any combination of one or more computer-usable or computer-readable media may be used. For example, a computer readable medium may include one or more of a portable computer diskette, a hard disk, a random access memory (RAM) device, a read only memory (ROM) device, a programmable read only memory (EPROM or flash memory) device, a portable compact disc read only memory (CDROM) device, an optical disk Storage device and a magnetic storage device include. In selected embodiments, a computer-readable medium may include any non-transitory medium that may contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, device, or device.

Computer program code for performing operations of the present invention may be in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C ++ or the like, and conventional procedural programming languages such as the "C" programming language or the like Programming languages, be written. The program code may be executed entirely on a computer system as a stand-alone software package, on a stand-alone hardware device, partially on a remote computer some distance away from the computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the computer by any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, using the Internet using an internet service provider).

The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams and combinations of blocks in the flowchart illustrations and / or block diagrams may be implemented by computer program instructions or computer program code. These computer program instructions may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing device for manufacturing a machine such that the instructions executed via the processor of the computer or other programmable data processing device include means for setting up the computer Functions / actions specified in the block or blocks of the flowchart and / or the block diagram create.

These computer program instructions may also be stored in a non-transitory computer readable medium that causes a computer or other programmable data processing device to function in a particular manner such that the instructions stored in the computer readable medium produce an article of manufacture, including instruction means set up the function / action specified in the block or blocks of the flowchart and / or block diagram.

The computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of method steps to be performed on the computer or other programmable device to produce a computer-implemented method such that the instructions that rely on the computer or other programmable device, provide methods for establishing the functions / actions specified in the block or blocks of the flowchart and / or block diagram.

1 illustrates electronic components 100 that can be installed in a vehicle. The electronic components 100 can a controller 102 included in a vehicle. The vehicle may include any vehicle known in the art. The vehicle may include any structures and features of any vehicle known in the art, including wheels, a powertrain coupled to the wheels, a powertrain coupled engine, a steering system, a braking system, and other systems known in the art. that they are included in a vehicle. As will be discussed in greater detail herein, the controller may 102 controlling one or more vehicle systems in accordance with visual symbols, as described below with respect to the methods of 4 and 5 is described.

The control 102 may include one or more data streams from one or more imaging sensors 104 receive. The imaging sensors 104 may be three-dimensional imaging sensors that provide both an image of objects in their field of view and a depth of the objects. For example, the imaging sensor 104 a depth sensor as provided by MICROSOFT KINECT or the like.

Other sensors can also be connected to the controller 102 be coupled and provide an estimate of a depth for an object in its field of view. For example, an ultrasonic sensor 106 to the controller 102 be coupled. The control 102 can also be connected to a microphone 108 coupled in the cabin of the vehicle to receive spoken or other audible commands.

The control 102 can be a symbol recognition module 110 execute the data streams from the sensors 104 . 106 receives and identifies visual symbols that are executed by an occupant of the vehicle. For example, executed symbols may be American Sign Language (ASL) symbols. Although the systems and methods disclosed herein are described in terms of ASL, any other sign language may be implemented in an identical manner.

The symbol recognition module 110 may be a sensor preprocessing module 112a include. The sensor preprocessing module 112a Each sensor processes raw outputs individually and identifies targets in the sensor outputs. Targets as used herein refer to portions of a hand, arm or face of an occupant. In particular, fingertips, joints, spans between joints, a palm, a back of the hand, and a wrist can be identified for one hand. An elbow, a shoulder, a forearm and an upper arm can be identified for one arm. For a face, eyes, nose, mouth can be identified. The manner in which targets are identified may include any method known in the field of image processing, such as those implemented by MICROSOFT KINECT or similar systems. The output of the sensor preprocessing module 112a may be a list of target locations, such entries containing the values <a, x, y, z>, where 'a' is an identifier of the target (fingertip, eye, palm, first ankle of the index finger, etc.) and, x, y, z 'are three-dimensional coordinates of the target in the field of view of the sensor. In some embodiments, the list of target sites may represent or be converted to skeletal imaging vectors that represent the position of bone alignment of the hand to facilitate identification of a symbol performed by the user. Skeletal imaging vectors may be embodied as line segments defining the various bones in the hands, wrist, forearm, etc., and having translation and rotation relative to the body center. These line segments vary over time. These line segments can be tracked over time and assigned to the correct body part. The movement of the body pairs can then be detected and interpreted.

The symbol recognition module 110 can be a destination assignment module 112b include. If several sensors 104 . 106 have an occupant in their field of vision, everyone can see the same goal. Accordingly, the target mapper module may attempt to match target sites that correspond to the same destination. As will be described in greater detail below, this may involve performing a Rao Blackwelled Monte Carlo Data Association (RBMCDA) algorithm. The output of the destination mapper 112b may be a list of locations for the tracked destinations, where the entries for each list correspond to a single destination, that of the multiple measurements of the single destination by the multiple sensors 104 . 106 derived according to the RBMCDA algorithm.

The output of the destination mapper 112b becomes a target tracking module 112c entered, which processes the list of target sites in the context of the movement of the target over time. For example, the destination tracking module 112c process the locations of each destination over time according to a Kalman filter using context information. The Kalman filter removes noise in the recognized location of each destination, i. h random deviation, outliers that are clearly flawed, etc.

The Kalman filter, also referred to as a linear quadratic estimate, is a filter that performs a series of measurements over time and generates estimates that are more accurate than those based on a single measurement. The context information is the measurements made in the past and future of a concrete estimate. For example, the estimate for time step = k uses information from time steps = k-1, k and k + 1.

The bodies of the goals, as determined by the target tracking module 112c can be processed in the decoding module 112d be entered. The decoding module 112d maps a state of the targets at a given time and / or a progression of the targets over time to an ASL word or character output data (eg, text) corresponding to the ASL word or character.

The output of the decoding module 112d can in a vehicle system interface 114 be entered. The vehicle system interface 114 can be a computing device or one through the controller 102 self-implemented functionality be the commands to one or more vehicle systems 116a - 116e couples and / or receives status information from them. For example, the vehicle systems may be a sound system 116a , a navigation system 116b , mobile applications 116c , Cellphone features 116d or other functions 116e include. In some embodiments, the sound system may 116a Be part of a vehicle infotainment system.

The control 102 can be connected to a display device 118 and an input device 120 be coupled. The display device 118 and the input device 120 may be configured as a touch screen. In some embodiments, the display device is 118 a heads-up display (HUD) that displays information on the windshield of the vehicle or on another transparent screen. Status information for the vehicle systems 116a - 116e can through the display device 118 being represented. Inputs to control the vehicle systems may also be through the input device 120 be received. The through the vehicle system interface 114 via the input device 120 received inputs can invoke the same functions that can be invoked using ASL symbols generated by the symbol recognition module 110 were decoded.

2 FIG. 4 is a block diagram illustrating an example computing device. FIG 200 , The computing device 200 can be used to perform various procedures, such as those discussed herein. The control 102 and the vehicle system interface 114 may have some or all attributes of the computing device 200 exhibit.

The computing device 200 includes one or more processors 202 , one or more storage device (s) 204 , one or more interface (s) 206 , one or more mass storage device (s) 208 , one or more input / output (I / O) device (s) 210 and a display device 230 all to a bus 212 are coupled. The processor (s) 202 includes / include one or more processors or controllers executing instructions stored in the storage device (s) 204 and / or the mass storage device (s) 208 are stored. The processor (s) 202 may also include various types of computer readable media, such as a cache memory.

The storage device (s) 204 includes / include various computer-readable media, such as volatile memory (eg Random Access Memory - RAM 214 ) and / or a non-volatile memory (eg, a read-only memory - ROM 216 ). The storage device (s) 204 may also include a rewritable ROM, such as a flash memory.

The mass storage device (s) 208 include various computer-readable media such as magnetic tapes, magnetic disks, optical disks, solid-state storage (e.g., flash memory), and so forth. As in 2 As shown, a concrete mass storage device is a hard disk drive 224 , Different drives may also be in the mass storage device (s) 208 be included to enable reading from and / or writing to the various computer-readable media. The mass storage device (s) 208 includes / contains removable media 226 and / or non-removable media.

The I / O device (s) 210 includes / include various devices that enable data and / or other information to be included in the computing device 200 be entered or retrieved from it. To an exemplary I / O device (s) 210 These include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.

The display device 230 includes any type of device that provides information to one or more users of the computing device 200 can show. Examples of a display device 230 include a screen, a display device, a video projection device, and the like.

The interface (s) 206 includes / include various interfaces that enable the computing device 200 interacts with other systems, devices or computing environments. Exemplary interface (s) 206 include any number of different network interfaces 220 such as local area network (LAN), wide area network (WAN), wireless networks, and the Internet interfaces. Other interface (s) include a user interface 218 and a peripheral device interface 222 , To the interface (s) 206 may also include one or more peripheral interfaces, such as pointing device interfaces (mice, trackpad, etc.), keyboards, and the like.

The bus 212 allows communication of the processor (s) 202 , the storage device (s) 204 , the interface (s) 206 , the mass storage device (s) 208 , the I / O Device (s) 210 and the display device 230 with each other as well as with other devices or components attached to the bus 212 are coupled. The bus 212 Figure 1 illustrates one or more different types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

For purposes of illustration, programs and other executable program components are illustrated herein as discrete blocks, although it should be understood that such programs and components may be stored in different memory components of the computing device at different times 200 and by the processor (s) 202 be executed. Alternatively, the systems and operations described herein may be implemented in hardware or a combination of hardware, software and / or firmware. For example, one or more application specific integrated circuits (ASIC) may be programmed to execute one or more of the systems and operations described herein.

Referring to 3 can a vehicle 300 the controller housed therein 102 as well as some or all of the others in 1 include electronic components shown. As shown, the imaging sensors can 104 be mounted in the vehicle cabin so that they have an occupant of the cabin in their field of view. An ultrasonic sensor 106 or another type of sensor may also be mounted in the vehicle cabin having an occupant of the cabin in his field of view. In the illustrated example, the sensors have 104 . 106 a driver's seat position in their field of vision. In some embodiments, both the driver and front passenger seat positions are within the field of view of the sensors 104 . 106 , In some embodiments, separate sensors 104 . 106 used for each of the driver's seat position and the passenger's seat position. In some embodiments, rear seat positions are also within the field of view of the same sensors 104 . 106 or a separate set of sensors 104 . 106 ,

Referring to 4 can the illustrated method 400 through the controller 102 be executed, for example, within the vehicle system interface 114 to identify ASL symbols in the fields of view of the sensors 104 . 106 be executed. The procedure 400 can receive ( 402 ) of outputs of the sensors 104 . 106 include.

The outputs are then processed independently ( 404 ) to identify target locations in each sensor output. In particular, any image analysis technique known in the art may be used to individually identify targets in each sensor output. The identified targets may include portions of the user's hand, arm and face as described above. As also noted above, identifying target locations may include identifying a three-dimensional location of the target, ie, a location that includes depth as well as vertical and lateral locations within an image.

The procedure 400 can then assign ( 406 ) of the target points to each other. In particular, the locations determined from the sensor outputs corresponding to the same destination may be identified and grouped or otherwise combined. The manner in which outputs from a plurality of sensors determine particular target locations to conform to the same target may include executing the Rao Blackwelled Monte Carlo Data Association (RBMCDA) algorithm, where the target locations determined from each sensor output are the inputs and the outputs are groupings of locations, each grouping including destination locations of multiple sensor outputs determined according to the RBMCDA algorithm to correspond to the same destination.

The output of the mapping step 406 can be a list of destinations for a variety of destinations. The step of receiving ( 402 ) of sensor outputs and the subsequent processing steps 404 - 406 can be repeated periodically so that lists of target locations as a result of step 406 be issued periodically. The procedure 400 can also filter ( 408 ) of these lists of locations to remove noise and unequally erroneous outliers. Accordingly, each destination can be added to a series of preceding destinations for the same destination, and this series of values can be filtered ( 408 ), such as using a Kalman filter. The filtering step can take context information into account.

Referring to 5 is the output of the filtering step 408 also a list of destinations that are opposite to those in step 406 issued list may be modified, and may according to the method 500 are processed. Some ASL symbols involve movement as well as concrete configurations of the hand, fingers and arms. Accordingly, the method 500 with respect to a set of lists of target sites over a period of time to determine movements as well as position.

The procedure 500 can identify ( 502 ) of a symbol according to a list of destinations or a set of lists of destinations. For example, the system can be calibrated by having the user execute a variety of ASL symbols. When the user executes the symbols, a calibrated list of target locations or a calibrated set of lists of destination locations may be in the same manner as for the procedure 400 of the 4 be determined. A library of calibrated lists of destinations or calibrated sets of lists of destinations can be obtained in this manner, with each calibrated list or set of lists being mapped to a symbol that was executed when the sensor outputs used to obtain it were received ,

In some embodiments, the calibration is performed in combination with an application running on a mobile phone or other device of the user. Through the application, the user may be instructed to execute a symbol for a particular interface statement, word, character, or other data, and the execution of the symbol may then be recorded and that particular interface statement, word, character, or other data in the above be assigned described manner.

In some embodiments, as noted above, a list of target sites may be embodied or transformed into skeletal imaging vectors that represent the location and orientation of the user's bones based on the target sites. The skeletal imaging vectors may then be compared to calibrated skeletal imaging vectors obtained from the observation of the user performing symbols during the calibration phase as described above.

Identifying ( 502 ) of a symbol may include comparing a current list of target locations or a current set of lists of target locations based on sensor outputs during user execution of a symbol with the library. The calibrated list or set of lists that match the current list of destinations or the current set of lists of destinations can be identified.

Once a match is identified, the symbol mapped to the identified calibrated list or identify calibrated set of lists is mapped to a vehicle interface command ( 504 ). This command then becomes, for example, through the vehicle system interface 114 executed ( 506 ).

Referring to 6 can the sensors 104 . 106 in one example, capture a three-dimensional image of a user's face, hand, and forearm. For each sensor output can be places 600 are identified, assigned, filtered, and output as a list of destinations, as previously described 4 has been described. In the illustrated example, the configuration of the hand with respect to the user's face is the ASL icon for telephone. Accordingly, upon detection of this icon, the vehicle system interface may invoke a telephone call or other action related to a user's mobile phone. Multiple icons can be captured to call more complex tasks or to provide parameters that define the execution of a task.

In some embodiments, a translation function may be implemented such that a text or voice representation of one or more decoded symbols is output. For example, a driver may enter ASL symbols which may then be translated into text or speech for a passenger who does not understand ASL.

Referring to the 7A and 7B In some embodiments, feedback may be provided to the user. For example, as in 7A shown the hand 700 of a user (or other parts of the user that may be required to execute an ASL icon) on a display device 118 being represented. If the executed symbol was successfully decoded, the representation of the user's hand can be changed to confirm that the symbol has been decoded. In the illustrated example, a grid representation is shown 702 of the user's hand, for example, based on the target locations identified according to the methods disclosed herein.

The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of the equivalency of the claims are to be understood as included within the scope thereof.

Claims (20)

A method comprising, by a vehicle controller, performing the following: acquiring three-dimensional (3D) data of a user using a sensor; Determining that the user has executed a sign language icon from the 3D data; and calling a vehicle interface function according to the sign language icon. Method according to Claim 1 wherein the vehicle interface function is a navigation function. Method according to Claim 1 wherein the vehicle interface function is a sound system control function. Method according to Claim 1 wherein the vehicle interface function is an output of a text-based or audible translation of the sign language symbol. Method according to Claim 1 wherein the sensor is a three-dimensional optical sensor. Method according to Claim 1 wherein the sensor is one of a plurality of sensors, the plurality of sensors including a plurality of three-dimensional optical sensors. Method according to Claim 1 wherein the sensor is one of a plurality of sensors, the plurality of sensors including at least one of a three-dimensional optical sensor and an ultrasonic sensor. Method according to Claim 7 , further comprising: receiving a plurality of outputs from the plurality of sensors by the vehicle controller; for each output of the plurality of outputs, identifying a plurality of destination locations each corresponding to a portion of a user's hand; for each part of a plurality of parts of the user's hand: assigning a target location to a group of target locations for each part of the user's hand for each sensor of the plurality of sensors; and temporally filtering the group of target sites for each part to obtain a filtered target site; Identifying the sign language icon according to the filtered target locations for the plurality of parts of the user's hand. Method according to Claim 8 wherein assigning a target location to the group of destination locations for each part of the user's hand for each sensor of the plurality of sensors comprises performing a Rao Blackwellized Monte Carlo Data Association (RBMCDA) algorithm. Method according to Claim 9 wherein the time filtering of the group of target sites for each part to obtain the filtered target site comprises performing Kalman filtering. A vehicle comprising: a plurality of vehicle systems housed in the vehicle; a sensor having a seating position of the vehicle in a field of view of the sensor; and a controller housed in the vehicle and operably coupled to the plurality of vehicle systems and the sensor, the controller programmed to: Acquiring three-dimensional (3D) data of a user using the sensor; Determining that the user has executed a sign language icon from the 3D data; and Calling a vehicle interface function of one of the plurality of vehicle systems that corresponds to the sign language icon. Vehicle after Claim 11 wherein the vehicle interface function is a navigation function. Vehicle after Claim 11 where the vehicle interface function is a sound system control function. Vehicle after Claim 11 wherein the vehicle interface function is an output of a text-based or audible translation of the sign language symbol. Vehicle after Claim 11 wherein the sensor is a three-dimensional optical sensor. Vehicle after Claim 11 wherein the sensor is one of a plurality of sensors, the plurality of sensors including a plurality of three-dimensional optical sensors. Vehicle after Claim 11 wherein the sensor is one of a plurality of sensors, the plurality of sensors including at least one of a three-dimensional optical sensor and an ultrasonic sensor. System after Claim 11 wherein the controller is further programmed to: receive a plurality of outputs from the plurality of sensors; for each output of the plurality of outputs, identifying a plurality of destination locations each corresponding to a portion of a user's hand; for each part of a plurality of parts of the user's hand: assigning a target location to a group of target locations for each part of the user's hand for each sensor of the plurality of sensors; and temporally filtering the group of target sites for each part to obtain a filtered target site; and identifying the sign language icon according to the filtered destination locations for the plurality of portions of the user's hand. Vehicle after Claim 18 wherein the controller is further programmed to assign a target location to the group of destination locations for each portion of the user's hand for each sensor of the plurality of sensors by performing a Rao Blackwelled Monte Carlo Data Association (RBMCDA) algorithm. Vehicle after Claim 19 wherein the controller is further programmed to time-filter the group of target locations for each part to obtain the filtered target location by performing Kalman filtering.

Images