Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/002—Specific input/output arrangements not covered by G06F3/01 - G06F3/16
  • G06F3/005—Input arrangements through a video camera
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/0304—Detection arrangements using opto-electronic means
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/20—Movements or behaviour, e.g. gesture recognition
  • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/146—Instrument input by gesture
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/146—Instrument input by gesture
  • B60K2360/1464—3D-gesture
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/20—Optical features of instruments
  • B60K2360/21—Optical features of instruments using cameras

Definitions

• the present invention relates to a method and apparatus for operating functions in a vehicle using gestures executed in three-dimensional space and a related computer program product.
• US 2008/0065291 A1 discloses a method and a device for operating functions in a vehicle using gestures executed in three-dimensional space, in which it is determined whether a gesture carried out in three-dimensional space is detected by means of an image-based detection process or not, it is determined whether or not the detected gesture is a gesture associated with controlling a function, and the function is controlled if it is determined that the detected gesture is the gesture associated with controlling the function.
• a detected gesture is a gesture assigned to controlling a function
• movement of, for example, a user's finger or hand performed in a detection area of an image-based gesture detection means may be prohibited a function is erroneously determined as the gesture associated with controlling the function. Consequently, in this case, the function is carried out erroneously or unintentionally.
• a method for operating functions in a vehicle using gestures executed in three-dimensional space comprises a) determining whether or not a first gesture performed in three-dimensional space is detected by an image-based detection operation, b) determining whether the first gesture is a gesture associated with activating a control of a function or not, if it is determined that the first gesture has been detected, c) activating the control of the function, if it is determined that the detected first gesture is the one Activating controlling the gesture associated with the function; d) determining whether or not a second gesture performed in the three-dimensional space is detected by the image-based detection operation; e) determining whether the detected second gesture has a gesture associated with controlling the function or not, if it is determined that the second gesture has been detected, and f) a Controlling the function if it has been determined that the detected first gesture is the gesture associated with activating controlling the function, and if it is determined that the detected second gesture is the gesture associated with controlling the function.
• steps d) to f) are carried out repeatedly one after the other after carrying out steps a) to c).
• step b) it is determined in step b) that the detected first gesture is the gesture associated with activating the control of the function, if the detected first gesture is a first predetermined gesture, for a first predetermined time period in an interaction region of the three-dimensional space is static.
• a display element representing the activation of the function is displayed in step c).
• the display of the function-indicating display element is no longer displayed on the display unit after a fourth predetermined time period in which no gesture is detected.
• step e) it is determined in step e) that the detected second gesture is the gesture associated with controlling the function if the detected second gesture is a second predetermined gesture that is dynamic in the interaction area of the three-dimensional space.
• the interaction area is set smaller than a maximum detection area of the image-based detection process.
• the interaction area is set free of interfering objects.
• the interaction area is dynamically adapted context-dependent.
• the image-based detection process is camera-based, it detects a position of an object executing a gesture in the
• a device for operating functions in a vehicle using gestures executed in three-dimensional space comprises means adapted to carry out the method or its embodiments described above.
• a computer program product for operating functions in a vehicle using gestures executed in three-dimensional space is configured to interact with a computer or computer system directly or, after performing a predetermined routine, indirectly the method or its embodiments described above perform.
• the gesture associated with controlling the function since before being detected the gesture associated with controlling the function must detect a gesture associated with activating a control of a function by means of which the control of the function is activated.
• FIG. 1 is a schematic diagram showing a basic construction of a display unit and a detection concept according to an embodiment of the present invention
• Fig. 2 is a further schematic representation of the basic structure of
• FIG. 3 is a schematic diagram showing the basic structure of the display unit and a mounting location of a detection device in an overhead operating unit according to the embodiment of the present invention
• Fig. 4 is a schematic representation of the basic structure of the display unit and a mounting location of a detection device in an interior mirror according to the
• FIG. 5 is a flowchart of a method of operating functions in a vehicle using gestures executed in three-dimensional space according to the embodiment of the present invention.
• a display unit is a preferably central display of a vehicle, preferably of a motor vehicle, and a method for operating functions displayed on the display unit using in three-dimensional space
• executed gestures is performed in the vehicle.
• a gesture described below is a gesture made by a user of the vehicle by means of a hand or a finger of the user in the
• three-dimensional space is performed without touching a display, such as a touch screen, or a control element, such as a touchpad.
• the image-based detector described below may be any type of the image-based detector described below.
• a convenient camera that is capable of detecting a gesture in three-dimensional space, such as a depth camera, a camera with structured light, a Stereo camera, a camera based on time-of-flight technology or an infrared camera combined with a mono camera.
• a depth camera such as a depth camera, a camera with structured light, a Stereo camera, a camera based on time-of-flight technology or an infrared camera combined with a mono camera.
• An infrared camera combined with a mono camera improves detection capability because a mono camera with a high image resolution provides additional intensity information, which offers advantages in background segmentation, and a mono camera is insensitive to extraneous light.
• Fig. 1 shows a schematic representation of a basic structure of a
• reference numeral 10 denotes a display unit of a vehicle
• reference numeral 20 denotes a valid detection area of an image-based detector
• reference numeral 30 denotes an overhead operating unit of the vehicle
• reference numeral 40 denotes an interior mirror of the vehicle
• reference numeral 50 denotes a center console of the vehicle
• reference numeral 60 denotes a dome of the vehicle.
• the basic operating concept is that a gesture control for operating functions by means of a by a user's hand or finger
• the performed gesture is detected in the detection area 20 by the image-based detection means as a predetermined gesture.
• the valid detection area 20 is determined by an image-based detection device capable of detecting a three-dimensional position of the user's hand or finger in three-dimensional space.
• the image-based detection device is preferably a depth camera integrated into the vehicle.
• the image-based detection device must be integrated such that a
• Gesture control is permitted by a relaxed hand and / or Armen the user at any position in the area above the dome 60 and the center console 50 of the vehicle. Therefore, a valid detection range is limited upward from an upper edge of the display unit 10 and downwardly by a minimum distance to the dome 60 and the center console 50.
• a gesture control is activated if a first gesture in the valid
• Detection area 20 is detected, which is a first predetermined gesture.
• the first predetermined gesture is a static gesture made by moving the user's hand or finger into the valid detection area 20 and then momentarily holding the user's hand or finger for a first
• the gesture control is deactivated by guiding the user's hand or finger out of the valid detection range. Placing the user's hand or arm on the center console 20 and operating components of the vehicle is performed below the valid detection area 20, whereby no gesture control is activated.
• FIG. 2 is another schematic diagram showing the basic structure of the display unit and the detection concept according to the embodiment of the present invention.
• the same reference numerals as in FIG. 1 denote the same elements, and the reference numeral 70 designates an object existing in or on the center console 50 as an obstruction object such as a beverage container in one
• Cup holder
• a lower limit of the valid detection range 20 is dynamically applied to the
• Item 70 adapted.
• Such context-dependent adaptation of the valid detection area as an interaction area is performed such that a depth contour of the valid detection area is performed by depth information of the image-based detection means such as the depth camera in real time upon detection of a gesture. This means that a valid gesture must be made over the item 70.
• Arranging the image-based detection device in a roof area of the vehicle leads to the following advantages: No sunlight is radiated into an optic of the image-based detection device. It will be a complete one
• Detection area is also covered in a vicinity of the display unit 10 as a valid detection area 20. There is a high image resolution in
• the image-based detection device is from a normal field of vision of the driver and front passenger. Roof components can be strongly standardized across different series with fewer design variants. Low requirements for a detection range are required.
• Fig. 3 shows a schematic representation of the basic structure of
• Fig. 3 the same reference numerals as in Figs. 1 and 2 denote the same elements, and the reference numeral 100 denotes a maximum detection angle of an image-based integrated in the roof control unit 30 of the vehicle
• the complete valid detection range 20 is covered by the image-based detection device integrated in the roof control unit 30.
• Another advantage of the image-based detection device integrated in the roof control unit 30 is that the greatest possible vertical distance to the valid detection area 20 is achieved.
• Fig. 4 shows a schematic representation of the basic structure of
• FIG. 4 Display unit and a location of a detection device in an interior mirror according to the embodiment of the present invention.
• the same reference numerals as in FIGS. 1 and 2 denote the same elements, and the reference numeral 110 denotes a maximum detection angle of an image-based integrated in the vehicle interior mirror 40
• the complete valid detection area 20 is covered by the image-based detection device integrated in the roof control unit 30.
• an alignment offset of the image-based detection device In order to compensate for a changing orientation of the image-based detection device due to an adjustment of the inner mirror 40, based on a contour of the center console 50, an alignment offset of the image-based
• FIG. 5 shows a flowchart of a method for operating functions in a vehicle using gestures executed in three-dimensional space according to the embodiment of the present invention.
• a flow of processing of the flowchart in FIG. 5 is turned on after an initialization timing such as after the ignition of the vehicle is turned on and cyclically repeated until a termination timing such as turning off the ignition of the ignition Vehicle, is reached.
• the initialization timing may be, for example, the timing of starting an engine of the vehicle and / or the termination timing the timing of stopping the engine of the vehicle.
• the detected gesture is both a gesture executed by the driver and a gesture executed by the passenger can.
• the method of the flowchart in Fig. 5 is performed for both the driver and the passenger side.
• the processing sequence shown in Fig. 5 may conveniently be performed, for example, in parallel, serially, or in a nested manner for the driver and passenger sides.
• step S100 it is determined whether or not a first gesture is detected. If the first gesture is not detected, which corresponds to a "No" answer in step S100, the processing flow returns to step 100. If the first gesture is detected, which corresponds to a "Yes" answer in step S100, the processing flow advances to step S200.
• step S200 it is determined whether or not the detected first gesture is a gesture associated with activating control of a function. If the first gesture is not a gesture associated with activating the control of the function, which corresponds to a "No" answer in step S200, the processing flow returns to step S100. If the first gesture is the gesture associated with activating the control of the function, which corresponds to a "Yes" answer in step S200, the processing flow advances to step S300.
• the gesture associated with activating the control of the function is a first predetermined gesture that is for a first predetermined period of time in a
• Interaction area of the three-dimensional space is static.
• the first predetermined gesture is detected as previously described with reference to FIGS. 1 to 3.
• the interaction area corresponds to the valid one described above
• step S300 the control of the function is activated. After step S300, the processing flow advances to step S400.
• step S400 it is determined whether or not a predetermined cancellation condition is satisfied. If the predetermined termination condition is satisfied, which is a "Yes" answer Step S400, the processing flow returns to step S100. If the cancellation condition is not satisfied, which corresponds to a "No" answer in step S400, the processing flow advances to step S500.
• the predetermined termination condition may be, for example, that no gesture has been detected for a fourth predetermined period of time. If the predetermined
• step S400 Abort condition is satisfied in step S400, the function of the function indicating display element is no longer displayed on the display unit.
• step S500 it is determined whether or not a second gesture is detected. If the second gesture is not detected, which corresponds to a "No" answer in step S500, the processing flow returns to step S500. If the second gesture is detected, which corresponds to a "Yes" answer in step S500, the processing flow advances to step S600.
• step S600 it is determined whether or not the detected second gesture is a gesture associated with controlling the function. If the second gesture is not a gesture associated with controlling the function, which corresponds to a "No" answer in step S600, the processing flow returns to step S500. If the second gesture is the gesture associated with controlling the function, which corresponds to a "Yes" answer in step S600, the processing flow advances to step S700.
• the gesture associated with controlling the function is a second predetermined gesture that is dynamic in the interaction area of the three-dimensional space.
• step S700 the function is controlled.
• a display element indicating the control of the function may be displayed on the display unit.
• step S800 it is determined whether or not a predetermined abort condition is satisfied. If the predetermined abort condition is satisfied, which corresponds to a Yes answer in step S800, the processing flow returns to step S100. If the cancellation condition is not satisfied, which corresponds to a "No" answer in step S800, the processing flow returns to step S500.
• the predetermined termination condition may be, for example, that no gesture has been detected for the fourth predetermined time period. If the predetermined
• step S800 Abort condition is satisfied in step S800, that becomes the controlling of the function
• the method described above can be implemented by means that form a device for operating functions in a vehicle.
• Display unit is preferably a central display of the vehicle, preferably a motor vehicle.
• An application of the above-described embodiment is, for example, switching a menu, such as a menu, back and forth
• Main menus a radio station or a medium, such as a CD, in a central telematics unit of the vehicle by means of gestures, i. Hand or
• Touchscreen or a control element, such as a touchpad to touch.
• a user's learning process may be assisted by visual and / or audible feedback during gesture operation, allowing a user's blind operation after a user's learning phase.
• the user can switch off such optical and / or acoustic feedback manually or such visual and / or audible feedback messages are automatically turned off, for example, for a predetermined period of time after detecting a correct gesture operation by the user.
• the image-based gesture operation described above realizes a simple and fast operability which increases operating comfort, operating flexibility and an operator experience for the user and considerably increases design freedom for a vehicle interior.
• the embodiment described above can be realized as a computer program product, such as a storage medium, which is designed in cooperation with a computer or a plurality of computers, that is to say computer systems, or other computing units, a method according to the preceding one Execute embodiment.
• the computer program product may be configured to execute the method only after performing a predetermined routine, such as a setup routine.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Human Computer Interaction (AREA)
• Multimedia (AREA)
• Psychiatry (AREA)
• Social Psychology (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Chemical & Material Sciences (AREA)
• Combustion & Propulsion (AREA)
• Transportation (AREA)
• Mechanical Engineering (AREA)
• User Interface Of Digital Computer (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=47504797

Family Applications (1)

Country Status (5)

Families Citing this family (20)

Family Cites Families (14)

• 2012
  • 2012-01-10 DE DE102012000263A patent/DE102012000263A1/de not_active Withdrawn
  • 2012-12-08 EP EP12810080.7A patent/EP2802963A1/de not_active Withdrawn
  • 2012-12-08 WO PCT/EP2012/005080 patent/WO2013104389A1/de active Application Filing
  • 2012-12-08 US US14/371,090 patent/US20140361989A1/en not_active Abandoned
  • 2012-12-08 CN CN201280066638.4A patent/CN104040464A/zh active Pending

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events