CN116449959A - Gesture control method and device, computer equipment and storage medium - Google Patents

Gesture control method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN116449959A
CN116449959A CN202310459387.6A CN202310459387A CN116449959A CN 116449959 A CN116449959 A CN 116449959A CN 202310459387 A CN202310459387 A CN 202310459387A CN 116449959 A CN116449959 A CN 116449959A
Authority
CN
China
Prior art keywords
target
gesture
information
control
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310459387.6A
Other languages
Chinese (zh)
Inventor
赵阳阳
徐持衡
周舒岩
罗予晨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Lingang Jueying Intelligent Technology Co ltd
Original Assignee
Shanghai Lingang Jueying Intelligent Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Lingang Jueying Intelligent Technology Co ltd filed Critical Shanghai Lingang Jueying Intelligent Technology Co ltd
Priority to CN202310459387.6A priority Critical patent/CN116449959A/en
Publication of CN116449959A publication Critical patent/CN116449959A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60WCONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08Interaction between the driver and the control system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04812Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/59Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/70Labelling scene content, e.g. deriving syntactic or semantic representations
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02PCLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Automation & Control Theory (AREA)
  • Computational Linguistics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The disclosure provides a gesture control method, a gesture control device, computer equipment and a storage medium, wherein the gesture control method comprises the following steps: acquiring an image to be identified acquired in a target cabin, and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information; mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information; determining target control equipment associated with the target area in the target vehicle cabin; and controlling the target control equipment based on a target control instruction corresponding to the target semantic information.

Description

Gesture control method and device, computer equipment and storage medium
Technical Field
The disclosure relates to the field of computer technology, and in particular relates to a gesture control method, a gesture control device, computer equipment and a storage medium.
Background
The control of the traditional vehicle is mainly realized through the central control of the automobile, and for passengers in the rear row of the automobile cabin, the control of the vehicle is difficult due to the fact that the central control distance from the automobile is long.
In the related art, in order to improve riding experience of each passenger in the vehicle cabin, a voice control mode and other modes can be adopted to control the vehicle, but the vehicle always has certain noise in the running process, so that the voice control mode can not accurately realize control of control equipment in the vehicle cabin due to interference of the noise.
Disclosure of Invention
The embodiment of the disclosure at least provides a gesture control method, a gesture control device, computer equipment and a storage medium.
In a first aspect, an embodiment of the present disclosure provides a gesture control method, including:
acquiring an image to be identified acquired in a target cabin, and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information;
mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information;
determining target control equipment associated with the target area in the target vehicle cabin;
and controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
In a possible implementation manner, the image to be identified contains depth information;
the method further comprises determining three-dimensional position information of the hand in the target vehicle cabin according to the following steps:
determining image coordinates of target key points of the hand in the image to be identified;
and determining three-dimensional position information of the hand in the target vehicle cabin based on the image coordinates of the target key points and the depth information of the hand.
In a possible implementation manner, the preset plane is determined based on the position of a display device in the target cabin, and the preset plane comprises at least one first area corresponding to the display device and a plurality of second areas defined around the display device;
the second areas are used for controlling different control devices, and the control devices controlled by the second areas are related to the first position of each second area in the preset plane and the second position of each control device in the target cabin.
In a possible embodiment, after determining a target control device associated with the target area in the target vehicle cabin, the method further includes:
Displaying a target floating window on the display device, wherein a plurality of control device identifiers are displayed on the target floating window;
highlighting the identification of the target control device.
In a possible embodiment, in case the target control device is a continuously adjustable control device, the method further comprises:
responding to the change of the three-dimensional position information in the gesture information, and determining the motion change amount of the hand based on the changed three-dimensional position information; the motion variable quantity comprises a moving distance and/or a moving angle;
determining a state change amount at the time of controlling the target control device based on the motion change amount;
and determining a state control instruction to the target control device based on the state change amount to change the state of the target control device based on the state control instruction.
In a possible implementation manner, the method further comprises determining a target control instruction corresponding to the target semantic information according to the following steps:
determining cabin position information corresponding to the gesture information;
determining a target control instruction corresponding to the target semantic information based on a preset corresponding relation between the semantic information and the control instruction, wherein the preset corresponding relation is matched with the cabin position information; the corresponding relation between semantic information matched with the position information of different cabins and the control instruction is different.
In a second aspect, embodiments of the present disclosure further provide an intelligent driving vehicle, including: an image acquisition device, a data processor, an image display device, and at least one control device;
the image acquisition device is used for acquiring an image to be identified in the target vehicle cabin and transmitting the image to be identified to the data processor;
the data processor is configured to process an acquired image to be identified according to the gesture control method according to any one of the first aspect to obtain a target control instruction, and control a target control device according to the target control instruction;
the image display device is used for displaying the device name of the controlled target control device.
In a third aspect, an embodiment of the present disclosure further provides a gesture control apparatus, including:
the first determining module is used for acquiring an image to be identified acquired in the target cabin and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information;
the second determining module is used for mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information;
A third determining module, configured to determine a target control device associated with the target area in the target vehicle cabin;
and the control module is used for controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
In a fourth aspect, embodiments of the present disclosure further provide a computer device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the computer device is running, the machine-readable instructions when executed by the processor performing the steps of the first aspect, or any of the possible implementations of the first aspect.
In a fifth aspect, the presently disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the first aspect, or any of the possible implementations of the first aspect.
The effect descriptions of the gesture control apparatus, the computer device, and the computer-readable storage medium are referred to the description of the gesture control method, and are not repeated here.
According to the gesture control method, the gesture control device, the computer equipment and the storage medium, gesture information contained in the image to be identified and collected in the target cabin is determined, three-dimensional position information in the gesture information is mapped to the preset plane, and target control equipment corresponding to the two-position information can be determined after the three-dimensional position information is mapped to the preset plane, so that target control instructions corresponding to target semantic information in residual gesture information can be used for controlling the target control equipment. Like this, compare in traditional mode that directly controls through triggering the car center, can improve the mutual experience of passenger of car cabin back row, and compare in the control mode such as using voice control, carry out gesture control to the controlgear in the car cabin through the three-dimensional position information of hand in the car cabin, can avoid being disturbed by the noise in control process's problem, promote the control accuracy to the controlgear in the car cabin.
The foregoing objects, features and advantages of the disclosure will be more readily apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the embodiments are briefly described below, which are incorporated in and constitute a part of the specification, these drawings showing embodiments consistent with the present disclosure and together with the description serve to illustrate the technical solutions of the present disclosure. It is to be understood that the following drawings illustrate only certain embodiments of the present disclosure and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort.
FIG. 1 illustrates a flow chart of a gesture control method provided by an embodiment of the present disclosure;
fig. 2 is a schematic diagram of a preset plane in a gesture control method according to an embodiment of the disclosure;
FIG. 3 is a schematic diagram of a target floating window in a gesture control method according to an embodiment of the disclosure;
FIG. 4 illustrates a schematic architecture of a gesture control apparatus provided by embodiments of the present disclosure;
fig. 5 shows a schematic structural diagram of a computer device according to an embodiment of the disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. The components of the embodiments of the present disclosure, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be made by those skilled in the art based on the embodiments of this disclosure without making any inventive effort, are intended to be within the scope of this disclosure.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The term "and/or" is used herein to describe only one relationship, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
According to research, in order to improve riding experience of passengers in the vehicle cabin, the vehicle can be controlled in a voice control mode and the like, but the vehicle always has certain noise in the running process, so that the voice control mode can not accurately control equipment in the vehicle cabin due to interference of the noise.
Based on the above study, the disclosure provides a gesture control method, a device, a computer device and a storage medium, by determining gesture information contained in an image to be identified collected in a target cabin and mapping three-dimensional position information in the gesture information to a preset plane, a target control device corresponding to two-position information can be determined after mapping to the preset plane, so that a target control instruction corresponding to target semantic information in residual gesture information can be used for controlling the target control device. Like this, compare in traditional mode that directly controls through triggering the car center, can improve the mutual experience of passenger of car cabin back row, and compare in the control mode such as using voice control, carry out gesture control to the controlgear in the car cabin through the three-dimensional position information of hand in the car cabin, can avoid being disturbed by the noise in control process's problem, promote the control accuracy to the controlgear in the car cabin.
For the sake of understanding the present embodiment, first, a detailed description will be given of a gesture control method disclosed in the embodiments of the present disclosure, where an execution subject of the gesture control method provided in the embodiments of the present disclosure is generally a computer device having a certain computing capability, and the computer device includes, for example: the terminal device or server or other processing device may be a User Equipment (UE), a mobile device, a User terminal, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the gesture control method may be implemented by way of a processor invoking computer readable instructions stored in a memory.
Referring to fig. 1, a flowchart of a gesture control method according to an embodiment of the disclosure is shown, where the method includes S101 to S104, where:
s101: acquiring an image to be identified acquired in a target cabin, and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information.
S102: mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information.
S103: and determining target control equipment associated with the target area in the target vehicle cabin.
S104: and controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
The following is a detailed description of the above steps.
For S101, the target cabin may be a cabin in which an image acquisition device is disposed, where the image acquisition device may be disposed around a vehicle-mounted screen of the target cabin, and is configured to acquire images to be identified, which correspond to different cabin positions in the target cabin, in real time; the number of the image acquisition devices can be multiple, and different image acquisition devices can be deployed at different positions in the target car cabin, so that the images to be identified can be acquired from different shooting angles in the target car cabin; the image to be identified can be an image containing depth information, so that more accurate gesture information can be determined by combining the depth information when the image to be identified is identified; the target semantic information may be semantic information corresponding to a preset gesture, and the preset gesture may be, for example, a fist making gesture, a hand pushing gesture (a gesture in which five fingers are extended and separated), or the like.
Before the image to be identified is acquired, a user can be reminded in a prompt mode such as displaying prompt information, and the image to be identified is acquired after the user authorization is received for image acquisition.
In practical application, because the vehicle may bump or the like during running, the hand of the user may be briefly recognized to make some gestures, but the user may not have a corresponding control intention at this time, so that whether the user has a corresponding control intention at this time can also be determined during the gesture recognition process.
In a possible implementation manner, the three-dimensional position information of the hand in the target cabin can be determined through the following steps A1-A2:
a1: and determining image coordinates of target key points of the hand in the image to be identified.
A2: and determining three-dimensional position information of the hand in the target vehicle cabin based on the image coordinates of the target key points and the depth information of the hand.
Here, the target key point may be a key point satisfying a preset condition in the key points corresponding to the images to be identified, and the preset condition may be that an offset corresponding to the key point is smaller than a preset offset in the images to be identified acquired by continuous multiframes.
Specifically, when determining the image coordinates of the target key points of the hand in the images to be identified, the key points to be screened of the hand in the continuous multi-frame images to be identified can be determined, and the target key points in the candidate key points are determined according to the offset corresponding to each key point to be screened in the continuous multi-frame images to be identified.
Further, after the target key points corresponding to the hands are determined, three-dimensional position information of the hands in the target car cabin can be determined according to the image coordinates corresponding to the target key points and the depth information of the hands, so that gestures of a user can be accurately identified.
Specifically, when the target semantic information contained in the image to be identified is determined, the target key points can be identified, so that the target semantic information corresponding to the hand is obtained, and the preset gesture of the hand can be determined.
In a possible implementation manner, when determining three-dimensional position information of the hand in the target cabin when the number of the image acquisition devices is multiple, a plurality of image coordinates corresponding to target key points in images to be identified which are simultaneously acquired by the image acquisition devices can be determined, and based on the plurality of image coordinates, depth information of the hand acquired by the image acquisition devices and deployment positions corresponding to the image acquisition devices, three-dimensional position information of the hand in the target cabin is determined.
The depth information corresponding to the hand acquired by different image acquisition devices can be different, and the image coordinates of the target key points corresponding to the different image acquisition devices are also corresponding to a camera coordinate system, so that the image coordinates and the depth information acquired by the different image acquisition devices can be converted into a unified coordinate system, and the image coordinates and the depth information of the target key points corresponding to the image acquisition devices can be converted into a preset coordinate system according to the coordinate system conversion relation corresponding to any image acquisition device when the conversion is performed, and the three-dimensional position information of the hand in the target car cabin can be obtained after the conversion of each image acquisition device is completed.
Therefore, the gesture control intention of the user can be accurately judged by taking the key points with stable change as target key points for identifying the gesture information of the user, so that the identification accuracy of the gesture information can be improved.
In practical applications, passengers at different cabin positions may also make gestures at the same time, that is, gesture information to be recognized includes gestures of different users, and control operations corresponding to the different gestures may be conflicted, for example, user a wants to open a sunroof, and user B wants to close a window.
In a possible implementation manner, when the determined gesture information included in the image to be recognized includes hands corresponding to a plurality of users, target control instructions of each user for the control device may be sequentially determined according to gesture recognition priorities corresponding to cabin positions of each user, and the control device may be controlled according to the target control instructions; or, each user can control each control device simultaneously if there is a conflict in the target control instructions of each control device, and if there is a conflict, conflict prompt information can be generated to prompt.
When determining whether each target control instruction has a conflict, conflict determination may be performed from dimensions such as whether the control device targeted by each target control instruction is the same control device, whether the targeted control device is an opposite control instruction (for example, one is open and the other is closed), and the application does not limit specific conflict determination logic, so that implementation can be performed.
Aiming at S102,
Here, the preset plane may be determined based on a position of a display device in the target cabin, and the preset plane includes at least one first area corresponding to the display device and a plurality of second areas defined around the display device; the target area may be at least one of the first area and/or the second area.
The second areas are used for controlling different control devices, and the control devices controlled by the second areas are related to the first position of each second area in the preset plane and the second position of each control device in the target cabin.
As an example, as shown in fig. 2, the schematic diagram of the preset plane may be shown in fig. 2, where the preset plane is a first area (a white area in fig. 2) corresponding to a display device, and a plurality of second areas (gray areas in fig. 2) defined around the display device, where the second area on the left side of the preset plane corresponds to a control device on the left side of the vehicle cabin, such as a front left air conditioner, a front left window, a rear left window, and a rear left air conditioner, the second area on the right side of the preset plane corresponds to a control device on the right side of the vehicle cabin, such as a front right air conditioner, a front right window, a rear right window, and a rear right air conditioner, the second area above the preset plane corresponds to a roof window of the vehicle cabin, and the second area below the preset plane corresponds to an "exhale menu" function, and by triggering the second area corresponding to the "exhale menu", a display page including the preset display content may be displayed in the first area corresponding to the display device.
Specifically, when the three-dimensional position information is mapped to a preset plane to obtain two-dimensional position information, the three-dimensional position information of the hand in the target vehicle cabin, which is contained in the identified image to be identified, can be converted into the preset plane according to the conversion relation between the preset three-dimensional position information and the two-dimensional position information, so that a new three-dimensional position message corresponding to the hand in the image to be identified, which is acquired in real time, can be converted into the preset plane, and the gesture control intention of a user can be conveniently judged in the preset plane; under the condition that the number of the image acquisition devices is multiple, the conversion relations between the three-dimensional position information and the two-position information corresponding to different image acquisition devices can be different, and the conversion relations between the three-dimensional position information and the two-position information corresponding to any one of the image acquisition devices are related to the deployment positions of the image acquisition devices in the target vehicle cabin.
For S103,
Here, the target control device associated with the target area may include a plurality of control devices, and in the case where the determined target areas are the same, the control devices that the user of different cabin positions can control may be different, and the control devices may be control devices that are not related to driving of the vehicle, for example, may be windows, roof windows, air conditioners, and the like in the cabin.
Specifically, when determining the target control device associated with the target area in the target vehicle cabin, the target control device corresponding to the target area and the vehicle cabin position information may be determined according to the vehicle cabin position information corresponding to the target area and the gesture information.
Taking A, B, C, D as an example, after the user 1 makes a gesture 1 at the cabin position a, determining gesture information corresponding to the gesture 1 according to an image to be identified acquired in real time, converting three-dimensional position information in the gesture information to the preset plane, determining that a target area indicated by the gesture information corresponding to the gesture 1 is an area 1, and determining that target control equipment which can be controlled by the user 1 corresponding to the cabin position a at this time is equipment a and equipment B according to the area 1 and the cabin position a; after the user 2 makes the gesture 2 at the cabin position B, the gesture information corresponding to the gesture 2 can be determined according to the image to be recognized acquired in real time, after the three-dimensional position information in the gesture information is converted to the preset plane, the target area indicated by the gesture information corresponding to the gesture 2 is determined to be the area 1, and the target control equipment which is corresponding to the cabin position B and can be controlled by the user 2 at the moment is determined according to the area 1 and the cabin position B, wherein the target control equipment is represented as equipment C and equipment D.
In this way, by setting different gesture control interaction logics (namely, the same operation controls different devices) for users in different cabin positions, the control device which can be controlled by the driver is different from the control device which can be controlled by the passenger, for example, the passenger cannot control the window for opening the driving position or the seat for adjusting the driving position, so that the driver is prevented from being suddenly disturbed in the driving process, and the safety in the driving process is improved.
In addition, the vehicle door can keep one side arm of a user, so that a left arm or a right arm of the user can hardly easily make a specific gesture when the user takes a vehicle, the user can realize the control on the environment where the user is located according to an operation button in a vehicle cabin, for example, the user on the left rear passenger seat can hardly easily make the specific gesture, but the left arm can quickly realize the left rear window regulation and control through a window regulation button, and the left rear window regulation is not more convenient than the window regulation button through the gesture, other control equipment can be used for replacing the control equipment which is inconvenient to control at present at the moment, so that the control equipment which can be controlled among different passengers can also be different, the number of control equipment which can be controlled by the gesture can be increased, and the interaction logic during gesture control is enriched; on the other hand, gesture interaction logic which is more in line with the space environment in the vehicle cabin of the user can be set for different vehicle cabin positions at the same time, the existing vehicle cabin control mode can be effectively supplemented, and riding experience of the user at each vehicle cabin position in the target vehicle cabin is enriched.
In practical application, if the user does not obtain information feedback after making a gesture on the hand, it may be difficult to adjust gesture information, so as to control the control device that the user actually wants to control, so that the user can be informed of the information feedback according to the gesture information.
In a possible implementation manner, after determining the target control device associated with the target area in the target vehicle cabin, the user may also be prompted to the target control device currently controlled by the following steps B1-B2:
b1: and displaying a target floating window on the display device, wherein a plurality of control device identifiers are displayed on the target floating window.
B2: highlighting the identification of the target control device.
As an example, the schematic diagram of the target floating window may be shown in fig. 3, the control device identifier included in the target floating window displayed on the device is "top window", "left front window", "right front window", "left rear window", "right rear window", the currently highlighted target control device is "left front window", and the highlighted manner is to change the background color of the area where the identifier of the target control device is located.
In addition, when the device currently controlled by the user is prompted, the device can be prompted by other modes such as voice prompt, and when the device is prompted, voice prompt information can be played according to an audio playing device matched with the cabin position information corresponding to the user, for example, according to the cabin position information, the audio playing device arranged in the target cabin and at the rear left can be used for prompting the user on the passenger position at the rear left so as to prompt the target control device currently controlled by the user.
In this way, the target control equipment currently controlled by the user is timely prompted, so that the user can be helped to quickly control the control equipment which the user wants to control, and the gesture control efficiency of the user is improved; in addition, in the embodiment of the present disclosure, other information that may be prompted may also be prompted by any of the foregoing prompting manners, for example, voice prompt may be performed after gesture recognition is started, after adjustment is performed on the target control device, real-time state parameters of the target control device after adjustment may also be prompted by means of voice prompt or the like.
Aiming at S104,
Here, the corresponding relation between the semantic information and the control instruction may be preset, for example, the gesture "open after making a fist" may correspond to a trigger operation, the clockwise rotation of the index finger may correspond to an increase operation, the counterclockwise rotation of the index finger may correspond to a decrease operation, etc.; the controlling the target control device includes adjusting a state of the target control device.
Specifically, when the target control device is controlled based on a target control instruction corresponding to the target semantic information, a target parameter of the target control device may be adjusted according to the target control instruction, so as to control the target control device.
In practical applications, the control of the device may include, in addition to turning on or off a function, continuous control of the device, such as continuously increasing volume, continuously increasing brightness, continuously adjusting a seat angle, continuously adjusting an air conditioning fan direction, etc., where for a continuously adjustable control device, a control manner may be set in a targeted manner, so as to improve an adjustment efficiency of the continuously adjustable device.
In a possible embodiment, in the case that the target control device is a continuously adjustable control device, the state of the target control device may be changed by the following steps C1 to C3:
c1: responding to the change of the three-dimensional position information in the gesture information, and determining the motion change amount of the hand based on the changed three-dimensional position information; the motion variation includes a movement distance and/or a movement angle.
C2: based on the motion variation amount, a state variation amount at the time of controlling the target control device is determined.
Specifically, when the state variable quantity is determined based on the motion variable quantity, the state variable quantity may be determined according to a mapping relationship between a preset motion variable quantity and the state variable quantity, for example, the state variable quantity is correspondingly increased by clockwise rotation by 90 degrees, the light brightness of the first grade is correspondingly increased by right movement of the hand by 5cm, and the like.
And C3: and determining a state control instruction to the target control device based on the state change amount to change the state of the target control device based on the state control instruction.
The motion change amount may be proportional to a state change amount when the target control device is controlled, that is, the greater the motion amplitude of the hand is, the greater the state change amount corresponding to the target control device is.
Specifically, when the state of the target control device is changed based on the state control instruction, the state of the target control device may be adjusted in real time based on the state control instruction corresponding to the state change amount determined in real time, so as to realize continuous control of the target control device.
In another possible implementation manner, when the target control device is a continuously adjustable control device, the state of the target control device may be adjusted from a fixed state to an adjustable state according to a target control instruction corresponding to the target semantic information, and the target control device may be automatically adjusted according to a preset automatic adjustment manner until a preset stop adjustment condition is met;
Wherein the stop adjustment condition may include a user making a gesture corresponding to a stop adjustment operation, a user changing a gesture currently being made, and the like.
For example, for a car seat, a user can adjust the adjustable state of the seat from the fixed state through a gesture of rotating the index finger clockwise, and enable the car seat to automatically reduce the height of the seat back according to a preset adjusting mode, and stop the automatic adjustment of the seat back through making a fist-making gesture after the height of the seat back is proper, so that the height of the seat back can be adjusted to a desired height through the gesture.
In practical applications, the control requirements of users at different cabin positions may be different for the same control device, for example, for a certain cabin seat in a cabin, the users at the cabin seat positions often need to perform fine adjustment on the current cabin seat than the users at other cabin seat positions, so as to achieve more comfortable riding experience, therefore, the interaction logic of the users at the cabin positions on the same control device may be different, for example, the users at the current cabin positions may perform fine control (for example, multi-gear adjustment of 1-10 gears, etc.) on the environment where the current cabin positions are located, and perform coarse control (for example, gear adjustment of 1-3 gears, etc.) on the environment where other cabin positions are located.
In a possible implementation manner, when determining the target control instruction corresponding to the target semantic information, the following steps D1 to D2 may be used:
d1: and determining the cabin position information corresponding to the gesture information.
D2: determining a target control instruction corresponding to the target semantic information based on a preset corresponding relation between the semantic information and the control instruction, wherein the preset corresponding relation is matched with the cabin position information; the corresponding relation between semantic information matched with the position information of different cabins and the control instruction is different.
The corresponding relation between the semantic information and the control instruction can be used for representing interaction logic aiming at the control equipment, and the interaction logic matched with different cabin position information can be different, so that the interaction logic of the control equipment can meet different requirements of users at different cabin positions.
According to the gesture control method provided by the embodiment of the disclosure, the gesture information contained in the image to be identified and collected in the target cabin is determined, and the three-dimensional position information in the gesture information is mapped to the preset plane, so that the target control equipment corresponding to the two-position information can be determined after the mapping to the preset plane, and the target control equipment can be controlled by using the target control instruction corresponding to the target semantic information in the residual gesture information. Like this, compare in traditional mode that directly controls through triggering the car center, can improve the mutual experience of passenger of car cabin back row, and compare in the control mode such as using voice control, carry out gesture control to the controlgear in the car cabin through the three-dimensional position information of hand in the car cabin, can avoid being disturbed by the noise in control process's problem, promote the control accuracy to the controlgear in the car cabin.
In another possible implementation manner, the embodiment of the present disclosure further provides an intelligent driving vehicle, including: an image acquisition device, a data processor, an image display device, and at least one control device; wherein, the liquid crystal display device comprises a liquid crystal display device,
the image acquisition device is used for acquiring an image to be identified in the target vehicle cabin and transmitting the image to be identified to the data processor;
the data processor is configured to process an acquired image to be identified according to the gesture control method according to any one of the embodiments of the present disclosure to obtain a target control instruction, and control a target control device according to the target control instruction;
the image display device is used for displaying the device name of the controlled target control device.
It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.
Based on the same inventive concept, the embodiments of the present disclosure further provide a gesture control device corresponding to the gesture control method, and since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to that of the gesture control method in the embodiments of the present disclosure, the implementation of the device may refer to the implementation of the method, and the repetition is omitted.
Referring to fig. 4, an architecture diagram of a gesture control apparatus according to an embodiment of the disclosure is shown, where the apparatus includes: a first determining module 401, a second determining module 402, a third determining module 403, and a control module 404; wherein, the liquid crystal display device comprises a liquid crystal display device,
the first determining module 401 is configured to obtain an image to be identified acquired in a target cabin, and determine gesture information included in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information;
a second determining module 402, configured to map the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determine, based on the two-dimensional position information, that the gesture information indicates a selected target area in the preset plane;
a third determining module 403, configured to determine a target control device associated with the target area in the target vehicle cabin;
and the control module 404 is used for controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
In a possible implementation manner, the image to be identified contains depth information;
the first determining module 401 is further configured to determine three-dimensional position information of the hand in the target cabin according to the following steps:
Determining image coordinates of target key points of the hand in the image to be identified;
and determining three-dimensional position information of the hand in the target vehicle cabin based on the image coordinates of the target key points and the depth information of the hand.
In a possible implementation manner, the preset plane is determined based on the position of a display device in the target cabin, and the preset plane comprises at least one first area corresponding to the display device and a plurality of second areas defined around the display device;
the second areas are used for controlling different control devices, and the control devices controlled by the second areas are related to the first position of each second area in the preset plane and the second position of each control device in the target cabin.
In a possible implementation manner, after determining the target control device associated with the target area in the target vehicle cabin, the third determining module 403 is further configured to:
displaying a target floating window on the display device, wherein a plurality of control device identifiers are displayed on the target floating window;
highlighting the identification of the target control device.
In a possible implementation manner, in a case where the target control device is a continuously adjustable control device, the control module 404 is further configured to:
responding to the change of the three-dimensional position information in the gesture information, and determining the motion change amount of the hand based on the changed three-dimensional position information; the motion variable quantity comprises a moving distance and/or a moving angle;
determining a state change amount at the time of controlling the target control device based on the motion change amount;
and determining a state control instruction to the target control device based on the state change amount to change the state of the target control device based on the state control instruction.
In a possible implementation manner, the control module 404 is further configured to determine a target control instruction corresponding to the target semantic information according to the following steps:
determining cabin position information corresponding to the gesture information;
determining a target control instruction corresponding to the target semantic information based on a preset corresponding relation between the semantic information and the control instruction, wherein the preset corresponding relation is matched with the cabin position information; the corresponding relation between semantic information matched with the position information of different cabins and the control instruction is different.
According to the gesture control device provided by the embodiment of the disclosure, the gesture information contained in the image to be identified and collected in the target cabin is determined, and the three-dimensional position information in the gesture information is mapped to the preset plane, so that the target control equipment corresponding to the two-position information can be determined after the mapping to the preset plane, and the target control equipment can be controlled by using the target control instruction corresponding to the target semantic information in the residual gesture information. Like this, compare in traditional mode that directly controls through triggering the car center, can improve the mutual experience of passenger of car cabin back row, and compare in the control mode such as using voice control, carry out gesture control to the controlgear in the car cabin through the three-dimensional position information of hand in the car cabin, can avoid being disturbed by the noise in control process's problem, promote the control accuracy to the controlgear in the car cabin.
The process flow of each module in the apparatus and the interaction flow between the modules may be described with reference to the related descriptions in the above method embodiments, which are not described in detail herein.
Based on the same technical concept, the embodiment of the disclosure also provides computer equipment. Referring to fig. 5, a schematic structural diagram of a computer device 500 according to an embodiment of the disclosure includes a processor 501, a memory 502, and a bus 503. The memory 502 is configured to store execution instructions, including a memory 5021 and an external memory 5022; the memory 5021 is also referred to as an internal memory, and is used for temporarily storing operation data in the processor 501 and data exchanged with an external memory 5022 such as a hard disk, the processor 501 exchanges data with the external memory 5022 through the memory 5021, and when the computer device 500 is running, the processor 501 and the memory 502 communicate through the bus 503, so that the processor 501 executes the following instructions:
Acquiring an image to be identified acquired in a target cabin, and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information;
mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information;
determining target control equipment associated with the target area in the target vehicle cabin;
and controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
The disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the gesture control method described in the above method embodiments. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The embodiments of the present disclosure further provide a computer program product, where the computer program product carries program code, and instructions included in the program code may be used to perform the steps of the gesture control method described in the foregoing method embodiments, and specifically reference may be made to the foregoing method embodiments, which are not described herein.
Wherein the above-mentioned computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that: the foregoing examples are merely specific embodiments of the present disclosure, and are not intended to limit the scope of the disclosure, but the present disclosure is not limited thereto, and those skilled in the art will appreciate that while the foregoing examples are described in detail, it is not limited to the disclosure: any person skilled in the art, within the technical scope of the disclosure of the present disclosure, may modify or easily conceive changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features thereof; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the disclosure, and are intended to be included within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims (10)

1. A gesture control method, comprising:
acquiring an image to be identified acquired in a target cabin, and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information;
Mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information;
determining target control equipment associated with the target area in the target vehicle cabin;
and controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
2. The method of claim 1, wherein the image to be identified includes depth information;
the method further comprises determining three-dimensional position information of the hand in the target vehicle cabin according to the following steps:
determining image coordinates of target key points of the hand in the image to be identified;
and determining three-dimensional position information of the hand in the target vehicle cabin based on the image coordinates of the target key points and the depth information of the hand.
3. The method according to claim 1 or 2, wherein the preset plane is determined based on a position of a display device in the target cabin, the preset plane including therein at least one first area corresponding to the display device and a plurality of second areas defined around the display device;
The second areas are used for controlling different control devices, and the control devices controlled by the second areas are related to the first position of each second area in the preset plane and the second position of each control device in the target cabin.
4. A method according to claim 3, wherein after determining a target control device associated with the target zone within the target vehicle cabin, the method further comprises:
displaying a target floating window on the display device, wherein a plurality of control device identifiers are displayed on the target floating window;
highlighting the identification of the target control device.
5. The method according to claim 1, wherein in case the target control device is a continuously adjustable control device, the method further comprises:
responding to the change of the three-dimensional position information in the gesture information, and determining the motion change amount of the hand based on the changed three-dimensional position information; the motion variable quantity comprises a moving distance and/or a moving angle;
determining a state change amount at the time of controlling the target control device based on the motion change amount;
And determining a state control instruction to the target control device based on the state change amount to change the state of the target control device based on the state control instruction.
6. The method of claim 1, further comprising determining a target control instruction corresponding to the target semantic information according to:
determining cabin position information corresponding to the gesture information;
determining a target control instruction corresponding to the target semantic information based on a preset corresponding relation between the semantic information and the control instruction, wherein the preset corresponding relation is matched with the cabin position information; the corresponding relation between semantic information matched with the position information of different cabins and the control instruction is different.
7. An intelligent driving vehicle, comprising: an image acquisition device, a data processor, an image display device, and at least one control device;
the image acquisition device is used for acquiring an image to be identified in the target vehicle cabin and transmitting the image to be identified to the data processor;
the data processor is configured to process an acquired image to be identified according to the gesture control method according to any one of claims 1 to 6 to obtain a target control instruction, and control a target control device according to the target control instruction;
The image display device is used for displaying the device name of the controlled target control device.
8. A gesture control apparatus, comprising:
the first determining module is used for acquiring an image to be identified acquired in the target cabin and determining gesture information contained in the image to be identified; the gesture information comprises three-dimensional position information of the hand in the target car cabin and target semantic information;
the second determining module is used for mapping the three-dimensional position information to a preset plane to obtain two-dimensional position information, and determining that the gesture information indicates a selected target area in the preset plane based on the two-dimensional position information;
a third determining module, configured to determine a target control device associated with the target area in the target vehicle cabin;
and the control module is used for controlling the target control equipment based on a target control instruction corresponding to the target semantic information.
9. A computer device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the computer device is running, the machine-readable instructions when executed by the processor performing the steps of the gesture control method of any one of claims 1 to 6.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps of the gesture control method according to any of claims 1 to 6.
CN202310459387.6A 2023-04-25 2023-04-25 Gesture control method and device, computer equipment and storage medium Pending CN116449959A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310459387.6A CN116449959A (en) 2023-04-25 2023-04-25 Gesture control method and device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310459387.6A CN116449959A (en) 2023-04-25 2023-04-25 Gesture control method and device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116449959A true CN116449959A (en) 2023-07-18

Family

ID=87130065

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310459387.6A Pending CN116449959A (en) 2023-04-25 2023-04-25 Gesture control method and device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116449959A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117962742A (en) * 2024-03-29 2024-05-03 长城汽车股份有限公司 Control method and device of vehicle component and vehicle

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117962742A (en) * 2024-03-29 2024-05-03 长城汽车股份有限公司 Control method and device of vehicle component and vehicle
CN117962742B (en) * 2024-03-29 2024-06-21 长城汽车股份有限公司 Control method and device of vehicle component and vehicle

Similar Documents

Publication Publication Date Title
CN111931579B (en) Automatic driving assistance system and method using eye tracking and gesture recognition techniques
US10409382B2 (en) Smart tutorial for gesture control system
KR101730315B1 (en) Electronic device and method for image sharing
CN110045825A (en) Gesture recognition system for vehicle interaction control
EP2258587A1 (en) Operation input device for vehicle
US9493125B2 (en) Apparatus and method for controlling of vehicle using wearable device
CN110487296A (en) Calculation method, device, motor vehicle and the program that " augmented reality " shows
CN113330395B (en) Multi-screen interaction method and device, terminal equipment and vehicle
CA2914712C (en) Gesture input apparatus for car navigation system
CN116449959A (en) Gesture control method and device, computer equipment and storage medium
KR20200093091A (en) Terminal device, vehicle having the same and method for controlling the same
CN110015236A (en) A kind of vehicle display device, method and vehicle
CN112162688A (en) Vehicle-mounted virtual screen interactive information system based on gesture recognition
CN111891037A (en) Cockpit lighting control method, device, equipment and storage medium
CN109484328A (en) The user's interface device of vehicle
EP3659848A1 (en) Operating module, operating method, operating system and storage medium for vehicles
US20210382560A1 (en) Methods and System for Determining a Command of an Occupant of a Vehicle
US20240211126A1 (en) Human-machine interaction method, electronic device and storage medium
CN112667084B (en) Control method and device for vehicle-mounted display screen, electronic equipment and storage medium
CN113492756A (en) Method, device, equipment and storage medium for displaying vehicle external information
WO2015153835A1 (en) Systems and methods for the detection of implicit gestures
CN112383826A (en) Control method and device of vehicle-mounted entertainment terminal, storage medium, terminal and automobile
CN115848138A (en) Cabin visual angle switching method, device and equipment and vehicle
CN111469663A (en) Control system for a vehicle
CN112771480B (en) Interaction method and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination