CN112905003A - Intelligent cockpit gesture control method and device and storage medium - Google Patents
Intelligent cockpit gesture control method and device and storage medium Download PDFInfo
- Publication number
- CN112905003A CN112905003A CN202110084053.6A CN202110084053A CN112905003A CN 112905003 A CN112905003 A CN 112905003A CN 202110084053 A CN202110084053 A CN 202110084053A CN 112905003 A CN112905003 A CN 112905003A
- Authority
- CN
- China
- Prior art keywords
- gesture
- recognized
- preset
- sub
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Abstract
The application relates to a gesture control method, a gesture control device and a storage medium for an intelligent cockpit, wherein the method comprises the steps of obtaining an image sequence to be recognized; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one; sequentially recognizing a first gesture and a second gesture from a preset frame number of an image sequence to be recognized; determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture; tracking the second gesture from the first frame image of the recognized second gesture, and recognizing a motion track corresponding to the second gesture; and determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction. Therefore, the control range of gesture operation can be greatly expanded in the intelligent cabin, and interaction experience is improved.
Description
Technical Field
The application relates to the technical field of automobiles, in particular to a gesture control method and device for an intelligent cabin and a storage medium.
Background
In recent years, the intelligent automobile cabin is an important research field of various large host factories at home and abroad. The intelligent cockpit is embodied in many aspects, and one important aspect is the innovation of man-machine interaction, and the touch screen is no longer used as an operation core.
At present, the innovation of the intelligent cockpit on voice interaction is leap forward, and an important stage achievement is achieved. Like voice interaction, gesture interaction is a natural human interaction mode, is the second major channel for human to output information outwards in a natural mode, and is also the most important supplementary communication means under the condition of unsmooth language communication (such as sign language of deaf-mutes and gesture stroke when the language is obstructed).
At present, gesture operation in an automobile cabin mainly takes static gesture recognition and single dynamic gesture recognition as main functions, the gesture state is simple, and an operation command corresponding to an automobile is single; some gestures are not very common and are not in accordance with the intuition of human operation.
Disclosure of Invention
The embodiment of the application provides a gesture control method and device for an intelligent cabin and a storage medium, and the control range of gesture operation can be greatly expanded in the intelligent cabin, so that interactive experience is improved.
On one hand, the embodiment of the application provides an intelligent cockpit gesture control method, which comprises the following steps:
acquiring an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one;
sequentially recognizing a first gesture and a second gesture from a preset frame number of an image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value;
determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture;
tracking the second gesture from the first frame image of the recognized second gesture, and recognizing a motion track corresponding to the second gesture;
and determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction.
Optionally, after the unit to be controlled is controlled according to the control instruction, the method further includes:
recognizing a third gesture from the remaining images of the image sequence to be recognized; the third gesture corresponds to any one or more of the plurality of sub-regions; the similarity degree value of the third gesture and the second gesture is smaller than a second preset value; the residual images are images except for images corresponding to the preset frame number and images used for tracking the second gesture in the image sequence to be identified;
and determining a release instruction according to the third gesture, and exiting the state of controlling the unit to be controlled according to the release instruction.
Optionally, the first gesture and the second gesture are sequentially recognized from a preset frame number of the image sequence to be recognized, including:
acquiring a trained first gesture recognition model;
sequentially inputting preset frame numbers of an image sequence to be recognized into a first gesture recognition model to perform first gesture recognition;
and when the first gesture is recognized, sequentially inputting a next frame of image of the currently recognized first gesture and an image after the next frame of image into the obtained trained second gesture recognition model for second gesture recognition until a second gesture is recognized.
Optionally, tracking the second gesture from the first frame image of the recognized second gesture, and recognizing an activity track corresponding to the second gesture, including:
determining the position of the second gesture in each frame image from the first frame image in which the second gesture is recognized;
determining an activity track to be recognized according to the position of the second gesture in each frame of image;
acquiring a preset activity track set;
and determining an active track matched with the active track to be recognized from the preset active track set, and determining the active track matched with the active track to be recognized as the active track corresponding to the second gesture.
Optionally, the gesture collecting unit includes a camera; the preset area is positioned above the auxiliary instrument desk in the intelligent cabin;
acquiring an image sequence to be identified, comprising:
when the intelligent cockpit enters a gesture control state, the camera continuously shoots the upper part of the auxiliary instrument desk in the intelligent cockpit or shoots the upper part of the auxiliary instrument desk according to a set frequency to obtain an image sequence to be recognized.
Optionally, the plurality of steerable units includes a co-pilot seat; the sub-area corresponding to the first gesture corresponds to the co-driver seat;
the movable track comprises any one or a combination of more of moving to the first direction, moving to the second direction, moving to the third direction, moving to the fourth direction, moving to the fifth direction, moving to the sixth direction, overturning forwards and overturning backwards;
determining a manipulation instruction based on the activity trajectory, comprising:
when the movable track moves towards the first direction, determining that the control command is to move the front passenger seat forwards;
or; when the active trajectory is forward turning, determining that the manipulation command is to tilt the backrest of the front passenger seat forward.
Optionally, the first gesture and the third gesture are both palm opening, and the second gesture is palm fist making.
Optionally, the preset area is located above the rear control console of the intelligent cabin;
acquiring an image sequence to be identified, comprising:
when the intelligent cockpit enters the gesture control state, the gesture collection unit is used for continuously shooting the upper part of a console at the back row of the intelligent cockpit or shooting the upper part of the console according to the set frequency, and an image sequence to be recognized is obtained.
On the other hand, the embodiment of the application provides an intelligence passenger cabin gesture controlling means, includes:
the acquisition module is used for acquiring an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one;
the first recognition module is used for sequentially recognizing a first gesture and a second gesture from preset frame numbers of the image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value;
the first determining module is used for determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture;
the second recognition module is used for tracking the second gesture from the first frame image of the second gesture, and recognizing a motion track corresponding to the second gesture;
and the second determining module is used for determining a control instruction based on the activity track and controlling the unit to be controlled according to the control instruction.
In another aspect, an embodiment of the present application provides a computer storage medium, where at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the above intelligent cockpit gesture control method.
The method, the device and the storage medium for controlling the gestures of the intelligent cabin have the following beneficial effects:
acquiring an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one; sequentially recognizing a first gesture and a second gesture from a preset frame number of an image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value; determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture; tracking the second gesture from the first frame image of the recognized second gesture, and recognizing a motion track corresponding to the second gesture; and determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction. Therefore, the control range of gesture operation can be greatly expanded in the intelligent cabin, and interaction experience is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of an application scenario of an intelligent cabin provided in an embodiment of the present application;
fig. 2 is a schematic flowchart of a gesture control method for an intelligent cockpit according to an embodiment of the present application;
fig. 3 is a schematic diagram of a preset region and a sub-region in an image to be recognized according to an embodiment of the present disclosure;
FIG. 4 is a diagram illustrating a specific gesture provided by an embodiment of the present application;
FIG. 5 is a schematic diagram of a gesture recognition result provided in an embodiment of the present application;
fig. 6 is a schematic structural diagram of an intelligent cockpit gesture control device provided in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, fig. 1 is a schematic view of an application scenario of an intelligent cabin provided in an embodiment of the present application, a gesture acquisition unit 101 is disposed in the intelligent cabin, and the gesture acquisition unit 101 is configured to acquire a preset area 102 and input acquired information to a calculation unit for calculation and recognition.
It should be noted that, in fig. 1, the gesture collecting unit 101 is disposed above the central control display screen, the preset area 102 is located above the central control console, and the right hand movement area of the driver is located; in other embodiments, the gesture collecting unit 101 may also be disposed at other positions of the intelligent cabin, such as the rear side of the front central armrest, according to actual requirements, so as to detect and collect a preset area above the rear central armrest.
Acquiring a preset area 102 within preset time through a gesture acquisition unit 101 to obtain an image sequence to be recognized; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one; then, sequentially recognizing a first gesture and a second gesture from a preset frame number of the image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value; then, determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture; tracking the second gesture from the first frame image of the recognized second gesture, and recognizing a motion track corresponding to the second gesture; and finally, determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction.
Optionally, the plurality of controllable units in the intelligent cabin include a seat controller, a window controller, an electric door controller, an air-conditioning outlet controller, a streaming media rearview mirror, an instrument controller, and the like.
The following describes a specific embodiment of an intelligent cockpit gesture control method according to the present application, and fig. 2 is a schematic flowchart of an intelligent cockpit gesture control method according to the embodiment of the present application, where the present specification provides the method operation steps as in the embodiment or the flowchart, but more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In practice, the system or server product may be implemented in a sequential or parallel manner (e.g., parallel processor or multi-threaded environment) according to the embodiments or methods shown in the figures. Specifically, as shown in fig. 2, the method may include:
s201: acquiring an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one.
In the embodiment of the application, the gesture collection unit is arranged in the intelligent cabin, the image sequence to be recognized, which is acquired by collecting the preset area in the preset time, is obtained through the gesture collection unit, the main purpose of limiting the preset time is that only gesture information collected in the preset time is analyzed, and the gesture information collected in the preset time is considered as invalid information. The preset area can be characterized as a three-dimensional detection space in a three-dimensional space, namely, the gesture collection unit only identifies gesture actions falling into the three-dimensional detection space, the multiple sub-areas are obtained by dividing the three-dimensional detection space, the multiple sub-areas correspond to the multiple controllable units in the intelligent cabin one by one, and the controllable units refer to components or equipment which can be controlled by a controller in the intelligent cabin, namely, each sub-area controls one component or equipment; the volume sizes of the sub-regions can be the same or different according to actual requirements. The preset area represents the image size of the image sequence to be identified in the dimension of the two-dimensional image, and each subarea in the plurality of subareas corresponds to the same part in each frame of image of the image sequence to be identified; for example, as shown in fig. 3, fig. 3 is a schematic diagram of a preset region and a sub-region in an image to be recognized provided in an embodiment of the present application, the sub-region a corresponds to a front passenger seat in an intelligent cabin, the sub-region b corresponds to air conditioning equipment in the intelligent cabin, the sub-region c corresponds to a front passenger window in the intelligent cabin, and the sub-region d and the like may correspond to components or equipment in other intelligent cabins according to actual requirements; the sub-areas a, b, c and d correspond to different monitoring spaces in a three-dimensional space; it should be noted that fig. 3 only shows a division manner of neutron regions in an elliptical image, and may also be set as a 9-grid or other irregular division manner according to actual requirements.
In an optional embodiment, the gesture collecting unit comprises a camera; the preset area is positioned above an auxiliary instrument desk in the intelligent cabin, and the area above the auxiliary instrument desk can be understood as a right-hand activity area of a driver, so that the driver can conveniently perform unfolding operation and control; step S201 may specifically include: when the intelligent cockpit enters a gesture control state, the camera continuously shoots the upper part of the auxiliary instrument desk in the intelligent cockpit or shoots the upper part of the auxiliary instrument desk according to a set frequency to obtain an image sequence to be recognized. The judgment of entering the gesture control state of the intelligent cockpit can be carried out based on an entity key or other non-contact modes such as voice and the like.
In another optional implementation mode, the preset area is located above a rear-row console of the intelligent cabin, namely above a central armrest of a rear-row seat, so that a rear-row passenger can conveniently control the intelligent cabin; step S201 may specifically include: when the intelligent cockpit enters the gesture control state, the gesture collection unit is used for continuously shooting the upper part of a console at the back row of the intelligent cockpit or shooting the upper part of the console according to the set frequency, and an image sequence to be recognized is obtained.
S203: sequentially recognizing a first gesture and a second gesture from a preset frame number of an image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value.
In the embodiment of the application, the first gesture represents selection of the controllable unit, switching from the first gesture to the second gesture represents selection of the current controllable unit, and the first gesture and the second gesture are necessarily two gestures that can be recognized by a computer, so that a similarity degree value of the first gesture and the second gesture needs to be less than or equal to a first preset value; in an extreme case, the first preset value is 0, that is, the first gesture and the second gesture are completely different gestures; the sensitivity and accuracy of recognition can be adjusted by setting the first preset value, the greater the first preset value is, the more sensitive the recognition is but the accuracy is low, and the smaller the first preset value is, the less the recognition sensitivity is but the accuracy is high. The preset frame number is used for limiting the operation duration of the user, a first gesture and a second gesture are sequentially recognized from the preset frame number of the image sequence to be recognized, the first gesture and the second gesture are used by the user continuously, and meanwhile, the second gesture is the same as a sub-region corresponding to the first gesture.
In an optional implementation manner, the step S203 may specifically include: acquiring a trained first gesture recognition model; sequentially inputting preset frame numbers of an image sequence to be recognized into a first gesture recognition model to perform first gesture recognition; and when the first gesture is recognized, sequentially inputting a next frame of image of the currently recognized first gesture and an image after the next frame of image into the obtained trained second gesture recognition model for second gesture recognition until a second gesture is recognized.
Specifically, a training image of a first gesture is collected, and a first gesture recognition model is trained to obtain a trained first gesture recognition model; collecting a training image of a second gesture, and training a second gesture recognition model to obtain a trained second gesture recognition model; and identifying a first gesture and a second gesture from preset frame numbers of the image sequence to be identified based on an image detection algorithm.
S205: and determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture.
In the embodiment of the application, a unit to be manipulated is determined from the multiple manipulatable units according to the first gesture, the second gesture, and the sub-region corresponding to the first gesture, for example, if the sub-region corresponding to the first gesture and the second gesture is the sub-region a in fig. 3, the passenger seat is determined as the unit to be manipulated.
S207: and tracking the second gesture from the first frame image of which the second gesture is recognized, and recognizing an activity track corresponding to the second gesture.
S209: and determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction.
In the embodiment of the application, after an image containing a second gesture is recognized from a to-be-recognized image, the second gesture is tracked from a first frame image of the recognized second gesture, an activity track corresponding to the second gesture is recognized, and the activity track corresponding to the second gesture represents the operation intention of a user, namely the user wants to control a certain component or device to realize a function, so that an operation command is determined according to the activity track, and an operation unit is operated according to the operation command.
In an optional implementation manner, the step S207 may specifically include: determining the position of the second gesture in each frame of image from the first frame of image in which the second gesture is recognized; determining an activity track to be recognized according to the position of the second gesture in each frame of image; acquiring a preset activity track set; and determining an active track matched with the active track to be recognized from the preset active track set, and determining the active track matched with the active track to be recognized as the active track corresponding to the second gesture.
Specifically, the preset active track set may include any one or a combination of a first direction movement, a second direction movement, a third direction movement, a fourth direction movement, a fifth direction movement, a sixth direction movement, a forward turning and a backward turning.
In an alternative embodiment, the plurality of steerable units includes a passenger seat; the sub-area corresponding to the first gesture corresponds to a front passenger seat, namely the unit to be controlled is the front passenger seat; when the movable track moves towards the first direction, determining that the control command is to move the front passenger seat forwards; or; when the active trajectory is forward turning, determining that the manipulation command is to tilt the backrest of the front passenger seat forward.
In an optional embodiment, after step S209, the method may further include: recognizing a third gesture from the remaining images of the image sequence to be recognized; the third gesture corresponds to any one or more of the plurality of sub-regions; the similarity degree value of the third gesture and the second gesture is smaller than a second preset value; the residual images are images except for images corresponding to the preset frame number and images used for tracking the second gesture in the image sequence to be identified; and determining a release instruction according to the third gesture, and exiting the state of controlling the unit to be controlled according to the release instruction. Wherein the third gesture may be the same as the first gesture.
Specifically, as shown in fig. 4, the first gesture and the third gesture are both palm open, and the second gesture is palm fist making. The following describes steps S201 to S209 and some optional embodiments by a specific example. In the application scenario shown in fig. 1, a to-be-recognized image sequence is obtained by capturing a preset region 102 in a preset time through a gesture capturing unit 101, and the results of sequentially recognizing a first gesture (palm opening) and a second gesture (palm fist) from preset frame numbers of the to-be-recognized image sequence are shown in fig. 5, where t0 and t1 frames show that the palm opening is in a sub-region a, which indicates that the to-be-recognized image sequence extends to a passenger seat, and similarly, a second gesture (palm fist) is detected in the sub-region a, which indicates that the to-be-controlled unit is selected, and then the to-be-controlled unit is determined to be the passenger seat corresponding to the sub-region a; if the moving track of the second gesture (palm fist) recognized according to the t 3-tn frame images moves towards the first direction (front), determining that the control command is to move the passenger seat forwards; similarly, when the moving track of the second gesture (palm fist) recognized by the t 3-tn images moves towards the second direction (back), determining that the control command is to move the passenger seat backwards; when the moving track of the second gesture (palm fist) recognized by the t 3-tn images is backward turning, determining that the control instruction is that the seat back of the front passenger is backward inclined; and when a third gesture (palm opening) is recognized in the tn +1 frame, generating a release instruction, and exiting the state of controlling the passenger seat according to the release instruction.
In fig. 5, the front, rear, left, and right represent the movement in the first direction, the movement in the second direction, the movement in the third direction, and the movement in the fourth direction, respectively, and the movement in the fifth direction and the movement in the sixth direction may be determined according to the change in the size of the second gesture in the t3 to tn frame images, for example, when the proportion of the first in the tn frame image is larger than the proportion of the first in the t3 frame image, the movement in the fifth direction is indicated, and when the proportion of the first in the tn frame image is smaller than the proportion of the first in the t3 frame image, the movement in the sixth direction is indicated.
The control passenger seat is similarly suitable for controlling air-conditioning equipment, passenger windows, passenger doors and the like in the intelligent cabin; for example, when the moving track of the second gesture (palm fist) recognized by the t 3-tn frames of images moves to the third direction (left), determining that the control instruction is to adjust the air conditioner air outlet direction to the left; when the moving track of a second gesture (palm fist) identified by t 3-tn frame images moves towards the fourth direction (right), determining that the control instruction is to adjust the air outlet direction of the air conditioner towards the right; when the moving track of the second gesture (palm fist) recognized by the t 3-tn frame images moves towards the fifth direction, determining that the air conditioning air volume is reduced by the control instruction; for another example, when the moving track of the second gesture (palm fist) recognized by the t3 to tn frame images moves to the first direction (front), the control command is determined to move the passenger car window glass upwards (the window is closed); when the moving track of a second gesture (palm fist) recognized by the t 3-tn frame images moves towards a second direction (back), determining that the control instruction is to move the copilot window glass downwards (the window is opened); for another example, when the motion trajectory of the second gesture (palm fist) recognized by the t3 to tn frame images is to move to the fourth direction (right), it is determined that the control command is to open the electric copilot door; other electric control components or equipment of the whole vehicle can be operated according to the operation, and the detailed description is omitted. Therefore, the gesture control method for the intelligent cabin, provided by the embodiment of the application, can greatly expand the control range of gesture operation and improve interaction experience in the automobile cabin.
The method provided by the embodiment of the application can be executed in a computer terminal, a server or a similar operation device.
An embodiment of the present application further provides an intelligent cockpit gesture control device, fig. 6 is a schematic structural diagram of the intelligent cockpit gesture control device provided in the embodiment of the present application, and as shown in fig. 6, the device includes:
an obtaining module 601, configured to obtain an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one;
the first recognition module 602 is configured to sequentially recognize a first gesture and a second gesture from preset frame numbers of an image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value;
the first determining module 603 is configured to determine a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture, and the sub-region corresponding to the first gesture;
the second recognition module 604 is configured to track the second gesture from the first frame image in which the second gesture is recognized, and recognize an activity track corresponding to the second gesture;
the second determining module 605 is configured to determine a control instruction based on the activity track, and control the unit to be controlled according to the control instruction.
The device and method embodiments in the embodiments of the present application are based on the same application concept.
In an optional embodiment, the apparatus further includes a third determining module, configured to identify a third gesture from remaining images of the image sequence to be identified; the third gesture corresponds to any one or more of the plurality of sub-regions; the similarity degree value of the third gesture and the second gesture is smaller than a second preset value; the residual images are images except for images corresponding to the preset frame number and images used for tracking the second gesture in the image sequence to be identified; and determining a release instruction according to the third gesture, and exiting the state of controlling the unit to be controlled according to the release instruction.
In an optional implementation manner, the first identifying module 602 is specifically configured to: acquiring a trained first gesture recognition model; sequentially inputting preset frame numbers of an image sequence to be recognized into a first gesture recognition model to perform first gesture recognition; and when the first gesture is recognized, sequentially inputting a next frame of image of the currently recognized first gesture and an image after the next frame of image into the obtained trained second gesture recognition model for second gesture recognition until a second gesture is recognized.
In an optional implementation manner, the second identifying module 604 is specifically configured to: determining the position of the second gesture in each frame image from the first frame image in which the second gesture is recognized; determining an activity track to be recognized according to the position of the second gesture in each frame of image; acquiring a preset activity track set; and determining an active track matched with the active track to be recognized from the preset active track set, and determining the active track matched with the active track to be recognized as the active track corresponding to the second gesture.
In an optional embodiment, the gesture collecting unit comprises a camera; the preset area is positioned above the auxiliary instrument desk in the intelligent cabin; the obtaining module 601 is specifically configured to: when the intelligent cockpit enters a gesture control state, the camera continuously shoots the upper part of the auxiliary instrument desk in the intelligent cockpit or shoots the upper part of the auxiliary instrument desk according to a set frequency to obtain an image sequence to be recognized.
In an alternative embodiment, the plurality of steerable units includes a passenger seat; the sub-area corresponding to the first gesture corresponds to the co-driver seat; the movable track comprises any one or a combination of more of moving to the first direction, moving to the second direction, moving to the third direction, moving to the fourth direction, moving to the fifth direction, moving to the sixth direction, overturning forwards and overturning backwards; the second determining module 605 is specifically configured to: when the movable track moves towards the first direction, determining that the control command is to move the front passenger seat forwards; or; when the active trajectory is forward turning, determining that the manipulation command is to tilt the backrest of the front passenger seat forward.
In an alternative embodiment, the first gesture and the third gesture are both palm open, and the second gesture is palm fist making.
In an optional implementation mode, the preset area is located above a rear control console of the intelligent cabin; the obtaining module 601 is specifically configured to: when the intelligent cockpit enters the gesture control state, the gesture collection unit is used for continuously shooting the upper part of a console at the back row of the intelligent cockpit or shooting the upper part of the console according to the set frequency, and an image sequence to be recognized is obtained.
Embodiments of the present application further provide a storage medium, which may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing an intelligent cockpit gesture control method in the method embodiments, where the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the intelligent cockpit gesture control method.
Alternatively, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
According to the embodiment of the gesture control method, the gesture control equipment or the gesture control storage medium for the intelligent cockpit, an image sequence to be recognized is obtained; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the plurality of sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one; sequentially recognizing a first gesture and a second gesture from a preset frame number of an image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value; determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture; tracking the second gesture from the first frame image of the recognized second gesture, and recognizing a motion track corresponding to the second gesture; and determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction. Therefore, the control range of gesture operation can be greatly expanded in the intelligent cabin, and interaction experience is improved.
It should be noted that: the sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (10)
1. An intelligent cockpit gesture control method is characterized by comprising the following steps:
acquiring an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one;
sequentially recognizing a first gesture and a second gesture from a preset frame number of the image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value;
determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture;
tracking the second gesture from the first frame image of the second gesture, and identifying a motion track corresponding to the second gesture;
and determining a control instruction based on the activity track, and controlling the unit to be controlled according to the control instruction.
2. The method according to claim 1, wherein after the manipulating the unit to be manipulated according to the manipulation instruction, the method further comprises:
recognizing a third gesture from the rest images of the image sequence to be recognized; the third gesture corresponds to any one or more of the plurality of sub-regions; the similarity degree value of the third gesture and the second gesture is smaller than a second preset value; the residual images are images except for the image corresponding to the preset frame number and the image used for tracking the second gesture in the image sequence to be identified;
and determining a release instruction according to the third gesture, and exiting from the state of controlling the unit to be controlled according to the release instruction.
3. The method according to claim 1, wherein the sequentially recognizing the first gesture and the second gesture from the preset number of frames of the image sequence to be recognized comprises:
acquiring a trained first gesture recognition model;
sequentially inputting the preset frame number of the image sequence to be recognized into the first gesture recognition model to perform first gesture recognition;
and when the first gesture is recognized, sequentially inputting a next frame of image of the currently recognized first gesture and an image after the next frame of image into the obtained trained second gesture recognition model for second gesture recognition until the second gesture is recognized.
4. The method according to claim 1 or 3, wherein tracking the second gesture from the first image in which the second gesture is recognized, and recognizing an activity track corresponding to the second gesture comprises:
determining the position of the second gesture in each frame of image from the first frame of image in which the second gesture is recognized;
determining an activity track to be recognized according to the position of the second gesture in each frame of image;
acquiring a preset activity track set;
and determining an activity track matched with the activity track to be recognized from the preset activity track set, and determining the activity track matched with the activity track to be recognized as an activity track corresponding to the second gesture.
5. The method of claim 1, wherein the gesture capture unit comprises a camera; the preset area is positioned above an auxiliary instrument desk in the intelligent cabin;
the acquiring of the image sequence to be recognized includes:
when the intelligent cockpit enters a gesture control state, shooting the upper part of the auxiliary instrument desk in the intelligent cockpit continuously or according to a set frequency through the camera to obtain the image sequence to be recognized.
6. The method of claim 1, wherein the plurality of steerable units comprises a co-pilot seat; the sub-area corresponding to the first gesture corresponds to the co-driver seat;
the moving track comprises any one or a combination of more of moving to a first direction, moving to a second direction, moving to a third direction, moving to a fourth direction, moving to a fifth direction, moving to a sixth direction, overturning forwards and overturning backwards;
the determining a manipulation instruction based on the activity track includes:
when the movable track is the movement towards the first direction, determining that the control command is to move the front passenger seat forwards;
or; when the movable track is the forward overturning, determining that the control command is to incline the backrest of the front passenger seat forwards.
7. The method of claim 2, wherein the first gesture and the third gesture are both palm open and the second gesture is a palm fist.
8. The method of claim 1, wherein the preset area is located above a rear row console of the smart car;
the acquiring of the image sequence to be recognized includes:
when the intelligent cabin enters a gesture control state, shooting is continuously carried out above a rear control console of the intelligent cabin or according to a set frequency through the gesture collection unit, and the image sequence to be recognized is obtained.
9. An intelligent cockpit gesture control device, comprising:
the acquisition module is used for acquiring an image sequence to be identified; the image sequence to be recognized is obtained by acquiring a preset area within preset time by a gesture acquisition unit; the preset area comprises a plurality of sub-areas, and the sub-areas correspond to the plurality of controllable units in the intelligent cockpit one by one;
the first recognition module is used for sequentially recognizing a first gesture and a second gesture from the preset frame number of the image sequence to be recognized; the first gesture corresponds to any one of the plurality of sub-regions; the second gesture is the same as the sub-region corresponding to the first gesture; the similarity degree value of the first gesture and the second gesture is smaller than or equal to a first preset value;
the first determining module is used for determining a unit to be controlled from the plurality of controllable units according to the first gesture, the second gesture and the sub-area corresponding to the first gesture;
the second recognition module is used for tracking the second gesture from the first frame image of the second gesture is recognized, and recognizing an activity track corresponding to the second gesture;
and the second determining module is used for determining a control instruction based on the activity track and controlling the unit to be controlled according to the control instruction.
10. A computer storage medium, characterized in that at least one instruction or at least one program is stored in the storage medium, which is loaded and executed by a processor to implement the intelligent cockpit gesture control method according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110084053.6A CN112905003A (en) | 2021-01-21 | 2021-01-21 | Intelligent cockpit gesture control method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110084053.6A CN112905003A (en) | 2021-01-21 | 2021-01-21 | Intelligent cockpit gesture control method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112905003A true CN112905003A (en) | 2021-06-04 |
Family
ID=76118201
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110084053.6A Pending CN112905003A (en) | 2021-01-21 | 2021-01-21 | Intelligent cockpit gesture control method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112905003A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113253849A (en) * | 2021-07-01 | 2021-08-13 | 湖北亿咖通科技有限公司 | Display control method, device and equipment of control bar |
| CN115424356A (en) * | 2022-11-07 | 2022-12-02 | 科大讯飞股份有限公司 | Gesture interaction method and device in cabin |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105190477A (en) * | 2013-03-21 | 2015-12-23 | 索尼公司 | Head-mounted device for user interactions in an amplified reality environment |
| CN105353634A (en) * | 2015-11-30 | 2016-02-24 | 北京地平线机器人技术研发有限公司 | Household appliance and method for controlling operation by gesture recognition |
| CN111580652A (en) * | 2020-05-06 | 2020-08-25 | Oppo广东移动通信有限公司 | Control method and device for video playing, augmented reality equipment and storage medium |
| CN111857345A (en) * | 2020-07-23 | 2020-10-30 | 上海纯米电子科技有限公司 | Gesture-based control method and device |
| CN111857356A (en) * | 2020-09-24 | 2020-10-30 | 深圳佑驾创新科技有限公司 | Method, device, equipment and storage medium for recognizing interaction gesture |
-
2021
- 2021-01-21 CN CN202110084053.6A patent/CN112905003A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105190477A (en) * | 2013-03-21 | 2015-12-23 | 索尼公司 | Head-mounted device for user interactions in an amplified reality environment |
| CN105353634A (en) * | 2015-11-30 | 2016-02-24 | 北京地平线机器人技术研发有限公司 | Household appliance and method for controlling operation by gesture recognition |
| CN111580652A (en) * | 2020-05-06 | 2020-08-25 | Oppo广东移动通信有限公司 | Control method and device for video playing, augmented reality equipment and storage medium |
| CN111857345A (en) * | 2020-07-23 | 2020-10-30 | 上海纯米电子科技有限公司 | Gesture-based control method and device |
| CN111857356A (en) * | 2020-09-24 | 2020-10-30 | 深圳佑驾创新科技有限公司 | Method, device, equipment and storage medium for recognizing interaction gesture |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113253849A (en) * | 2021-07-01 | 2021-08-13 | 湖北亿咖通科技有限公司 | Display control method, device and equipment of control bar |
| CN115424356A (en) * | 2022-11-07 | 2022-12-02 | 科大讯飞股份有限公司 | Gesture interaction method and device in cabin |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9725098B2 (en) | Vehicle driver identification | |
| US9335826B2 (en) | Method of fusing multiple information sources in image-based gesture recognition system | |
| US9965169B2 (en) | Systems, methods, and apparatus for controlling gesture initiation and termination | |
| CN112905003A (en) | Intelligent cockpit gesture control method and device and storage medium | |
| EP3013621A1 (en) | Motor vehicle control interface with gesture recognition | |
| US20140168061A1 (en) | System and method for manipulating user interface in vehicle using finger valleys | |
| DE102011053449A1 (en) | Man-machine interface on finger-pointer and gesture-based for vehicles | |
| KR101490908B1 (en) | System and method for providing a user interface using hand shape trace recognition in a vehicle | |
| CN108919959B (en) | Vehicle human-computer interaction method and system | |
| CN112905004B (en) | Gesture control method and device for vehicle-mounted display screen and storage medium | |
| EP3486805A1 (en) | Method for operating an electronic device or an application and corresponding device | |
| US20230078074A1 (en) | Methods and devices for hand-on-wheel gesture interaction for controls | |
| Zobl et al. | A usability study on hand gesture controlled operation of in-car devices | |
| EP3488320B1 (en) | Device and method for assisting a user | |
| KR20210057358A (en) | Gesture recognition method and gesture recognition device performing the same | |
| WO2014108150A2 (en) | User interface for handwritten character input in a device | |
| DE102018220693B4 (en) | Control system and method for controlling a function of a vehicle, and vehicle with such | |
| DE102013000081B4 (en) | Operator interface for contactless selection of a device function | |
| Sucar et al. | From HCI to HRI-usability inspection in multimodal human-robot interactions | |
| EP3025214B1 (en) | Method for operating an input device, and input device | |
| EP3966064A1 (en) | Method for operating an operating system in a vehicle and operating system in a vehicle | |
| Le et al. | Multimodal smart interactive presentation system | |
| EP3426516B1 (en) | Operating device and method for detecting a user selection of at least one operating function of the operating device | |
| DE102019131944A1 (en) | Method for controlling at least one display unit, motor vehicle and computer program product | |
| WO2021004681A1 (en) | Method for operating an operator control system, and operator control system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |