CN116627260A - Method and device for idle operation, computer equipment and storage medium - Google Patents
Method and device for idle operation, computer equipment and storage medium Download PDFInfo
- Publication number
- CN116627260A CN116627260A CN202310909359.XA CN202310909359A CN116627260A CN 116627260 A CN116627260 A CN 116627260A CN 202310909359 A CN202310909359 A CN 202310909359A CN 116627260 A CN116627260 A CN 116627260A
- Authority
- CN
- China
- Prior art keywords
- gesture
- blank
- area
- image sequence
- space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000000007 visual effect Effects 0.000 claims abstract description 85
- 230000004044 response Effects 0.000 claims abstract description 27
- 238000004590 computer program Methods 0.000 claims description 23
- 238000006073 displacement reaction Methods 0.000 claims description 23
- 230000004888 barrier function Effects 0.000 claims description 6
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 5
- 238000010191 image analysis Methods 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 101150104012 TOP2 gene Proteins 0.000 description 1
- 210000003811 finger Anatomy 0.000 description 1
- 210000005224 forefinger Anatomy 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Abstract
The application relates to a method, a device, computer equipment and a storage medium for isolating operation, wherein the method comprises the steps of acquiring a first image sequence, and images in the first image sequence comprise a first isolating gesture of a user; obtaining a track shape of the first blank gesture according to the first image sequence; determining a blank operation area according to the track shape; acquiring a second image sequence, wherein images in the second image sequence comprise a second blank gesture of a user; in response to the operation of the second space-apart gesture in the space-apart operation area, an operation instruction of the visual interface is sent.
Description
Technical Field
The present application relates to the field of electronic technology, and in particular, to a method, an apparatus, a computer device, and a storage medium for performing a space-saving operation.
Background
The space gesture operation is a technology which enables a user to operate without touching a screen or equipment by means of a sensor or a computer vision technology and through motion recognition of hands or bodies, and the user can interact more conveniently and rapidly through the space gesture operation, so that usability and user experience of the equipment are improved.
The operation experience of the rear-row passengers of the vehicle cabin on the central control large screen of the vehicle can be greatly improved through the operation of the space-isolation gestures, but more interference exists for the narrow space scene of the vehicle cabin, for example, the recognition effect can be influenced by irrelevant actions of other passengers, actions of the outside passersby of the vehicle windows and the like.
Disclosure of Invention
Based on the above, a method, a device, a computer device and a storage medium for isolating operation are provided, so that the problem of poor anti-interference performance of isolating operation in the prior art is solved.
In one aspect, a method of operation in a barrier is provided, the method comprising:
acquiring a first image sequence, wherein images in the first image sequence comprise a first blank gesture of a user;
obtaining a track shape of the first blank gesture according to the first image sequence;
determining a blank operation area according to the track shape;
acquiring a second image sequence, wherein images in the second image sequence comprise a second blank gesture of a user;
and responding to the operation of the second air-stop gesture in the air-stop operation area, and sending an operation instruction of a visual interface.
In one embodiment, the second spaced apart gesture includes one of a cursor control gesture, a swipe gesture, and a drag gesture; the operation responsive to the second blank gesture located within the blank operation area includes:
determining a blank operation proportion according to the interface size of the visual interface and the pixel size of the blank operation area in an image space;
and converting the operation of the second blank gesture according to the blank operation proportion so as to obtain an operation instruction of the visual interface.
In one embodiment, the determining the proportion of the idle operation includes:
determining a width ratio according to the transverse dimension of the blank operation area in the image space and the transverse dimension of the visual interface;
and determining the height ratio according to the longitudinal dimension of the blank operation area in the image space and the longitudinal dimension of the visual interface.
In one embodiment, the converting the operation of the second blank gesture according to the blank operation proportion includes:
and determining the coordinates of the response points on the visual interface according to the space operation proportion according to the coordinates of the second space gesture in the space operation area, so as to respond to the operation of the second space gesture and send operation instructions corresponding to the response points.
In one embodiment, the converting the operation of the second blank gesture according to the blank operation proportion includes:
and determining an operation vector on the visual interface according to the space operation proportion according to the displacement vector of the second space gesture in the space operation area, and responding to the displacement operation of the second space gesture and sending a displacement operation instruction aiming at the visual interface.
In one embodiment, when the second blank gesture is a cursor control gesture, after determining the coordinates of the response point on the visual interface according to the blank operation ratio, the method further includes:
acquiring a gesture area of the cursor control gesture in the second image sequence;
determining the area fluctuation amount of the cursor control gesture according to the fluctuation of the gesture area;
determining a click operation when the area fluctuation amount is greater than a fluctuation threshold;
and responding to the clicking operation, and sending a clicking instruction at a response point of the cursor control gesture.
The acquiring the gesture area of the cursor control gesture in the second image sequence includes:
acquiring the area increment and the area decrement when the cursor control gesture fluctuates;
and determining a clicking operation when the area increasing amount and the area decreasing amount are both larger than the fluctuation threshold.
The acquiring the gesture area of the cursor control gesture in the second image sequence includes:
and acquiring the area increment and the area decrement of the cursor control gesture in a preset time threshold.
In one embodiment, after the operation in response to the second air-blocking gesture located within the air-blocking operation region, the method further includes:
acquiring a third image sequence, wherein images in the third image sequence comprise a third blank gesture of a user;
and responding to the third blank gesture in the blank operation area, and sending a blank operation ending instruction.
In yet another aspect, a blank handling device is provided, the device comprising:
the image acquisition module is used for acquiring an image sequence, the image sequence at least comprises a first image sequence and a second image sequence, the images in the first image sequence comprise a first blank gesture of a user, and the images in the second image sequence comprise a second blank gesture of the user;
the image recognition module is used for obtaining a first track shape of the first blank gesture according to the first image sequence, determining a blank operation area according to the first track shape, and recognizing a second blank gesture positioned in the blank operation area;
and the execution module is used for responding to the operation of the second blank gesture in the blank operation area and sending an operation instruction of the visual interface.
In yet another aspect, a computer apparatus is provided comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method when the computer program is executed.
There is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method.
According to the method, the device, the computer equipment and the storage medium for the blank operation, the first blank gesture in the first image sequence is identified, the track shape of the first blank gesture is further identified, the blank operation area is determined according to the track shape, the space display range of the blank gesture operation is identified, in the subsequent identification process, only the operation of the second blank gesture in the blank operation area is responded, interference of gesture actions outside the blank operation area can be avoided, and the anti-interference capability of the blank operation is improved.
Drawings
FIG. 1 is a flow diagram of a method of operation in a null mode in one embodiment;
FIG. 2 is a schematic diagram of determining a blanking area in one embodiment;
FIG. 3 is a schematic diagram of gesture area fluctuation in another embodiment;
FIG. 4 is a block diagram of a structure of a hollow operating device in one embodiment;
fig. 5 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
At present, a large number of functions related to a vehicle can be operated on a vehicle-mounted central control large screen of an intelligent vehicle, and various function services are also available in a vehicle-mounted entertainment system. However, the vehicle-mounted central control large screen is arranged between the front row main driving and the auxiliary driving of the vehicle, and is always operated based on screen touch operation or near-field voice instruction operation, so that passengers sitting in the rear row cannot touch the large screen, and the functions on the large screen cannot be independently operated.
The problem that the space-apart gesture operation technology well solves is solved, an image containing a user specific gesture is shot through an image shooting unit, the user operation gesture is captured in real time and corresponding operation is executed, but for a multi-person scene in a narrow space in a vehicle cabin, how to improve the anti-interference capability of gesture operation becomes a development key point.
According to the blank operation method provided by the application, the blank operation area is defined by the user, and the blank operation gestures in the blank operation area are identified, so that gesture interference outside the blank operation area is reduced.
In one embodiment, as shown in fig. 1, the method of operation for blocking air comprises the steps of:
step 101, a first image sequence is acquired, wherein images in the first image sequence comprise a first blank gesture of a user.
It can be understood that after the automobile is electrified, the vehicle-mounted visual operating system is started, and meanwhile, the camera unit and the image analysis chip unit related to the system are started, the camera unit, the image analysis chip unit and the vehicle-mounted visual operating system are all in a working state, and the camera unit can be arranged at the center of the front row of the automobile in the automobile cabin so as to conveniently shoot most of space in the automobile, or user operation gestures can be captured from various angles in real time through a plurality of camera units.
Illustratively, the visual operating system of the present application presets the following five spaced gestures:
the visual operation system can distinguish the above-mentioned space gesture according to the shape characteristics of the gesture of the user, the shape of the gesture is for example, fist making, five fingers opening, forefinger stretching and the like, and when the visual operation system recognizes the first space gesture through the camera unit, the space operation function is started.
The first image sequence is a continuous image sequence of the time period from when the image analysis chip unit recognizes the first space gesture to when the first space gesture is finished.
The first blank gesture is further used for identifying a space display range of the operation of the present blank gesture, namely, an effective range of the present operation is identified through a motion range of the first blank gesture, and specifically, the method comprises the following steps:
step 102, obtaining the track shape of the first air-spaced gesture according to the first image sequence.
Illustratively, the first image sequence may include several frames or more of images, the gesture motion of the user causing the locations of the gestures in the images to be different, the gesture locations in all of the images in the first image sequence together comprising the trajectory of the spaced gesture.
It will be appreciated that a certain feature point of the gesture may be selected to represent the whole gesture, where coordinates of the feature point in the image are position coordinates of the whole gesture, for example, in some embodiments, a center point of the gesture is selected as a position point of the gesture, and in this embodiment, a top left corner (left, top) of the gesture in a coordinate range (left, top, right, bottom) on the image may be used as a pointing position (Xg, yg) of the gesture.
And step 103, determining a blank operation area according to the track shape.
The track shape of the first space gesture may include a closed shape such as a rectangle, a circle, etc., in some embodiments, the defined closed range is directly used as the space operation area, in this embodiment, the track shape is a circular track as shown in fig. 2, the minimum and maximum x coordinates and the minimum and maximum y coordinates are taken on the closed geometric figure, and the space operation space M formed by surrounding four points is identified on the image:
M=[(Xmin，Ymin),(Xmax,Ymin),(Xmax,Ymax),(Xmin,Ymax)]。
the space-apart operation space of the rectangle is determined by selecting the minimum and maximum points of the x coordinate and the minimum and maximum points of the y coordinate, and the space-apart operation space can be matched with a visual interface of the rectangle as much as possible.
In some embodiments, the track shape of the first blank gesture may also include a non-closed shape such as a diagonal line, a V-shape, etc., to provide a more range defining manner, and also take the x coordinate minimum and maximum of the non-closed shape, and the y coordinate minimum and maximum four points identify the blank operation space M on the image.
After the function of the blank operation is started and the blank operation area is defined, the blank operation can be performed.
Step 104, a second image sequence is acquired, wherein the images in the second image sequence comprise a second spaced gesture of the user.
Similar to the first image sequence, the second image sequence may also include several frames or more of images, and in this embodiment, the second spaced gesture may include a cursor control gesture, a swipe gesture, and a drag gesture.
And step 105, responding to the operation of the second blank gesture in the blank operation area, and sending an operation instruction of a visual interface, namely a screen.
It can be understood that in the image shot by the camera unit, the second space gesture is effective when being located in the delimited space operation area, and when the visual operation system identifies the effective second space gesture, corresponding instructions such as cursor control, sliding or dragging are executed; otherwise, do not respond.
According to the method for isolating operation, the first isolating gesture of the user wakes up and demarcates the gesture operation range, and when the visual operation system identifies the isolating gesture, the visual operation system only identifies and responds to the isolating gesture in the isolating operation area, so that interference from outside the isolating operation area is reduced.
On the other hand, the passengers in the vehicle cabin may be located on the left side or the right side of the vehicle cabin, the space-apart operation is performed based on the space-apart operation area defined by the user, and the user can perform the space-apart operation at any position within the range of the camera.
In some embodiments, a plurality of cameras can be used to collect images of the cabin of the vehicle from different angles in real time to form a first image sequence, and when a plurality of camera units exist in the vehicle to capture images of the user from all angles, any one of the cameras can wake up the space operation function when collecting the images with the first space gesture.
After waking up the blank operation, determining a main camera unit for blank operation gesture recognition according to a first image sequence, for example, matching according to the position relation between the position of a gesture sender and the camera unit, or comparing a currently acquired gesture image with a preset standard image, taking the camera unit with the highest gesture matching degree as the main camera unit, or taking the camera unit with the maximum gesture image area as the main camera unit, determining the track shape of the first blank gesture according to the image acquired by the main camera unit in the first image sequence in the subsequent blank gesture recognition, and responding the second blank gesture according to a second image sequence containing the second blank gesture, which is acquired by the main camera unit.
By adopting the mode, the passenger operation in the cabin can be identified from various angles, and the blocking of obstacles such as a cabin seat and the like can be avoided.
In general, the shape and size of the blank operation area defined by the user do not completely match the shape and size of the visual interface.
In one embodiment, the operation responsive to the second blank gesture located within the blank operation area comprises:
determining a blank operation proportion according to the interface size of the visual interface and the pixel size of the blank operation area in an image space;
and converting the operation of the second blank gesture according to the blank operation proportion so as to obtain an operation instruction of the visual interface.
Through the conversion of the operation proportion, a user can operate the elements on the whole visual interface in a blank operation area which is not completely consistent with the shape and the size of the visual interface.
It can be appreciated that the determining the blank operation proportion includes determining a width proportion and a height proportion, specifically, determining the width proportion according to the lateral dimension of the blank operation area in the image space and the lateral dimension of the visual interface; and determining the height ratio according to the longitudinal dimension of the blank operation area in the image space and the longitudinal dimension of the visual interface.
Illustratively, the visual operating system of the present application determines the lateral dimensions from the coordinate range of the isolated operating region in the first image sequence: width_m=xmax-Xmin, longitudinal dimension: height_m=ymax-Ymin.
For a vehicle center control screen, it has determined lateral and longitudinal dimensions of the visual interface: width_s and height_s.
From the above data, the width scale and height scale can be calculated:
scaleX=width_M/widht_S；
scaleY=height_M/height_S。
the gesture operation can be subsequently converted according to the ratio, for example, for one sliding operation, the lateral sliding distance of the gesture in the image acquired by the camera unit is converted by the width ratio, and the lateral sliding distance on the visual interface can be obtained.
In some embodiments, the converting the operation of the second space gesture includes converting gesture coordinates, where coordinates of the response point on the visual interface are determined according to the space operation proportion according to coordinates of the second space gesture in the space operation area.
As shown in fig. 2, for the cursor control gesture P, the user expects to display a cursor indication P' on the visual interface, that is, after defining the blank operation area, the cursor control gesture is made, and when the system recognizes the cursor control gesture, the coordinate range of the cursor control gesture on the image is calculated: (left, top, right, bottom); the upper left corner of the cursor control gesture is used as the pointing position of the cursor control gesture: (Xg, yg) = (left, top), the coordinates of the response points (Xs, ys) on the visual interface are calculated using the following mathematical expression: (Xg-Xmin)/scalex=xs, (Yg-Ymin)/scaley=ys, and outputting the coordinates (Xs, ys) to the visual operating system, and displaying the cursor indication P' on the coordinates (Xs, ys) by the visual operating system large screen.
For the cursor control gesture, the cursor control gesture can also be used for clicking operation, so that a user can conveniently move the cursor to a place to be clicked and then directly click without replacing the gesture.
In this embodiment, after determining the coordinates of the response point on the visual interface according to the blank operation ratio, the visual operating system further obtains a gesture area of the cursor control gesture in each image of the second image sequence.
The visual operating system determines whether a click operation occurs according to the fluctuation of the gesture area.
For example, the visual operating system continuously records the gesture area in each frame of image according to the time sequence, determines the area fluctuation amount of the cursor control gesture, when the clicking action occurs, the area of the cursor control gesture changes in the image, as shown in fig. 3, based on the habit of the user, the gesture may continuously become larger and smaller or continuously become smaller and larger in a short time, and when the area fluctuation amount is greater than the fluctuation threshold, the user can determine that the clicking operation is performed.
As indicated by the solid line portion in fig. 3, the area increment a and the area decrement B may be different, in some embodiments, when either one of the area increment a and the area decrement B satisfies that the area increment a and the area decrement B exceeds the fluctuation threshold, the click operation may be considered to be sent, and in order to improve the anti-interference effect, when the area increment a and the area decrement B are required to be simultaneously greater than the fluctuation threshold, it is confirmed that the click operation occurs, and the visual operating system may respond to the click operation and send a click command at the response point of the cursor control gesture, and output a click (Xs, ys) signal.
It should be noted that when the clicking operation occurs, the clicking operation is required to be completed within the time threshold C, that is, the duration C is estimated forward with the current moment as the base point, and the gesture area has the characteristics as described above, so that the clicking operation can be triggered.
By defining the time threshold and the fluctuation threshold, the shake disturbance occurring at the time of traveling can be eliminated.
In some embodiments, the converting the operation of the second space gesture further includes a displacement conversion of the gesture, including determining an operation vector on the visual interface according to the space operation proportion according to a displacement vector of the second space gesture in the space operation area, so as to respond to the displacement operation of the second space gesture, and sending a displacement operation instruction for the visual interface.
Taking a sliding gesture as an example for illustration, when the system recognizes the sliding gesture, a coordinate range (left, top, right, bottom) of the sliding gesture on the image is received; the upper left corner of the swipe gesture is taken as the pointing position of the swipe gesture: when the sliding gesture is continuously received, the coordinates of the sliding gesture are set to be Point 2= (left 2, top 2) by taking Point 1= (left 1, top 2) as a starting Point, and the pointing positions of two consecutive times are subtracted to obtain a displacement vector VM= (left 2-left1, top2-top 1) = (Vx, vy) between the two points.
Converting the displacement vector according to the space operation proportion to obtain an operation vector VS on the visual interface: vs= (Vx/scaleX, vy/scaleY), the command of Move VS is output.
The swipe gesture may be used as a view of a map area, a transition of an interface, and so on.
Coordinate conversion and displacement conversion can be applied simultaneously, and the present embodiment is described taking a drag gesture as an example.
The drag gesture can drag the element on the visual interface, such as an icon, when the system recognizes the drag gesture, the coordinate of the drag gesture on the image for the first time is calculated and obtained, namely, the drag Point of this time is (Xd, yd) = (left, top), and the drag Point actually on the visual interface is (Xds, yds) = ((Xd-Xmin)/scaleX, (Yd-Ymin)/scaleY) according to the blank operation proportion.
When the drag gesture is continuously received, let the drag gesture coordinate at this time be Point 2= (left 2, top2, right2, bottom 2) and the value obtained by subtracting Point1 from Point2 is the displacement vector between two times, VM= (left 2-left1, top2-top 1) = (Vx, vy), and the drag vector representing this time is VM= (Vx, vy) and the drag vector Vs on the visual interface is obtained by conversion: vs= (Vx/scaleX, vy/scaleY), the visual operating system may drag the element from the drag point (Xds, yds) in the direction and size of the drag vector Vs on the visual interface. The last point Pf of the coordinate sequence of the drag gesture is continuously recognized as the release point of the drag.
The visual operating system of this embodiment further performs a third blank gesture with the preset: ending the gesture. After the blank operation required by the user is completed, the user can make an ending gesture, and the camera unit acquires a third image sequence, wherein the images in the third image sequence comprise a third blank gesture of the user; and sending a blank operation ending instruction in response to the third blank gesture, wherein the third blank gesture is valid in a blank operation area.
The five kinds of spaced gesture operations provided by the application realize cursor display, cursor movement, a clicking event at a cursor, an interface element sliding event and an interface element dragging event. The above events can substantially satisfy a substantial portion of user operations of the visual operating system.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 1 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor do the order in which the sub-steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of other steps or sub-steps of other steps.
In one embodiment, as shown in fig. 4, there is provided a space-saving operation device comprising: an image acquisition module 201, an image recognition module 202, and an execution module 203, wherein:
an image acquisition module 201, such as a camera, for acquiring an image sequence, the image sequence comprising at least a first image sequence and a second image sequence, the images in the first image sequence comprising a first blank gesture of a user and the images in the second image sequence comprising a second blank gesture of the user;
an image recognition module 202, configured to obtain a first track shape of the first blank gesture according to the first image sequence, determine a blank operation area according to the first track shape, and recognize a second blank gesture located in the blank operation area;
and the execution module 203 is configured to send an operation instruction of the visual interface in response to an operation of the second space gesture in the space operation area.
According to the blank operation device, the first blank gesture in the first image sequence is recognized, the track shape of the first blank gesture is further recognized, the blank operation area is determined according to the track shape, the space display range of the blank gesture operation is identified, in the subsequent recognition process, only the operation of the second blank gesture in the blank operation area is responded, interference of gesture actions outside the blank operation area can be avoided, and the anti-interference capability of the blank operation is improved.
In one embodiment, the second blank gesture includes one of a cursor control gesture, a slide gesture and a drag gesture, and the execution module 203 determines a blank operation proportion according to an interface size of the visual interface and a pixel size of the blank operation area in an image space, adapts the blank operation area to the visual interface, and converts an operation of the second blank gesture according to the blank operation proportion to obtain an operation instruction of the visual interface.
For example for a gesture operation: after the shift of the skip operation ratio scaleX, the instruction executed by the execution module 203 is the lateral slide X/scaleX.
It can be appreciated that determining the blank operation proportion includes determining a width proportion according to a lateral dimension of the blank operation region in the image space and a lateral dimension of the visual interface; and determining the height ratio according to the longitudinal dimension of the blank operation area in the image space and the longitudinal dimension of the visual interface.
In one embodiment, the executing module 203 converts the operation of the second blank gesture according to the blank operation proportion, including determining, according to the blank operation proportion, coordinates of a response point on the visual interface according to coordinates of the second blank gesture in the blank operation area, so as to respond to the operation of the second blank gesture, and sending an operation instruction corresponding to the response point.
In another embodiment, the executing module 203 converts the operation of the second blank gesture according to the blank operation proportion, including determining an operation vector on the visual interface according to the blank operation proportion according to a displacement vector of the second blank gesture in the blank operation area, so as to respond to the displacement operation of the second blank gesture, and sending a displacement operation instruction for the visual interface.
When the second blank gesture is a cursor control gesture, after determining the coordinates of the response point on the visual interface according to the blank operation proportion, the method further includes:
acquiring a gesture area of the cursor control gesture in each image of the second image sequence;
determining the area fluctuation amount of the cursor control gesture according to the fluctuation of the gesture area;
determining a click operation when the area fluctuation amount is greater than a fluctuation threshold;
and responding to the clicking operation, and sending a clicking instruction at a response point of the cursor control gesture.
Through the cursor control gesture, the user can conveniently adjust the cursor indication position and perform clicking operation on the cursor indication position.
In one embodiment, the image obtaining module 201 is further configured to obtain a third image sequence, where an image in the third image sequence includes a third blank gesture of the user, the image identifying module 202 identifies the third blank gesture located in the blank operation area, and the executing module 203 sends a blank operation end instruction in response to the third blank gesture located in the blank operation area when the third blank gesture disappears.
For specific limitations of the barrier operation device, reference may be made to the above limitation of the barrier operation method, and no further description is given here. The respective modules in the above-described idle operation device may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of operation in a barrier. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in FIG. 5 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of when executing the computer program:
acquiring a first image sequence, wherein images in the first image sequence comprise a first blank gesture of a user;
obtaining a track shape of the first blank gesture according to the first image sequence;
determining a blank operation area according to the track shape;
acquiring a second image sequence, wherein images in the second image sequence comprise a second blank gesture of a user;
and responding to the operation of the second air-stop gesture in the air-stop operation area, and sending an operation instruction of a visual interface.
In one embodiment, the processor when executing the computer program further performs the steps of:
determining a blank operation proportion according to the interface size of the visual interface and the pixel size of the blank operation area in an image space;
and converting the operation of the second blank gesture according to the blank operation proportion so as to obtain an operation instruction of the visual interface.
Specifically, determining a width ratio according to the transverse dimension of the space operation area in the image space and the transverse dimension of the visual interface;
and determining the height ratio according to the longitudinal dimension of the blank operation area in the image space and the longitudinal dimension of the visual interface.
In one embodiment, the processor when executing the computer program further performs the steps of:
and determining the coordinates of the response points on the visual interface according to the space operation proportion according to the coordinates of the second space gesture in the space operation area, so as to respond to the operation of the second space gesture and send operation instructions corresponding to the response points.
In one embodiment, the processor when executing the computer program further performs the steps of:
and determining an operation vector on the visual interface according to the space operation proportion according to the displacement vector of the second space gesture in the space operation area, and responding to the displacement operation of the second space gesture and sending a displacement operation instruction aiming at the visual interface.
In one embodiment, the processor when executing the computer program further performs the steps of:
when the second spaced gesture is a cursor control gesture, acquiring a gesture area of the cursor control gesture in each image of the second image sequence; determining the area fluctuation amount of the cursor control gesture according to the fluctuation of the gesture area; determining a click operation when the area fluctuation amount is greater than a fluctuation threshold; and responding to the clicking operation, and sending a clicking instruction at a response point of the cursor control gesture.
In one embodiment, the processor when executing the computer program further performs the steps of:
acquiring a third image sequence, wherein images in the third image sequence comprise a third blank gesture of a user; and responding to the third blank gesture in the blank operation area, and sending a blank operation ending instruction.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a first image sequence, wherein images in the first image sequence comprise a first blank gesture of a user; obtaining a track shape of the first blank gesture according to the first image sequence;
determining a blank operation area according to the track shape;
acquiring a second image sequence, wherein images in the second image sequence comprise a second blank gesture of a user;
and responding to the operation of the second air-stop gesture in the air-stop operation area, and sending an operation instruction of a visual interface.
In one embodiment, the computer program when executed by the processor further performs the steps of:
determining a blank operation proportion according to the interface size of the visual interface and the pixel size of the blank operation area in an image space;
and converting the operation of the second blank gesture according to the blank operation proportion so as to obtain an operation instruction of the visual interface.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and determining the coordinates of the response points on the visual interface according to the space operation proportion according to the coordinates of the second space gesture in the space operation area, so as to respond to the operation of the second space gesture and send operation instructions corresponding to the response points.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and determining an operation vector on the visual interface according to the space operation proportion according to the displacement vector of the second space gesture in the space operation area, and responding to the displacement operation of the second space gesture and sending a displacement operation instruction aiming at the visual interface.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring a third image sequence, wherein images in the third image sequence comprise a third blank gesture of a user; and responding to the third blank gesture in the blank operation area, and sending a blank operation ending instruction.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (12)
1. A method of operation in a barrier comprising:
acquiring a first image sequence, wherein images in the first image sequence comprise a first blank gesture of a user;
obtaining a track shape of the first blank gesture according to the first image sequence;
determining a blank operation area according to the track shape;
acquiring a second image sequence, wherein images in the second image sequence comprise a second blank gesture of a user;
and responding to the operation of the second air-stop gesture in the air-stop operation area, and sending an operation instruction of a visual interface.
2. The method of claim 1, wherein the second hold gesture comprises one of a cursor control gesture, a swipe gesture, and a drag gesture; the operation responsive to the second blank gesture located within the blank operation area includes:
determining a blank operation proportion according to the interface size of the visual interface and the pixel size of the blank operation area in an image space;
and converting the operation of the second blank gesture according to the blank operation proportion so as to obtain an operation instruction of the visual interface.
3. The method of operation of claim 2, wherein determining the proportion of operation of the barrier comprises:
determining a width ratio according to the transverse dimension of the blank operation area in the image space and the transverse dimension of the visual interface;
and determining the height ratio according to the longitudinal dimension of the blank operation area in the image space and the longitudinal dimension of the visual interface.
4. The method according to claim 2, wherein the switching operation of the second blank gesture according to the blank operation ratio includes:
and determining the coordinates of the response points on the visual interface according to the space operation proportion according to the coordinates of the second space gesture in the space operation area, so as to respond to the operation of the second space gesture and send operation instructions corresponding to the response points.
5. The method according to claim 2, wherein the switching operation of the second blank gesture according to the blank operation ratio includes:
and determining an operation vector on the visual interface according to the space operation proportion according to the displacement vector of the second space gesture in the space operation area, and responding to the displacement operation of the second space gesture and sending a displacement operation instruction aiming at the visual interface.
6. The method according to claim 4, wherein when the second blank gesture is a cursor control gesture, after determining coordinates of a response point on the visual interface according to the blank operation scale, the method further comprises:
acquiring a gesture area of the cursor control gesture in the second image sequence;
determining the area fluctuation amount of the cursor control gesture according to the fluctuation of the gesture area;
determining a click operation when the area fluctuation amount is greater than a fluctuation threshold;
and responding to the clicking operation, and sending a clicking instruction at a response point of the cursor control gesture.
7. The method of claim 6, wherein the acquiring the gesture area of the cursor control gesture in the second image sequence comprises:
acquiring the area increment and the area decrement when the cursor control gesture fluctuates;
and determining a clicking operation when the area increasing amount and the area decreasing amount are both larger than the fluctuation threshold.
8. The method of claim 7, wherein the acquiring the gesture area of the cursor control gesture in the second image sequence comprises:
and acquiring the area increment and the area decrement of the cursor control gesture in a preset time threshold.
9. The method of claim 1, wherein following the operation in response to the second blank gesture within the blank operation area, further comprising:
acquiring a third image sequence, wherein images in the third image sequence comprise a third blank gesture of a user;
and responding to the third blank gesture in the blank operation area, and sending a blank operation ending instruction.
10. A space-time operating device, the device comprising:
the image acquisition module is used for acquiring an image sequence, the image sequence at least comprises a first image sequence and a second image sequence, the images in the first image sequence comprise a first blank gesture of a user, and the images in the second image sequence comprise a second blank gesture of the user;
the image recognition module is used for obtaining a first track shape of the first blank gesture according to the first image sequence, determining a blank operation area according to the first track shape, and recognizing a second blank gesture positioned in the blank operation area;
and the execution module is used for responding to the operation of the second blank gesture in the blank operation area and sending an operation instruction of the visual interface.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 9 when the computer program is executed by the processor.
12. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310909359.XA CN116627260A (en) | 2023-07-24 | 2023-07-24 | Method and device for idle operation, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310909359.XA CN116627260A (en) | 2023-07-24 | 2023-07-24 | Method and device for idle operation, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116627260A true CN116627260A (en) | 2023-08-22 |
Family
ID=87592479
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310909359.XA Pending CN116627260A (en) | 2023-07-24 | 2023-07-24 | Method and device for idle operation, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116627260A (en) |
Citations (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102446032A (en) * | 2010-09-30 | 2012-05-09 | 中国移动通信有限公司 | Information input method and terminal based on camera |
| CN103839040A (en) * | 2012-11-27 | 2014-06-04 | 株式会社理光 | Gesture identification method and device based on depth images |
| CN103869967A (en) * | 2012-12-14 | 2014-06-18 | 歌乐株式会社 | Control apparatus, vehicle, and portable terminal |
| US20140201666A1 (en) * | 2013-01-15 | 2014-07-17 | Raffi Bedikian | Dynamic, free-space user interactions for machine control |
| CN103984416A (en) * | 2014-06-10 | 2014-08-13 | 北京邮电大学 | Gesture recognition method based on acceleration sensor |
| CN104364735A (en) * | 2012-04-13 | 2015-02-18 | 诺基亚公司 | Free hand gesture control of automotive user interface |
| US20150331492A1 (en) * | 2014-05-14 | 2015-11-19 | Samsung Electronics Co., Ltd. | Method and apparatus for identifying spatial gesture of user |
| CN105278763A (en) * | 2015-05-28 | 2016-01-27 | 维沃移动通信有限公司 | Gesture recognition method and apparatus capable of preventing mistaken touch |
| WO2016113969A1 (en) * | 2015-01-13 | 2016-07-21 | 三菱電機株式会社 | Gesture recognition device and method, program, and recording medium |
| KR20160124361A (en) * | 2015-04-17 | 2016-10-27 | 가천대학교 산학협력단 | Hand Feature Extraction Algorithm using Curvature Analysis For Recognition of Various Hand Feature |
| CN106503651A (en) * | 2016-10-21 | 2017-03-15 | 上海未来伙伴机器人有限公司 | A kind of extracting method of images of gestures and system |
| CN107678551A (en) * | 2017-10-19 | 2018-02-09 | 京东方科技集团股份有限公司 | Gesture identification method and device, electronic equipment |
| CN109116984A (en) * | 2018-07-27 | 2019-01-01 | 冯仕昌 | A kind of tool box for three-dimension interaction scene |
| WO2019080780A1 (en) * | 2017-10-24 | 2019-05-02 | 京东方科技集团股份有限公司 | Gesture detection method, gesture processing device, and computer readable storage medium |
| CN109948511A (en) * | 2019-03-14 | 2019-06-28 | 广东美的白色家电技术创新中心有限公司 | Gesture identification method and device |
| CN110045819A (en) * | 2019-03-01 | 2019-07-23 | 华为技术有限公司 | A kind of gesture processing method and equipment |
| CN110276251A (en) * | 2019-05-13 | 2019-09-24 | 联想（上海）信息技术有限公司 | A kind of image-recognizing method, device, equipment and storage medium |
| CN111596757A (en) * | 2020-04-02 | 2020-08-28 | 林宗宇 | Gesture control method and device based on fingertip interaction |
| CN112383805A (en) * | 2020-11-16 | 2021-02-19 | 四川长虹电器股份有限公司 | Method for realizing man-machine interaction at television end based on human hand key points |
| CN112753009A (en) * | 2021-01-04 | 2021-05-04 | 华为技术有限公司 | Man-machine interaction method and man-machine interaction device |
| CN113335185A (en) * | 2021-08-06 | 2021-09-03 | 智己汽车科技有限公司 | In-vehicle multifunctional information display device based on aerial imaging and control method |
| WO2021184356A1 (en) * | 2020-03-20 | 2021-09-23 | Huawei Technologies Co., Ltd. | Methods and systems for hand gesture-based control of a device |
| CN115100742A (en) * | 2022-06-23 | 2022-09-23 | 上海妙文会展服务有限公司 | Meta-universe exhibition and display experience system based on air-separating gesture operation |
| CN115291724A (en) * | 2022-08-01 | 2022-11-04 | 北京地平线机器人技术研发有限公司 | Man-machine interaction method and device, storage medium and electronic equipment |
-
2023
- 2023-07-24 CN CN202310909359.XA patent/CN116627260A/en active Pending
Patent Citations (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102446032A (en) * | 2010-09-30 | 2012-05-09 | 中国移动通信有限公司 | Information input method and terminal based on camera |
| CN104364735A (en) * | 2012-04-13 | 2015-02-18 | 诺基亚公司 | Free hand gesture control of automotive user interface |
| CN103839040A (en) * | 2012-11-27 | 2014-06-04 | 株式会社理光 | Gesture identification method and device based on depth images |
| CN103869967A (en) * | 2012-12-14 | 2014-06-18 | 歌乐株式会社 | Control apparatus, vehicle, and portable terminal |
| US20140201666A1 (en) * | 2013-01-15 | 2014-07-17 | Raffi Bedikian | Dynamic, free-space user interactions for machine control |
| US20150331492A1 (en) * | 2014-05-14 | 2015-11-19 | Samsung Electronics Co., Ltd. | Method and apparatus for identifying spatial gesture of user |
| CN103984416A (en) * | 2014-06-10 | 2014-08-13 | 北京邮电大学 | Gesture recognition method based on acceleration sensor |
| WO2016113969A1 (en) * | 2015-01-13 | 2016-07-21 | 三菱電機株式会社 | Gesture recognition device and method, program, and recording medium |
| KR20160124361A (en) * | 2015-04-17 | 2016-10-27 | 가천대학교 산학협력단 | Hand Feature Extraction Algorithm using Curvature Analysis For Recognition of Various Hand Feature |
| CN105278763A (en) * | 2015-05-28 | 2016-01-27 | 维沃移动通信有限公司 | Gesture recognition method and apparatus capable of preventing mistaken touch |
| CN106503651A (en) * | 2016-10-21 | 2017-03-15 | 上海未来伙伴机器人有限公司 | A kind of extracting method of images of gestures and system |
| CN107678551A (en) * | 2017-10-19 | 2018-02-09 | 京东方科技集团股份有限公司 | Gesture identification method and device, electronic equipment |
| WO2019080780A1 (en) * | 2017-10-24 | 2019-05-02 | 京东方科技集团股份有限公司 | Gesture detection method, gesture processing device, and computer readable storage medium |
| CN109116984A (en) * | 2018-07-27 | 2019-01-01 | 冯仕昌 | A kind of tool box for three-dimension interaction scene |
| CN110045819A (en) * | 2019-03-01 | 2019-07-23 | 华为技术有限公司 | A kind of gesture processing method and equipment |
| CN109948511A (en) * | 2019-03-14 | 2019-06-28 | 广东美的白色家电技术创新中心有限公司 | Gesture identification method and device |
| CN110276251A (en) * | 2019-05-13 | 2019-09-24 | 联想（上海）信息技术有限公司 | A kind of image-recognizing method, device, equipment and storage medium |
| WO2021184356A1 (en) * | 2020-03-20 | 2021-09-23 | Huawei Technologies Co., Ltd. | Methods and systems for hand gesture-based control of a device |
| CN111596757A (en) * | 2020-04-02 | 2020-08-28 | 林宗宇 | Gesture control method and device based on fingertip interaction |
| CN112383805A (en) * | 2020-11-16 | 2021-02-19 | 四川长虹电器股份有限公司 | Method for realizing man-machine interaction at television end based on human hand key points |
| CN112753009A (en) * | 2021-01-04 | 2021-05-04 | 华为技术有限公司 | Man-machine interaction method and man-machine interaction device |
| CN113335185A (en) * | 2021-08-06 | 2021-09-03 | 智己汽车科技有限公司 | In-vehicle multifunctional information display device based on aerial imaging and control method |
| CN115100742A (en) * | 2022-06-23 | 2022-09-23 | 上海妙文会展服务有限公司 | Meta-universe exhibition and display experience system based on air-separating gesture operation |
| CN115291724A (en) * | 2022-08-01 | 2022-11-04 | 北京地平线机器人技术研发有限公司 | Man-machine interaction method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111857356B (en) | Method, device, equipment and storage medium for recognizing interaction gesture | |
| US9129154B2 (en) | Gesture recognition apparatus, robot system including the same and gesture recognition method using the same | |
| KR101534742B1 (en) | System and method for gesture recognition of vehicle | |
| CN106406710B (en) | Screen recording method and mobile terminal | |
| KR102465532B1 (en) | Method for recognizing an object and apparatus thereof | |
| US11249557B2 (en) | Methods and systems for controlling a device using hand gestures in multi-user environment | |
| WO2019033957A1 (en) | Interaction position determination method and system, storage medium and smart terminal | |
| US9235269B2 (en) | System and method for manipulating user interface in vehicle using finger valleys | |
| CN108960163B (en) | Gesture recognition method, device, equipment and storage medium | |
| JP4929109B2 (en) | Gesture recognition apparatus and method | |
| KR20130112550A (en) | Apparatus for setting parking position based on around view image and method thereof | |
| EP4361771A1 (en) | Gesture recognition method and apparatus, system, and vehicle | |
| JP6671288B2 (en) | Gesture device, operation method thereof, and vehicle equipped with the same | |
| EP4318186A1 (en) | Multi-screen interaction method and apparatus, and terminal device and vehicle | |
| CN108897589B (en) | Human-computer interaction method and device in display equipment, computer equipment and storage medium | |
| CN112507918A (en) | Gesture recognition method | |
| CN107357414B (en) | Click action recognition method and device | |
| CN111656313A (en) | Screen display switching method, display device and movable platform | |
| CN105759955B (en) | Input device | |
| CN108363485B (en) | Control method, device and system of non-touch screen display terminal and computer equipment | |
| CN111986229A (en) | Video target detection method, device and computer system | |
| CN116627260A (en) | Method and device for idle operation, computer equipment and storage medium | |
| US9761009B2 (en) | Motion tracking device control systems and methods | |
| CN112788244A (en) | Shooting method, shooting device and electronic equipment | |
| KR20130078496A (en) | Apparatus and method for controlling electric boards using multiple hand shape detection and tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |