TW202338590A - Human-machine interface controllable by hand gesture - Google Patents

Abstract

The present invention provides a human-machine interface controllable by hand gesture, including a display interface, a database, a camera module, and a control module. The display interface displays multimedia information. The database stores a plurality of hand gestures and corresponding operation commands. The camera module captures an image of a non-contact operation performed by a user corresponding to the multimedia information. The control module receives the operation image from the camera module, and extracts the hand image through image processing, marks a framework image from the hand image, and compares framework image with each hand gestures in the database, so as to receive the matching hand gestures and corresponding operation command. The control module sends the received operation command back to the multimedia information, so as to carry out a non-contact operation to the multimedia information, achieving a non-contact controlling effect.

Description

Human-machine interface device with hand gesture control

The present invention relates to the field related to human-machine interface, and in particular, to a human-machine interface device with hand gesture control.

Press, human-machine interface devices are widely used from common cash machines, Ibon, self-checkout machines, to industrial touch computers, etc. Among them, human-machine interface devices have touch screens, and touch screens can Inductive LED displays, LCD displays, flat screens or electronic ink devices that receive input signals from contacts (including fingers or pen tips, etc.).

When the graphic buttons on the screen are touched, the tactile feedback system on the screen can drive various connection devices according to pre-programmed programs, which can be used to replace the mechanical button panel and create vivid audio and video effects through the display screen.

However, due to the epidemic of COVID-19 in recent years, users are often reluctant to directly touch public touch screens due to the impact of the Covid-19 epidemic, in order to avoid the risk of contact infection caused by touch screens. Instead, When operating through other touch components (such as a stylus), since the touch component contacts the touch screen, there may be a delay in response, so the user will increase the force of the touch component, causing the touch screen to The screen is easily damaged.

In addition, the touch screen of the human-machine interface device requires direct touch control by the user. For people with disabilities, it is often difficult to operate. For example, the touch screen of the human-machine interface device is installed too high, resulting in People with disabilities who have immobile legs cannot use it smoothly, causing difficulty in use.

In order to solve the above problems, the present invention provides a human-machine interface device with hand gesture control, which can control the multimedia information presented on the display interface using a non-contact hand image control method to achieve a non-contact control effect.

One embodiment of the present invention provides a human-machine interface device with hand gesture control, which includes: a machine body; a display interface provided on the machine body and providing for displaying multimedia information; a database, It is provided on the machine body and coupled with the display interface. The database contains a plurality of hand postures and an operation command corresponding to each hand posture; a camera module is provided on the machine body. The camera module provides the user with corresponding photos. An operation image for non-contact control of multimedia information; and a control module, which is located on the machine body and coupled to the display interface, database and photography module. The control module receives the operation image from the photography module and controls the module. The group extracts a hand image through an image calculation process on the operating image, and marks a skeleton image with the hand image. The control module compares the skeleton image with each hand posture existing in the database to obtain a matching hand. The control module will feedback the obtained operation instructions to the multimedia information to perform contactless control of the multimedia information.

Through the above, the present invention can capture the user's corresponding multimedia operation images, analyze the operation images to obtain the corresponding operation instructions, and use the operation instructions to feedback control the multimedia information presented on the display interface; thereby achieving contactless The control effect improves the concern that conventional direct-touch human-machine interface devices are prone to contact infection.

Furthermore, the present invention provides convenience in use for people with disabilities or users who are inconvenient to directly control the human-machine interface device through a contactless control method.

In order to facilitate the explanation of the central idea of the present invention expressed in the above summary column, specific embodiments are hereby expressed. Various objects in the embodiments are drawn according to proportions, sizes, deformations or displacements suitable for illustration, rather than according to the proportions of actual components, and are explained first.

Referring to FIGS. 1 to 5 , the present invention provides a human-machine interface device 100 with hand gesture control, which includes:

A machine body 10 has a shell 11, as shown in Figure 4. It should be noted that Figure 4 is only for illustration and does not limit the appearance of the machine body 10 of the present invention.

A display interface 20 is provided on the casing 11 of the machine body 10. The display interface 20 provides for displaying a multimedia information 21. The multimedia information 21 can be pictures, videos, text or combinations thereof. The present invention does not limit the multimedia information 21. type.

A database 30 is provided on the machine body 10 and coupled to the display interface 20. The database 30 stores a plurality of hand postures 31 and an operation command corresponding to each hand posture 31; in the embodiment of the present invention, the hand posture The operation instructions corresponding to the hand posture 31 can be (1) the hand posture 31 is the index finger upward, and the corresponding operation instruction is move up; (2) the hand posture 31 is the index finger downward, and the corresponding operation instruction is move down; (3) the hand posture 31 is the index finger downward, and the operation instruction is move down; The gesture 31 is a fist and the corresponding operation command of the finger face display interface 20 is return; (4) the hand gesture 31 is a palm push, and the corresponding operational command is continue or forward; (5) the hand gesture 31 is an OK gesture, and the corresponding operational command is click. ; (6) The hand posture 31 is a fist, and the corresponding operation command of the palm face display interface 20 is stop; (7) The hand posture 31 is thumb to the left, and the corresponding operation command is move left; (8) The hand posture 31 is The operation command corresponding to the thumb moving to the right is to move right, as shown in Figure 2; but the present invention is not limited to this.

A camera module 40 is provided on the casing 11 of the machine body 10 and adjacent to the display interface 20. The camera module 40 provides an operating image for capturing the user's contactless control of the multimedia information 21, as shown in Figures 1 and 4 shown; among them, the camera module 40 can dynamically track the operation image of the user performing contactless control corresponding to the multimedia information 21. In the embodiment of the present invention, the operation image is a dynamic image.

A control module 50 is installed in the housing 11 of the machine body 10 and is coupled to the display interface 20, the database 30 and the camera module 40. The control module 50 receives the operation image from the camera module 40, and the control module 50 extracts a hand image H through an image calculation process from the operation image, and marks a skeleton image I from the hand image H, as shown in Figures 1 and 3; the control module 50 stores the skeleton image I with the database 30 Each hand posture 31 is compared to obtain one of the matching hand postures 31 and the corresponding operation command. The control module 50 feeds back the obtained operation command to the multimedia information 21 to perform seamless processing on the multimedia information 21. Contact control is shown in Figure 1, Figure 4 and Figure 5; in the embodiment of the present invention, the image computing processing is computer vision (Computer Vision) combined with deep learning (Deep Learning).

Furthermore, the control module 50 marks the skeleton image I on the hand image H through a data labeling program (LabelImg). The skeleton image I has a plurality of fulcrums P and a plurality of struts S, and each fulcrum P corresponds to each knuckle of the hand. and the junction of the hand and the wrist. Each pole S is connected between two adjacent points P. Each pole P has an exclusive code, as shown in Figure 3; and the control module 50 can determine the relationship between each pole P and each other. position relationship to obtain one of the matching hand gestures 31 from the database 30 .

In addition, the control module 50 will receive the dynamic operation image transmitted by the photography module 40, and the control module 50 will divide the operation image into a plurality of stage operation images according to time sequence, and the control module 50 will perform image calculation on each stage operation image. Process, and generate a plurality of hand images H marked with the skeleton image I according to the time sequence, and obtain the operation instructions from the database 30 according to the time sequence, the control module 50 will feedback the obtained operation instructions to the multimedia information 21 to respond to the multimedia information 21 for continuous non-contact control.

An editing module 60 is located in the shell 11 of the machine body 10 and is coupled to the display interface 20, the database 30, the photography module 40 and the control module 50. The editing module 60 can edit each element in the database 30. The hand gestures 31 and the corresponding operation instructions are displayed on the display interface 20. The editing module 60 can select one of the hand gestures 31 for editing to change the selected hand gesture 31 and the corresponding operation instructions.

Furthermore, the editing module 60 can enable the photography module 40 to capture a sample image, and send the sample image to the control module 50 for image calculation processing to generate a hand image H marked with the skeleton image I. The editing module 60 can The hand image H marked with the skeleton image I is replaced with one of the hand postures 31 in the database 30 or set to correspond to one of the operation instructions.

A financial module 70 is coupled to the display interface 20 and the control module 50. The financial module 70 has a checkout option. The checkout option is displayed on the display interface 20 in conjunction with the multimedia information 21. The control module can perform the obtained operations. The command feedback responds to the multimedia information 21 and clicks the checkout option to perform a contactless checkout operation on the multimedia information 21. The financial module 70 can present a consumption amount on the display interface 20 and provide the payment amount.

For example: when the multimedia information 21 displayed on the display interface 20 is shopping consumption multimedia information 21, the multimedia information 21 will be presented on the display interface 20 in combination with the checkout options, and the user can make purchases through contactless operations, and Checkout occurs via the financial module 70 .

Based on the above, the present invention can achieve the following effects:

1. The human-machine interface device 100 with hand gesture control of the present invention can capture the user's operation image corresponding to multimedia, analyze the operation image to obtain the corresponding operation instruction, and use the operation instruction feedback control to display the information presented by the display interface 20 Multimedia information 21; to achieve contactless control effect.

2. The human-machine interface device 100 with hand posture control of the present invention provides convenience in use for people with disabilities or users who are inconvenient to directly control the human-machine interface device 100 through a contactless control method.

3. The human-machine interface device 100 with hand posture control of the present invention can provide users with the ability to create or modify hand postures 31 and operation instructions to adapt to different users' operating habits, thus improving the breadth of use and convenience.

The above embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. All modifications or changes that do not violate the spirit of the present invention fall within the scope of the invention.

100: Human-machine interface device 10:Machine body 11: Shell 20:Display interface 21:Multimedia information 30:Database 31:Hand posture 40:Photography module 50:Control module 60:Edit module 70:Financial module H:Hand image I: skeleton image P: fulcrum S: support rod

Figure 1 is an architectural block diagram of the present invention. Figure 2 is a schematic diagram of hand postures and corresponding operation instructions of the present invention. Figure 3 is a hand image marked with a skeleton image according to the present invention. Figure 4 is a schematic diagram (1) of an embodiment of the present invention, showing a user's contactless control of multimedia information. Figure 5 is a schematic diagram (2) of the embodiment of the present invention, showing hand postures and corresponding operation instructions in response to multimedia information.

100: Human-machine interface device

10:Machine body

20:Display interface

30:Database

40:Photography module

50:Control module

60:Edit module

70:Financial module

Claims (10)

A human-machine interface device with hand gesture control, which includes: One machine body; A display interface, which is provided on the machine body and provides for displaying multimedia information; A database is provided on the machine body and coupled to the display interface. The database contains a plurality of hand postures and an operation command corresponding to each hand posture; A camera module, which is installed on the body of the machine. The camera module provides an operating image of the user's contactless control of the multimedia information; and A control module is provided on the machine body and is coupled to the display interface, the database and the camera module. The control module receives the operation image from the camera module, and the control module controls the operation. The image extracts a hand image through an image calculation process, and a skeleton image is marked by the hand image. The control module compares the skeleton image with each hand posture existing in the database to obtain the matching one. A hand gesture and the corresponding operation command, the control module will feedback the obtained operation command to the multimedia information to perform contactless control of the multimedia information. The human-machine interface device with hand posture control as described in claim 1, wherein the skeleton image has a plurality of fulcrums and a plurality of struts, each fulcrum corresponding to each knuckle of the hand and the junction of the hand and the wrist, each A rod is connected between two adjacent points. The human-machine interface device with hand gesture control as described in claim 2, wherein each fulcrum has a unique code, and the control module determines the positional relationship between the fulcrums to obtain matching data from the database. One of the hand poses. The human-machine interface device with hand gesture control as described in claim 1, wherein the camera module can dynamically track the operation image of the user's contactless control of the multimedia information. The human-machine interface device with hand gesture control as described in claim 4, wherein the operation image is a dynamic image, and the control module divides the operation image into a plurality of stage operation images according to time sequence, and the control module divides each stage into One stage operates the image to perform the image operation and processing, and generates a plurality of hand images marked with the skeleton image according to the time sequence, and obtains the operation instructions from the database according to the time sequence, and the control module will obtain all the hand images. The operation instructions are fed back to the multimedia information to perform continuous contactless control of the multimedia information. The human-machine interface device with hand gesture control as described in claim 1 further has an editing module coupled to the display interface and the database. The editing module can edit each hand gesture in the database. The hand postures and corresponding operation instructions are displayed on the display interface, and the editing module can select one of the hand postures for editing to change the selected hand posture and the corresponding operation instructions. The human-machine interface device with hand gesture control as described in claim 6, wherein the editing module is further coupled to the photography module and the control module, and the editing module can enable the photography module to shoot the same This image is sent to the control module to generate the hand image marked with the skeleton image through the image calculation process. The editing module can replace the hand image marked with the skeleton image. One of the hand postures or settings in the database corresponds to one of the operation commands. The human-machine interface device with hand gesture control as described in claim 1 further has a financial module coupled to the display interface and the control module. The financial module has a checkout option. The billing options are displayed on the display interface in conjunction with the multimedia information. The control module can feedback the obtained operation instructions to the multimedia information and click the checkout option to perform contactless control of the checkout of the multimedia information. Operation, the financial module can present a consumption amount on the display interface and provide payment of the consumption amount. The human-machine interface device with hand gesture control as described in any one of claims 1 to 8, wherein the control module marks the skeleton image on the hand image through a data labeling program (LabelImg). The human-machine interface device with hand gesture control as described in claim 9, wherein the image processing is computer vision combined with deep learning.

Images