CN111258429A - Man-machine interaction method, system and device for controlling electronic equipment - Google Patents
Man-machine interaction method, system and device for controlling electronic equipment Download PDFInfo
- Publication number
- CN111258429A CN111258429A CN202010064640.4A CN202010064640A CN111258429A CN 111258429 A CN111258429 A CN 111258429A CN 202010064640 A CN202010064640 A CN 202010064640A CN 111258429 A CN111258429 A CN 111258429A
- Authority
- CN
- China
- Prior art keywords
- head
- axis
- movement
- angular velocity
- human
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/10—Earpieces; Attachments therefor ; Earphones; Monophonic headphones
- H04R1/1016—Earpieces of the intra-aural type
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Acoustics & Sound (AREA)
- Position Input By Displaying (AREA)
Abstract
The invention provides a human-computer interaction method, a system and a device for controlling electronic equipment, wherein the method can point and click (point-and-click) to control an electronic product just by rotating a head, and is mainly characterized in that an angular velocity sensor is additionally arranged on an earphone (such as a Bluetooth earphone) to detect and perceive the up-down, left-right movement (direction and displacement) and actions (such as nodding and shaking) of the head along the watching direction of eyes, so that a more natural (such as where a cursor points when the eyes watch) and more convenient human-computer interaction mode is realized, and the input of information can be realized under the support of a visual soft keyboard user interface output by a screen of the electronic equipment. The microphone of the original earphone is combined to input sound, so that more complex head movement and sound control combination can be realized; by using the sound output function of the original earphone, the user output (such as outputting voice prompt or voice menu) and the user input can be simultaneously realized.
Description
Technical Field
The present invention relates to a method, a system and a device for human-computer interaction, and more particularly, to a method, a system and a device for controlling an electronic product and inputting information by using a point-and-click (point-and-click) to realize human-computer communication.
Background
A human-machine interaction (HMI) system is composed of a user input device and a user output device, and the user output device is composed of the earliest printer and an indicator lamp, and has been developed to the display screen commonly used at present, even 3D image projection, Virtual Reality VR glasses (Virtual Reality) and augmented Reality ar (augmented Reality).
The user input technology is an important ring of human-computer interaction, the use experience and the use efficiency of a user are directly influenced, and user input equipment is developed from the earliest keys, a keyboard, a mouse, voice input and a stylus pen to the capacitive touch screen which is popular at present, touch is achieved by hands, and the user input equipment does not need to use the stylus pen and even utilizes the cardioreading technology of brain wave input. These input methods are more natural and more convenient than one.
Each time of progress of the man-machine interaction technology revolutes the IT industry, all operations can be completed by pointing And Clicking (Points And Clicking) a computer-made Windows And Macintosh created graphical interfaces; by the Touch Screen (cell) mobile phone age, it is believed that many people will not forget the novelty experienced when a user first sees a picture displayed on a Screen in an advertisement by opening and closing a finger.
At present, a touch screen is well used, but as electronic equipment is smaller and smaller, a screen is also smaller and smaller, fingers cannot be small, the fingers cannot press the touch screen well, letters are input by using a virtual QWERTY keyboard of a mobile phone touch screen at present, the letters or numbers beside the mobile phone touch screen are often input by mistake due to the large fingers, and for some users who are inconvenient to vacate two hands for operation, for example: for example, a driver, a disabled person, a worker holding tools, a fireman, a police, and the like, the touch screen is not a good choice, the capacitive touch screen is not a good choice for a person wearing gloves, the gesture input also requires a hand, and is not a good choice, and the input mode is more convenient and natural than the touch screen, and the two hands can be liberated to do other things.
Face recognition is now popular but is limited to access authorization (access authorization), such as unlocking a cell phone, since computer vision based face recognition technology is mature, it is easy to recognize facial expressions, movements and directions of movement, why it is not used as user input for human-computer interaction.
The method is characterized in that face expression, action and movement direction are detected by face recognition to serve as user input, and then the face recognition technology based on optical photography is low in recognition rate and high in power consumption when light is weak.
Also, techniques based on optical recognition of head movement utilize photocells, photosensors, or tof (time of flight) to track head or eye movement, but these methods emit infrared or laser light and reflect off of the eye, which is not good for the eye.
In view of the above existing factors, it is a topic worth discussing how to better utilize a new human-computer interaction method to achieve the control of electronic products and to solve the operational convenience.
Disclosure of Invention
The present invention provides a method for human-computer interaction, which uses head point-and-click to realize human-computer communication for controlling electronic products and inputting information.
The simplest human-computer interaction system consists of user input and output equipment, for example, the most common user output equipment is a screen, the user input equipment is a key, all input operations are completed only by a simple 5-way navigation key (navigation key), the 5-way navigation key comprises an upper direction key, a lower direction key, a left direction key, a right direction key and an ENTER confirmation key, and if a Back key (Back) is added, the operation can be faster.
The mouse simulates 5-direction navigation keys, and the mouse Points and Clicking can complete all input operations.
Clicking the head to click, which is equivalent to pressing a confirm ENTER key; rocking the head to move back corresponds to a back key (backspace or ESC).
Therefore, the mouse does not need to be sat on a desk to play or stand to wave the laser pen, too much activity space does not need to be occupied, two-dimensional surface plane space for operating the mouse to move does not need to be reserved, and the mouse can move in a free 3D space so as to achieve the effect of convenience in operation and control.
Further details regarding other functions and embodiments of the present invention will be described below with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the implementation examples or the prior art solutions of the present invention, the drawings used in the implementation examples or the technical descriptions will be briefly introduced below, it is obvious that the drawings in the following description are only some implementation examples described in the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is an example of one use of the present invention.
FIG. 2 is a system composition and architecture of the present invention.
FIG. 3 is an example of a setup interface of the present invention from which control behavior corresponding to head movements or actions may be defined.
Fig. 4a and 4b are examples of a "head point and click" (point and click) manipulation user interface in the present invention, which is a manipulation full graphic menu and a simple menu, respectively.
FIG. 5 is an example of the present invention entering information with a Head point and click.
Fig. 6 shows the mounting position of the sensor of angular velocity on the headset, the coordinate system used and the direction of rotation of the head movement in 3D space in accordance with the present invention.
Fig. 7a is a plot of angular velocity in three axes measured for right to left head movement in the present invention.
Fig. 7b is a graph of angular velocity in three axes measured for a head moving from left to right in the present invention.
Fig. 7c is a graph of angular velocity in three axes measured from the head moving upward from below in the present invention.
Fig. 7d is a graph of angular velocity in three axes measured for the head moving from top to bottom in the present invention.
Fig. 7e is a graph of angular velocity in three axes measured for nodding motion (rapid head movement from top to bottom) in the present invention.
Fig. 7f is a graph of the angular velocity of three axes measured for the panning motion (fast head left and right movement) in the present invention.
Fig. 7g illustrates how angular velocity can be integrated using the trapezoidal area method to obtain the rotation angle in the present invention.
FIG. 8 is a flowchart illustrating the overall software process from recognizing head movements to completing human-computer interaction in the present invention.
Detailed Description
The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown. However, the present invention should not be construed as limited to the embodiment examples set forth herein, but rather, these embodiment examples are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
The following description is given with reference to fig. 1 to 7 for further understanding of the present invention in terms of description and implementation principles:
as shown in fig. 1, a wireless bluetooth headset 103 is selected as a carrier for following head movements, a rotation sensor 104 is additionally installed on the headset 103 to detect and sense the up-down, left-right movement direction and displacement, movement (such as nodding, shaking, etc.) of the head following the eye gaze direction, head movement track is tracked, and the detected movement events are encoded into corresponding control instructions and then transmitted to the electronic devices 101 to be controlled through a wireless or wired network 102 (such as bluetooth), after receiving the control instructions, the electronic devices 101 interpret and execute corresponding preset movements 303 and 304, a user can move a cursor by shaking the head and shaking the brain, and the moving distance of the cursor is proportional to the displacement of the head movement; clicking the head can be achieved (Clicking), shaking the head can be backed (backspace or ESC), a sound input function of a microphone 209 of an original earphone is combined, the picture can be amplified by blowing the whistle head upwards, and the picture … can be reduced by blowing the whistle head downwards, so that a more natural (such as where the cursor points when looking at the touch screen) and more convenient man-machine interaction mode is realized compared with the existing touch screen, and the information input can be realized under the support of the visual soft keyboard user interface 502 of the host end 101.
The voice output 208 of the original earphone is utilized to even realize the user output (such as outputting voice prompt or voice menu), the user input and the user output are integrated, and the human-computer interaction can be completed only by using the simple electronic equipment (such as a music player) which does not need to output a complex user interface and does not have a screen and inputting and outputting 2-in-1 equipment to prompt by voice, nod or shake the head.
The reason why the present embodiment selects earphones as the carrier rather than glasses or the like is to reduce the foreign body sensation and to take care of the wearer who cannot wear more than one pair of glasses, and is to use the sound input of the earphones to make more interactive action combinations.
Referring to fig. 2 again to explain the system composition and architecture, the system is composed of a Human Interface Device (HID) earphone 200 and a Host electronic Device 210, which are connected together in a wireless (e.g. Bluetooth/Zigbee/WiFi, etc.) or wired manner 102, wherein the Human Interface Device (HID) earphone 200 is disposed on the head with the earphone 103 as a carrier, and detects and identifies the up-down, left-right movement of the head, the direction and displacement of the nodding head, the shaking head movement, etc., and encodes the detected direction and displacement of the movements and the movement event of the tracked head movement track into corresponding control commands; a Human Interface Device (HID) headset 200 includes a micro-electromechanical system (MEMS) Angular velocity rotation Sensor 202 and a microprocessor; the microprocessor is internally provided with a preset action event parameter table corresponding to head movement and behavior control, correspondingly detects the head rotation direction and amplitude through an angular velocity sensor to obtain the movement angular velocity, calculates, identifies and judges which direction of movement and movement displacement or preset action event occurs to the head, and packs and codes data according to key values or axis codes of standard input equipment and a man-machine input interface protocol; and then transmitted to the electronic device through the wireless or wired transmission device. The main electronic device 210 interprets and executes the corresponding preset action according to the received control command, and outputs and displays the control command, so that the head movement input event and the output display are corresponding to each other, and the human-computer interaction is completed together.
As shown in fig. 2, the HID headset is composed of a headset with a digital three-axis angular velocity sensor 202, and the headset is composed of a microphone 209, a receiver 208, an audio Codec (ADC/DAC converter) 207, a microprocessor 201, a memory 205, a wireless or wired data transmitter 203, an antenna or wire 204, a power supply 206, a USB interface and charger 211, and a power on/off/pairing/button 212.
HOST device (HOST) 210 may be an electronic device or computer (e.g., cell phone, tablet, PDA, e-book, PC, laptop, projector, radio, walkie-talkie, set-top box, gaming device, etc.).
The motion signal processing flow is further described with reference to fig. 2 and 3: the digital three-axis angular velocity sensor 202 is used for detecting the head rotation direction and amplitude to obtain the angular velocity and transmitting the angular velocity to the microprocessor 201, the microprocessor 201 calculates, identifies and judges which direction of head movement or preset action event occurs, in other words, the y axis of the angular velocity rotation sensor 202 is parallel to the head Yaw rotation (figure 6), the z axis is parallel to the head Pitch rotation, and the x axis is parallel to the head Roll rotation, and is used for detecting and identifying the head up-down left-right movement and head nodding and shaking actions along the direction of the head following the eyes, and high-pass filtering in the angular velocity rotation sensor 202 is used for filtering low-frequency static noise sound and obtaining movement angular velocity data through debounce processing and transmitting the movement angular velocity data to the microprocessor 201.
And according to the control tables 303, 304, 305 corresponding to the preset head movements or actions in fig. 3, the received action events are encoded (generally, coded as key values or axis codes of standard input devices) and packed according to a man-machine input interface protocol (such as HIDprotocol), and then transmitted to the host 210 through the wireless or wired data transmitter 203.
After receiving the data, the host 210 is decoded by a device driver in a kernel of a host built-in operating system and then converted into a standard input event, and an input event stream is added until the input event stream is scheduled to a corresponding display window by software, so that the head movement input event and the user output display correspond to each other, and the human-computer interaction is completed together.
Please refer to fig. 2 to describe the processing flow of the earphone sound input signal: the microphone 209 picks up the analog audio signal, and sends it to the audio codec 207 for A/D converter to be converted into digital signal, which is processed by the microprocessor 201 and temporarily stored in the memory 205 to be converted into audio stream (Media Flow) to be sent to the wireless or wired data transmitter 203 for further transmission to the host 210.
Please refer to fig. 2 again to describe the processing flow of the earphone sound output signal or the alert sound:
after the HID headset 200 is connected to the pairing host 210 (for example, bluetooth pairing), the host considers the HID headset 200 as a standard HID input device and a standard audio input and output device, as long as the host sets audio input and output from the headset (bluetooth or wired), all sounds (such as voice menus, alert tones, music, conversation sounds, ring tones, etc.) emitted from the host are transmitted to the wireless or wired data transmitter 203 in a wired or wireless manner, decoded by the microprocessor 201 and temporarily stored in the memory 205, transmitted to the audio codec 207 for D/a conversion into analog signals, and then amplified in the audio codec 207 for power amplification and then the earphone 208 is driven to emit sound.
The control definition and settings corresponding to the preset head movements are explained with reference to fig. 3:
as shown, the setup program is typically stored in a computer and transmits setup parameters to the headset via USB 211; or in the built-in software of the host 101 or in the plug-in APP, and these setting parameters are exchanged when the host 101 and the headset 102 are connected in a pairing manner.
The output of the headset 103 after head movement recognition corresponds to various detected head movement event codes 304, which are generally standard input device key values corresponding to various head movements, such as: the head moves upwards to correspond to the key value of the upwards moving cursor key; the key value of the ENTER key is correspondingly pressed by pointing the head; the key value of a Backspace/Escapes key is correspondingly pressed by shaking the head;
in addition to this, a function allowing disallowed head movement recognition 302; and pulling the slider 306 to adjust the angular velocity threshold to adjust the detection sensitivity; 305 is a drop down menu for the head action event 304, with 2 options, one is NONE, representing no setting, and the other is a preset value.
Please refer to fig. 4a and 4b to illustrate an example of manipulating a user menu by pointing and clicking (Head Point-and-Click) with a Head, wherein fig. 4a is an example of manipulating a menu of a full graphic interface composed of icons, such as a main menu of a smart phone; fig. 4b is an example of text menu manipulation for a simple electronic device using a black and white screen.
401 is an icon of each application APP, when the head moves along with the direction of eye gaze, the cursor 402 also moves, when moving to the icon of the APP to enter, the user opens the application by pointing the head (equivalent to CLICK), and the user exits or closes the application by shaking the head.
Also for a simple screen electronic device (fig. 4b), when the head moves up and down, the highlight bar 404 moves up and down along with the head, the head is clicked (rather clicked) to enter the item of the menu, and the head is shaken to exit and return to the main menu.
Fig. 5 illustrates an example of inputting information by pointing and Clicking (Head Point-and-Clicking) with a Head, taking a social chat APP as an example, to input information, a cursor 501 is moved up and down and left and right by the Head to a Point after an input box 506, then a display screen pops up a soft keyboard 502, a cursor 501 is moved up and down and left and right by the Head to a letter to be input, the input is performed by pointing the Head, then the cursor is moved to a SEND (SEND) key, and a click clicks to SEND the information. 505 is a chat username, 504 is the other user identification to chat with user 505, and 503 is chat content.
Referring to fig. 6, a coordinate system used when the angular velocity sensor 601 is mounted on the earphone and a rotation direction of the head movement and motion in the 3D space are described, as shown in the figure, when a person stands or sits on the earphone and does not move the head, the head movement is limited by the neck and generally cannot move arbitrarily, so the head movement is not translation, but rotates around the neck as an axis and a rotation angle of not more than +/-180 degrees, and when the head moves left and right or shakes, the head rotates around the Yaw axis 606, and when the head moves up and down or nods, the head rotates around the Pitch axis 605.
The sensor 601 is positioned so that the y-axis of the sensor's coordinate system 603 is parallel to the head Yaw 606, the z-axis of 603 is parallel to the head Pitch axis, and the x-axis is parallel to the head Roll axis 604, so that the head moves up and down or nods around the z-axis and moves left and right or shakes around the y-axis.
As shown in 601, a digital three-axis angular velocity sensor, for example: l3GD20H of ST company is only 3x3x1 mm in size, and is different from an old analog angular velocity sensor which outputs an analog voltage signal to reflect the size of angular velocity, and can directly output an angular velocity value because the analog signal is converted into a digital signal through ADC sampling, the angular velocity sensor can report the angular velocity of equipment along 3 coordinate axes, the angular velocity sensor has positive and negative values, the positive value indicates that the equipment rotates anticlockwise, the negative value indicates that the equipment rotates clockwise, the direction of the angular velocity is obtained by a right-hand spiral rule, and a High-Pass Filter (High-Pass Filter) and a Low-Pass Filter (Low-Pass Filter) are built in the sensors, so that some direct-current or High-frequency noise interference can be conveniently removed.
The sensor of angular velocity also typically provides an interrupt function, i.e. detects that the value of angular velocity is above or below a set Threshold (Threshold) and outputs an interrupt signal to the microprocessor 201. in the sensor 601 interrupt register you can choose which axis has a value above or below this Threshold (this Threshold can be adjusted by 306 in fig. 3), and also can combine the axes you want to detect. The interrupt register may also define a DURATION (DURATION) to define the time of detection, when this time is reached and a threshold is exceeded an interrupt may be triggered; a practical motion and Wake-up (motion and Wake-up) interrupt is also provided for power management, i.e. the headset is in a power-saving sleep state if not rotated at all times, and enters a normal operating mode only if the headset is rotated.
Referring to fig. 6 with reference to fig. 7a to 7g, a method for implementing "Point and click" is mainly described, that is, a head movement direction (Pointing) and a Point action (Clicking) are detected and recognized, and a head movement direction and a movement distance are detected to implement the Pointing (Pointing), so that the movement of the cursor can be controlled according to the 2 indicators.
FIG. 7a is a plot of angular velocity in three axes measured as the head moves from right to left:
as can be seen from FIG. 6, when the head moves left and right or shakes, the head is rotated around the Yaw axis 606 (i.e. y axis), and the other 2 axes are not rotated, so that the motion from right to left is characterized by the angular velocity of the y axis being positive (indicating rotation in the positive direction, counterclockwise) and the angular velocities of the x and z axes being substantially unchanged, and as long as the angular velocity of the y axis exceeds a certain threshold value and has positive polarity, and the angular velocities of the x and z axes do not change much, which is measured at a certain time (DURATION, e.g. 50 ms), we consider that the head moves to the left. Written as an expression is:
Ay>0&Ay>threshold&Ax<∣thresholdmin∣&Az<∣thresholdmin- (where Ay represents the angular velocity of the y-axis; Ax represents the angular velocity of the x-axis; Ax and Az)<∣thresholdmin| represents that the x and z axis angular velocities do not vary much, within a small range).
Fig. 7b is a plot of the angular velocity of the head measured from left to right for three axes, the motion from left to right being characterized by the y-axis angular velocity being negative (indicating rotation in the negative direction, clockwise) and the x-and z-axes angular velocity being substantially constant, as long as the y-axis angular velocity is measured over a certain time (DURATION, e.g., 50 ms) above a certain threshold and the polarity is negative, and the x-and z-axes angular velocity do not vary much, we consider that the head movement to the right has occurred. Written as an expression is:
Ay<0&Ay>∣threshold∣&Ax<∣thresholdmin∣&Az<∣thresholdmin| where Ay represents the angular velocity of the y-axisAx represents the angular velocity of the x-axis; ax and Az<∣thresholdmin| represents that the x and z axis angular velocities do not vary much, within a small range).
Fig. 7c shows the measured angular velocity curves of the three axes of the head moving from bottom to top, and we can see from fig. 6 that the head moves up and down or nods the head, rotating around the Pitch axis 605 (i.e. z axis), so the characteristics of the motion from bottom to top are that the angular velocity of the z axis is negative (indicating rotating in the negative direction, clockwise) and the angular velocities of the x and y axes are not changed, and as long as the angular velocity of the z axis is measured to exceed a certain threshold value and the polarity is negative within a certain time (DURATION, e.g. 50 ms), and the angular velocities of the x and y axes are not changed greatly, we consider that the motion from top to top has occurred. Written as an expression is:
Az<0&Az>∣threshold∣&Ax<∣thresholdmin∣&Ay<∣thresholdmin- (where Ax represents the angular velocity of the x-axis, Ay represents the angular velocity of the y-axis; Az represents the angular velocity of the z-axis; Ax and Ay<∣thresholdmin| represents that the x and y axis angular velocities do not vary much, within a small range).
Fig. 7d shows the measured angular velocity curves of the three axes of the head moving from top to bottom, and we can see from fig. 6 that the head moves up and down or nods the head, rotating around the Pitch axis 605 (i.e. z axis), so the motion from top to bottom is characterized by the z axis angular velocity being positive (meaning rotating in the positive direction, counterclockwise) and the x and y axes angular velocities being constant, and we consider that this motion has occurred as long as the z axis angular velocity exceeds a certain threshold (e.g. 50 ms), the polarity is positive, and the x and y axes angular velocities do not vary greatly, at a certain DURATION (e.g. 50 ms). Written as an expression is:
Az>0&Az>∣threshold∣&Ax<∣thresholdmin∣&Ay<∣thresholdmin- (where Ax represents the angular velocity of the x-axis, Ay represents the angular velocity of the y-axis; Az represents the angular velocity of the z-axis; Ax and Ay<∣thresholdmin| represents that the x and y axis angular velocities do not vary much, within a small range).
As shown in fig. 7 a-7 d, the moving direction is detected, and the distance of the cursor to move to the moving direction is determined by detecting the Displacement (Displacement) of the movement, and since the head movement is the rotation around the neck, the distance of the head movement is the radian of the head rotation, and the radian of the rotation can be obtained by integrating the measured angular velocity, and the distance of the cursor movement can be obtained by multiplying a coefficient k, so that the distance of the cursor movement is proportional to the amplitude of the rotation.
Since the angular velocity is defined as：
∵A = dθ / dt(θThe degree of arc traversed, t is the time,Ais angular velocityAngular velocityUnit is rad/s)
∴=》θ=
(before each integration, care is taken to high-pass filter, otherwise the integration will accumulate very large drift errors due to the influence of dc and very low frequency components).
The integration isA （t）Curve to time axis t0-tnThe area of (see fig. 7 g), the processor 201 of the earphone cannot use a powerful floating-point arithmetic unit due to power consumption and cost, and the microprocessor 201 has difficulty in calculating the integral because of limited arithmetic capability, so that the curve area and the trapezoid area can be used as a near value.
Fig. 7g illustrates how angular velocity is integrated by trapezoidal area to obtain the angle of rotation, as shown, with angular velocity a on the Y-axis 702,A （t）the curve 701 may be implemented by a segment of broken line At0At1…At(n-1)AtnInstead, the sum SA of the areas of the individual trapezoids can be usedt0At1t1t0+ SAt1At2t2t1+…+ SAt(n-1)Atntnt(n-1)To show the area enclosed by the curve and the time axis 703, the expression is written
θ=
≈SAt0At1t1t0+ SAt1At2t2t1+…+ SAt(n-1)Atntnt(n-1)
= [ (At0+ At1)/2*(t1-t0)]+ [ (At1+ At2)/2*(t2-t1)]+…+ [ (Atn-1+ Atn)/2*(tn-tn-1)](according to the trapezoidal area formula = (long side A)t(n-1)tn-1+ short side Atntn）/2*(tn-tn-1),)
Since the angular velocity is sampled in equal time, so (t)n-tn-1)=…= (t2-t1)= (t1-t0) Is = Δ, then
Θ= [ (At0+ At1)/2+ (At1+ At2)/2 +…+ (At(n-1)+ Atn)/2]*∆t
= [(At0+ Atn)/2 +( At1+ At2+…+ At(n-1)]*∆t
The above equation is easier to calculate the addition by software programming than to calculate the integral directly, and the sample data is accumulated and multiplied by the sampling interval, so that the calculation can be performed without using a powerful CPU.
Fig. 7e is the angular velocity curve of three axes measured by nodding action (rapid movement of head from top to bottom), and we can see from fig. 6 that when the head moves up and down or nods, it rotates around the Pitch axis 605 (i.e. z axis), therefore, the motion characteristic of nodding action is that the angular velocity of z axis changes strongly with positive and negative alternation and the angular velocity of x and y axes is not changed, as long as the acceleration of z axis is measured to exceed a certain threshold value within a certain time (DURATION, e.g.: 100 ms), the polarity is positive and negative alternation, and the angular velocity change of x and y axes is not large, we consider that nodding action occurs, thus recognizing the nodding action, and writing the expression:
∣Az∣>threshold&Ax<∣thresholdmin∣&Ay<∣thresholdmin| where Ax represents the angular velocity of the x-axis, Ay represents the angular velocity of the y-axis, and Az represents the angular velocity of the z-axis.
Fig. 7f is the angular velocity curves of three axes measured for the head shaking motion (fast head left and right movement), and we can see from fig. 6 that the head moves left and right or shakes around the Yaw axis (y axis) 606, so the motion characteristics of the head shaking motion are that the angular velocity of the y axis changes strongly alternately positive and negative and the angular velocities of the x and z axes are not changed, as long as the acceleration of the y axis measured within a certain time (DURATION, e.g.: 100 ms) exceeds a certain threshold value and the polarities alternate positive and negative, and the angular velocities of the x and z axes do not change greatly, we consider that the head shaking motion occurs, and thus recognize the head shaking motion, and write the expression:
| Ay | threshold & Ax | threshold dmin | Az | threshold dmin | where Ax represents angular velocity of the x-axis, Ay represents angular velocity of the y-axis, and Az represents angular velocity of the z-axis.
For the input sound detection and sound recognition method which combines the head movement and sound control to control the function:
the microphone 209 picks up the analog audio signal, and sends it to the audio codec 207, where it is converted into a digital signal by an AD converter, and the microprocessor 201 determines: a sound is considered to be sounded as long as the measured sound intensity exceeds a certain threshold for a certain time (DURATION, e.g. 20 ms).
In summary, the present invention can be illustrated by fig. 8 as the whole software processing flow from recognizing head movement to completing human-computer interaction:
firstly, turning on power supplies of the HID earphone and the host, pairing 801 the HID earphone and the host, establishing a master-slave connection relation, exchanging setting parameters between the master and the slave, wherein the host is electronic equipment or a computer to be controlled, and the earphone is the slave.
The microprocessor 201 controls the digital triaxial angular velocity sensor 202 to detect the head movement direction and displacement and movement 802, and opens the high pass filter in the digital angular velocity sensor 202, filters out the low frequency static noise sound, and gets the movement angular velocity through the debouncing process and transmits to the microprocessor 201, and identifies and judges which direction Pointing (Pointing) and calculates the movement displacement occurs according to the movement characteristics of each head movement described in fig. 7, and identifies the head movement event 803 in which the head nodding or shaking occurs.
And encode 804 the received action event according to the control definition (fig. 3) corresponding to the preset head movement, which is generally encoded as an axis code and a key value of a mouse button corresponding to a standard input device mouse or Pointer, for example: the head left-right movement and movement displacement d are mapped to an X-axis code +/-d of the Android standard mouse device and used for reporting the displacement d of the cursor along the X axis, wherein +/-represents the movement towards the right, and' -representsthe movement towards the left; move up and down and move displacement d is mapped to the y-axis code of a standard mouse device: +/d, which reports the displacement d of the cursor along the y-axis, "+" represents moving up, "-" represents moving down, and action events such as: clicking and Clicking, and coding into a left mouse key value;
or all the keys can be mapped to key values of the keys, for example: head up-down-left-right movement is mapped to up-down-left-right direction key values of a corresponding keyboard, movement displacement is mapped to the times of pressing the direction keys, head nodding is mapped to an ENTER key value, head shaking is mapped to an ESC or Back key …
After encoding, the data 805 is packed according to a human interface protocol (e.g., HID protocol) and transmitted to the host 210 through the wireless or wired data transmitter 203. After receiving the data, the host 210 is decoded by a hid (human interface device) device driver in host software and converted into a standard input event, and adds an input event stream until the host is dispatched to a corresponding output display window by the software to execute a corresponding action 806, and feeds back an execution result on a screen, so that the head movement corresponds to the action input event and the output display, and the human-computer interaction is completed together.
Summarizing, the present invention has the following advantages.
1. The earphone (such as a wireless Bluetooth earphone) is not only an audio input/output device, but also a Human Interface Device (HID) input and output device. The input may be a head "point-and-click" input, or may be a voice input, or even a voiceprint input as an access authorization; the audio output of the earphone is changed into an output device for man-machine interaction, and sound prompts and even sound menus are output for the user to input and select.
2. The invention integrates two devices into one, and integrates 2-channel user input (motion detection input and voice input) and 1-channel user output (voice output).
3. The invention can be operated naturally (for example, where the cursor points when looking at the user), does not occupy space when in use, can release hands to do other things, is beneficial to specific people (for example, drivers, disabled persons, workers holding tools, firemen, policemen and the like), and is particularly useful for people who are inconvenient to vacate hands to operate machines.
4. The head movement is not translation because the head movement is limited by the neck, but the head movement rotates by taking the neck as the axis and the rotation angle does not exceed +/-180 degrees, so that the sensor for measuring the angular velocity of the rotating movement is more suitable for use, and has the characteristic of simple detection compared with an acceleration sensor (MEMS Accelerometer) for detecting the linear movement.
5. Head Point-and-click (Head Point-and-click) recognition of MEMS-based angular velocity sensors has the advantages of small size, low power consumption and low cost compared to optical (e.g., photocell, TOF, camera) based Head potential recognition.
6. The head movements are combined with the sound picked up by the sound sensor on the headset to achieve more input control combinations.
7. The device can be controlled in a wireless remote mode, and is particularly suitable for application scenes of large-screen demonstration, such as an electronic blackboard or a projector.
8. Compared with glasses or other wearing equipment, the earphone is selected as a carrier, so that the earphone not only can follow the head to move up and down, left and right along with the direction watched by eyes, but also can reduce the foreign body sensation during wearing and prevent a user wearing the glasses from wearing a plurality of glasses, and the reason is that more interactive action combinations can be performed by utilizing the sound input and output of the earphone.
The above embodiments and examples are only for illustrating the preferred embodiments or implementations of the present invention, and are not to be construed as limiting the embodiments of the present invention in any way, and any person skilled in the art may make some changes or modifications without departing from the scope of the technical means disclosed in the present disclosure to provide other equivalent embodiments, but still should be regarded as the technical or implementation examples substantially the same as the present invention.
Claims (18)
1. In a human-computer interaction method for controlling an electronic device, the method comprising:
selecting a carrier with the functions of detecting and identifying the up-down, left-right movement of the head, nodding the head, shaking the head and the like, following the head to move, and pairing the carrier and the electronic equipment in a wireless or wired mode to establish a master-slave relationship; coding control commands, and coding the directions, displacements and action events of the movements detected by the head movement track into corresponding control commands; transmitting the control instruction to the electronic equipment in a wireless or wired mode; and the control instruction received by the electronic equipment is interpreted and executed by matching with a corresponding User Interface (User Interface), so that the head motion input event and the User output display are corresponding to each other, and the man-machine interaction is completed together.
2. The human-computer interaction method for controlling electronic devices of claim 1, wherein the carrier is a wireless earphone or a wired earphone, and an angular velocity rotation sensor is attached to the earphone to detect and identify the head moving up and down, left and right, and the head nodding and shaking movements following the direction of the eyes, and the high-pass filtering in the angular velocity rotation sensor is used to filter out the low-frequency static noise and the motion angular velocity data is obtained through the de-jittering process.
3. The human-computer interaction method for controlling an electronic device according to claim 2, wherein the angular velocity rotation sensor is installed so that a y-axis of the angular velocity rotation sensor is parallel to the rotation of the head Yaw, a z-axis is parallel to the rotation of the head Pitch, and an x-axis is parallel to the rotation of the head Roll, and is installed so as to rotate around the z-axis when the head is moved up and down or nodding or to rotate around the y-axis when the head is moved left and right or shaking while the person stands or sits on the stationary head, and the head movement is recognized based on the movement characteristics.
4. A human-computer interaction method for controlling an electronic device according to claim 3, wherein the method for recognizing the leftward head movement according to the movement characteristics is that the angular velocity of the y-axis is positive (indicating that the head is rotated in a forward direction, and counterclockwise) and the angular velocities of the x-axis and the z-axis are substantially constant, and the head is considered to be rotated to the left as long as the angular velocity of the y-axis exceeds a certain threshold and the polarity is positive, and the angular velocities of the x-axis and the z-axis do not vary much for a certain duration;
similarly, if the y-axis angular velocity is negative and the x-and z-axes angular velocities are substantially unchanged, then the head is considered to be turning to the right;
similarly, the method for recognizing the upward movement of the head is that the angular velocity of the z-axis is negative (indicating that the head rotates in the negative direction, clockwise) and the angular velocities of the x-axis and the y-axis are basically unchanged, and as long as the angular velocity of the z-axis exceeds a certain threshold value and the polarity is negative, and the angular velocities of the x-axis and the y-axis do not change much within a certain duration, the head is considered to move upward;
similarly, the angular velocity of the z-axis is positive (indicating rotation in the forward direction, counterclockwise) and the angular velocities of the x-and y-axes are substantially unchanged, and as long as the angular velocity of the measurement axis exceeds a certain threshold for a certain duration and the polarity is positive, and the angular velocities of the x-and y-axes do not change much, it is assumed that the head-down movement has occurred;
the motion characteristic of the nodding action is that the angular speed of the z axis is changed alternately in positive and negative directions and the angular speeds of the x axis and the y axis are unchanged, and as long as the measured acceleration of the z axis exceeds a certain threshold value within a certain duration time, the polarity is alternately in positive and negative directions, and the angular speeds of the x axis and the y axis are not changed greatly, the nodding action is considered to occur, so that the nodding action is identified;
the motion characteristic of the shaking motion is that the angular speed of the y-axis changes strongly in a positive and negative alternating mode, the angular speed of the x-axis and the angular speed of the z-axis do not change, and as long as the acceleration of the y-axis exceeds a certain threshold value within a certain duration, the polarity changes in a positive and negative alternating mode, and the angular speed of the x-axis and the angular speed of the z-axis do not change greatly, the shaking motion is considered to occur, and therefore the shaking motion is recognized.
5. The method of claim 1, wherein the control command is encoded by a microprocessor, the microprocessor has a preset action event parameter table corresponding to head movement and behavior control built therein, and accordingly detects a head rotation event by the angular velocity sensor and informs the microprocessor to interrupt processing of received angular velocity data, the microprocessor determines which direction of head movement or preset action event occurs according to the movement characteristics, encodes the events into key values of a standard input device (e.g., mouse, keyboard), encodes the movement displacement calculated according to the angular velocity into an axis code, and then packs the encoded data according to a human input interface protocol (e.g., HID protocol).
6. A human-computer interaction method for controlling an electronic device as claimed in claim 1, wherein the controlled User Interface (User Interface) of the electronic device has a device driver for decoding the received motion event code and converting the decoded motion event code into a standard input event.
7. The method as claimed in claim 1, wherein the control command is transmitted wirelessly or by wire, and the wireless mode can be bluetooth, Zigbee or WiFi.
8. A man-machine interaction method for controlling an electronic device according to claim 2, further comprising the steps of combining a microphone and an audio Codec (Codec) already present on the carrier (headset) and a memory connected to the microprocessor, collecting analog audio signals from the microphone, sending the analog audio signals to the Codec for a/D conversion into digital signals, and determining by the microprocessor that the received sound intensity is greater than a threshold value within a certain time period that a sound input event is considered to have occurred, so that more control functions are combined with the sound input and the head movement input, such as: the picture can be enlarged by blowing the whistle head upwards, and the picture can be reduced by blowing the whistle head downwards.
9. A human-computer interaction device for controlling an electronic device, comprising: the device can be paired with controlled electronic equipment to establish a master-slave connection relation, and a carrier of the device is configured to be positioned on the head of a user and move along with the head, and at least comprises an angular speed rotation sensor, a microprocessor and a memory; the y axis of the angular speed rotation sensor is parallel to the rotation of the head Yaw, the z axis is parallel to the rotation of the head Pitch, and the x axis is parallel to the rotation of the head Roll, so that the angular speed rotation sensor is used for detecting and identifying the up-down left-right movement and the head nodding and shaking actions of the head along the direction watched by eyes, filtering low-frequency static noise sound by utilizing high-pass filtering in the angular speed rotation sensor, and obtaining movement angular speed data through debouncing processing;
the microprocessor prestores a preset action event parameter table corresponding to head movement and behavior control in a memory, calculates, identifies and judges which direction of head movement or preset action event occurs by utilizing head rotation angular velocity detected by an angular velocity sensor, and packs data for coding according to key values or axis codes of standard input equipment and a human-computer input interface protocol (such as HIDprotocol); then the data is transmitted to the controlled electronic equipment through the wireless or wired transmission device;
the controlled electronic equipment comprises an input equipment driving program which is used for decoding the received action event codes and converting the decoded action event codes into standard input events, so that the head movement input events and the user interface output display are mutually corresponding to finish man-machine interaction together.
10. The Human-computer interaction Device for controlling an electronic apparatus according to claim 9, wherein the carrier employs a wireless or wired headset with HID (Human Interface Device Human interaction) functionality; the electronic device to be controlled may be a mobile phone, a tablet pad, a PDA, an electronic book, a PC, a Notebook, a projector, a radio, an intercom, a set-top box, a pager, a game device, etc.
11. A human-computer interaction device as claimed in claim 9, wherein the wireless device is Bluetooth/Zigbee/WiFi or the like.
12. The human-computer interaction device as claimed in claim 10, wherein the human-computer interaction HID headset further comprises an audio Codec (Codec) for converting an analog audio signal picked up by a microphone in the HID headset into a digital signal by an a/D converter through the Codec.
13. A human-computer interaction device as claimed in claim 12, further comprising a memory connected to the microprocessor for processing the digital signals, wherein the microprocessor determines that the received sound level is above a threshold level within a predetermined period of time and determines that a sound input event has occurred, such that by inputting the sound, the picture is magnified by moving the whistle blow upwards and the picture is reduced by moving the whistle blow downwards.
14. A man-machine interaction system for controlling electronic equipment is characterized by comprising a man-machine Interface device HID (human Interface device) and the electronic equipment, wherein the man-machine Interface device HID and the electronic equipment are paired in a wireless or wired mode to establish a master-slave relationship, the man-machine Interface device is arranged on a head and is used for detecting and identifying the up-down left-right movement of the head, the head nodding movement, the head shaking movement and other directions, displacement and action events along with the head movement, and the directions, the displacement and the action events of the movements detected by tracking the head movement track are coded into corresponding control instructions; the electronic equipment interprets and executes the corresponding preset action according to the received control command, and then outputs and displays the control command, so that the head movement input event and the output display are corresponding to each other, and the human-computer interaction is completed together.
15. A human-computer interaction system for controlling an electronic device as claimed in claim 14, wherein the human-computer interface device comprises a carrier with a MEMS Angular velocity rotation Sensor (MEMS Angular velocity Sensor) and a microprocessor; the microprocessor is internally provided with a preset action event parameter table corresponding to head movement and behavior control, correspondingly detects the head rotation direction and amplitude through an angular velocity sensor to obtain the movement angular velocity, calculates, identifies and judges which direction of movement and movement displacement or preset action event occurs to the head, and packs and codes data according to key values or axis codes of standard input equipment and a man-machine input interface protocol; and then transmitted to the electronic device through the wireless or wired transmission device.
16. The human-computer interaction system for controlling an electronic device of claim 15, wherein the carrier is a human-computer interaction hid (human Interface device) wireless or wired headset; the electronic device may be a mobile phone, a tablet PC pad, a PDA, an electronic book, a PC, a Notebook, a projector, a radio, an intercom, a set-top box, a pager, a game device, or the like.
17. The human-computer interaction system of claim 15, wherein the headset further comprises an audio Codec (Codec) for converting analog audio signals picked up by a microphone in the HID headset to digital signals through the Codec.
18. A human-computer interaction system as in claim 17 further comprising a memory coupled to the microprocessor for processing the digital signals, wherein the microprocessor determines that the received sound level is above a threshold level for a period of time and determines that a sound input event has occurred, such that by inputting the sound, the picture is magnified by moving the whistle blow up and the picture is reduced by moving the whistle blow down.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010064640.4A CN111258429A (en) | 2020-01-20 | 2020-01-20 | Man-machine interaction method, system and device for controlling electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010064640.4A CN111258429A (en) | 2020-01-20 | 2020-01-20 | Man-machine interaction method, system and device for controlling electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111258429A true CN111258429A (en) | 2020-06-09 |
Family
ID=70949025
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010064640.4A Pending CN111258429A (en) | 2020-01-20 | 2020-01-20 | Man-machine interaction method, system and device for controlling electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111258429A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113856191A (en) * | 2020-06-30 | 2021-12-31 | 广州拓火科技有限公司 | Man-machine interactive game system |
| CN114253397A (en) * | 2021-11-18 | 2022-03-29 | 深圳大学 | Intelligent equipment interaction system based on ear-wearing type inertial sensor |
| CN114374906A (en) * | 2021-12-16 | 2022-04-19 | 深圳创优声学科技有限公司 | TWS earphone limb language operating system and using method |
| WO2024031614A1 (en) * | 2022-08-12 | 2024-02-15 | Tze Yuk MAK | A human-interface device and a method for human-machine interface |
| US12039098B2 (en) | 2021-07-19 | 2024-07-16 | Samsung Electronics Co., Ltd. | Electronic device and operation method of electronic device for controlling external electronic device |
-
2020
- 2020-01-20 CN CN202010064640.4A patent/CN111258429A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113856191A (en) * | 2020-06-30 | 2021-12-31 | 广州拓火科技有限公司 | Man-machine interactive game system |
| CN113856191B (en) * | 2020-06-30 | 2024-04-12 | 广州拓火科技有限公司 | Man-machine interactive game system |
| US12039098B2 (en) | 2021-07-19 | 2024-07-16 | Samsung Electronics Co., Ltd. | Electronic device and operation method of electronic device for controlling external electronic device |
| CN114253397A (en) * | 2021-11-18 | 2022-03-29 | 深圳大学 | Intelligent equipment interaction system based on ear-wearing type inertial sensor |
| CN114253397B (en) * | 2021-11-18 | 2024-06-04 | 深圳大学 | Intelligent equipment interaction system based on ear-wearing inertial sensor |
| CN114374906A (en) * | 2021-12-16 | 2022-04-19 | 深圳创优声学科技有限公司 | TWS earphone limb language operating system and using method |
| WO2024031614A1 (en) * | 2022-08-12 | 2024-02-15 | Tze Yuk MAK | A human-interface device and a method for human-machine interface |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258429A (en) | Man-machine interaction method, system and device for controlling electronic equipment | |
| US10921882B1 (en) | Human-machine interaction method, system and apparatus for controlling an electronic device | |
| US20220083149A1 (en) | Computing interface system | |
| US10534447B2 (en) | Multi-surface controller | |
| CN107003750B (en) | Multi-surface controller | |
| CN107678542B (en) | Ring type wearable device and man-machine interaction method | |
| JP6275839B2 (en) | Remote control device, information processing method and system | |
| US8570273B1 (en) | Input device configured to control a computing device | |
| US9285840B2 (en) | Detachable sensory-interface device for a wireless personal communication device and method | |
| EP2538309A2 (en) | Remote control with motion sensitive devices | |
| US20120208639A1 (en) | Remote control with motion sensitive devices | |
| US20120229509A1 (en) | System and method for user interaction | |
| KR102297473B1 (en) | Apparatus and method for providing touch inputs by using human body | |
| CN102722240A (en) | Text information input system, handwriting input device and text information input method | |
| CN104597971B (en) | A kind of wearable computer | |
| CN110045843A (en) | Electronic pen, electronic pen control method and terminal device | |
| JP6270557B2 (en) | Information input / output device and information input / output method | |
| CN205302186U (en) | Virtual reality control system based on external input | |
| WO2016049842A1 (en) | Hybrid interaction method for portable or wearable intelligent device | |
| CN110717993A (en) | Interaction method, system and medium of split type AR glasses system | |
| WO2024001828A1 (en) | Wrist-worn-device control method and related system, and storage medium | |
| CN101732862B (en) | Double-controller combination | |
| CN110113248A (en) | Processing method, equipment and the computer readable storage medium of history message | |
| CN118012325A (en) | Gesture recognition device | |
| JP7560591B2 (en) | Method, computing system, and computer-readable storage medium for controlling a user interface - Patents.com |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |