CN112083832A - Transplantable touch screen device based on IMU and infrared induction - Google Patents
Transplantable touch screen device based on IMU and infrared induction Download PDFInfo
- Publication number
- CN112083832A CN112083832A CN202010930016.8A CN202010930016A CN112083832A CN 112083832 A CN112083832 A CN 112083832A CN 202010930016 A CN202010930016 A CN 202010930016A CN 112083832 A CN112083832 A CN 112083832A
- Authority
- CN
- China
- Prior art keywords
- module
- mouse
- infrared
- data
- ring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F3/0383—Signal control means within the pointing device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0412—Digitisers structurally integrated in a display
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
- Y02B40/00—Technologies aiming at improving the efficiency of home appliances, e.g. induction cooking or efficient technologies for refrigerators, freezers or dish washers
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Position Input By Displaying (AREA)
Abstract
The invention provides a portable touch screen device based on an IMU (inertial measurement Unit) and infrared induction. The mouse ring comprises a ring body, an inertia measurement module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module; the infrared sensing device comprises an infrared sensing device body, an infrared sensor, a miniature infrared emission module, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module; the mouse control device includes: the control device comprises a control device main body, and a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body. The invention realizes the function of an electronic touch screen by adopting the ring, and the user realizes the control of the target equipment by wearing the mouse ring to move, thereby having the advantages of convenient carrying and beautiful use, and simultaneously having higher transportability and reducing the economic cost of the user.
Description
Technical Field
The invention relates to the field of input equipment of electronic computers, in particular to a portable touch screen device with multiple sensing signals.
Background
Computer monitors, also commonly referred to as computer monitors or computer screens, are a means of displaying certain electronic document information onto a screen via a specific transmission device and then reflected to the human eye. At present, computer displays are divided into two categories, namely an LED display screen and a liquid crystal display screen, most displays do not have an input function, a mouse and a keyboard are required to be controlled, and the application of the traditional LED display screen and the liquid crystal display screen is limited in some scenes where the keyboard and the mouse are inconvenient to be externally arranged outdoors, in markets, in banks and the like. The electronic touch screen changes the interaction mode of the original mouse and keyboard, is very convenient and fast, but has higher cost, is the hardware of the computer and has no portability. In order to solve the above problems, a more convenient, faster and efficient human-computer interaction device is needed.
Disclosure of Invention
The invention aims to solve the problems of low efficiency and more required equipment of the existing man-machine interaction means.
The invention discloses a portable touch screen device based on an IMU (inertial measurement Unit) and infrared induction.
The mouse ring comprises:
the intelligent finger ring comprises a finger ring body, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module;
the ring comprises a ring body, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery, a first wireless charging module, a first wireless data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module, wherein the inertia measurement module is connected with the miniature infrared emitter module, the miniature infrared emitter module is connected with the first data acquisition module, the first wireless communication module is connected with the first rechargeable battery, and the inertia measurement module, the miniature infrared emitter module, the first data acquisition module, the first wireless communication module, the first rechargeable battery and the first wireless charging module are all arranged on;
the inertia measurement module is used for generating inertia measurement data according to the movement of the finger of the user after the ring body is worn on the finger of the user, and sending the inertia measurement data to the first data acquisition module; the miniature infrared transmitter module is used for transmitting infrared beams for the infrared sensing device to receive;
the first data acquisition module is used for sending the inertia measurement data to a mouse control device connected with target equipment through the first wireless communication module so that the electronic touch screen control device generates a mouse instruction according to the inertia measurement data and controls the target equipment according to the mouse instruction;
the first wireless communication module is used for receiving the inertia measurement data sent by the first data acquisition module and sending the inertia measurement data to the data processing module;
the first rechargeable battery is used for supplying power to each module in the mouse finger ring;
the first wireless charging module is used for charging the first rechargeable battery in a wireless charging mode.
The implantable touch screen device based on the IMU and the infrared induction is characterized in that the mouse ring is worn on the index finger of the right hand of an operator, the inertial measurement module measures the angular velocity information of three axes and the acceleration information of the three axes, and the information is used for controlling the movement of a mouse cursor and the operation of clicking, double clicking and roller pressing; when an operator moves the right hand and wants to control the mouse cursor, the inertia measurement module on the mouse ring worn by the index finger of the right hand records the three-axis acceleration information in the finger moving process, the obtained acceleration is used for carrying out 2 times of integration on time to obtain displacement, and the integration is iterated according to the following two integration formulas:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2， (1)
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2， (2)
wherein v (t) and v (t-1) respectively represent target speeds at t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the inertia measurement module at t and (t-1), Δ t represents a time interval between t and (t-1), and s (t) and s (t-1) respectively represent target displacements at t and (t-1);
and (3) iterating the two integral formulas of the formula (1) and the formula (2) to obtain the vertical displacement within the time t.
The infrared induction device comprises:
the infrared induction device comprises an infrared induction device main body, an infrared inductor, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module;
the infrared sensor, the second data acquisition module, the second wireless communication module, the second rechargeable battery and the second wireless charging module are all arranged on the infrared sensing device main body;
the infrared sensor is placed under the display screen, the length of the infrared sensor is greater than that of the display screen, and the infrared sensor is placed at a position where the infrared sensor can receive infrared beams emitted by the miniature infrared emitter module of a user when the user wears the mouse ring to click the display screen; when a user wears the mouse ring to generate horizontal offset on the display screen, the beam emitted by the miniature infrared emitter module of the mouse ring also generates corresponding horizontal offset, the infrared sensor calculates horizontal offset data generated by the movement of the finger of the user according to the position of the received infrared beam, and sends the horizontal offset data to the second data acquisition module;
the second data acquisition module is used for sending the horizontal position offset data to a mouse control device connected with target equipment through the second wireless communication module so that the mouse control device generates a mouse instruction according to the horizontal position offset data and controls the target equipment according to the mouse instruction;
the second wireless communication module is used for receiving horizontal position offset data sent by the second data acquisition module and sending the horizontal position offset data to the data processing module;
the second rechargeable battery is used for supplying power to each module in the infrared sensor device;
the second wireless charging module is used for charging the second rechargeable battery in a wireless charging mode.
The infrared sensor is placed under the display screen, horizontal position offset data is generated according to the movement of a user finger, the finger is positioned at (x, y) at the moment t, and the position of the finger moves to (x + dx, y + dy) at the moment t + dt, so that the infrared sensor senses the shielding of the finger and directly obtains the horizontal position of the finger at the moment; the infrared sensor determines the position information of the right hand and is used for controlling the mouse cursor to move on the horizontal plane.
The mouse control device comprises: the control device comprises a control device main body, and a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body;
the third wireless communication module and the physical communication interface are both connected with the data processing module;
the control device main body is fixedly connected with target equipment through the physical communication interface;
the third wireless communication module is used for receiving inertial measurement data sent by a mouse ring and sending the inertial measurement data to the data processing module; the third wireless communication module is also used for receiving horizontal position offset data sent by an infrared sensing device and sending the horizontal position offset data to the data processing module;
the data processing module is used for generating a mouse instruction according to the inertia measurement data and the horizontal position offset data, and sending the mouse instruction to the target equipment through the physical communication interface so as to control the target equipment;
and the physical communication interface sends a mouse instruction to the target equipment so as to control the target equipment.
The control device main body extracts multi-mode features of horizontal position offset data captured by the infrared sensor and information measured by the inertia measurement module on the mouse ring, inputs the multi-mode features into the LSTM model to obtain a recognition probability result of the right-hand index finger pressing, sends a control signal of the mouse to the data processing module if the probability value is larger than a certain threshold value, and abandons the recognition of the action if the probability is smaller than the certain threshold value.
The control device main body performs multi-mode feature extraction on horizontal position offset data captured by the infrared sensor and inertial measurement module measurement information on the mouse ring, and feature extraction methods for the inertial measurement module measurement information comprise normalization and amplitude absolute value mean value extraction methods.
The control device main body extracts multi-mode features from horizontal position offset data captured by the infrared sensor and information measured by the inertial measurement module on the mouse ring, inputs the multi-mode features into the LSTM model, and the LSTM model automatically learns the time sequence features of the input multi-mode features and outputs the probability that the features belong to each classification, so that the classification and identification categories to which the input multi-mode features most possibly belong are obtained and serve as the identification probability result of the right-hand index finger pressing.
The control device main body carries out multi-mode feature extraction on horizontal position offset data captured by the infrared sensor and inertial measurement module measurement information on the mouse ring, the feature extraction method for the active section of the inertial measurement module comprises a normalization method and an amplitude absolute value mean value extraction method, and the calculation formula of the normalization extraction method is as follows:
where | xiL represents the absolute value of the frame data in the data of the inertia measurement module, min represents the numerical value with the minimum absolute value in the movable section of the inertia measurement module, and max represents the numerical value with the maximum absolute value in the movable section;
the calculation formula of the amplitude absolute value mean value is as follows:
wherein MAV represents the extracted absolute mean feature, N represents the length of the sliding window, and xiThe amplitude of the frame signal is represented, and the mean value of the absolute values of the amplitudes is the mean value of the absolute values of all data in the sliding window.
The control device main body extracts multi-mode features of horizontal position offset data captured by the infrared sensor and information measured by the inertial measurement module on the mouse ring, and then inputs the multi-mode features into the LSTM model, wherein the LSTM model comprises the following specific steps:
s1, the forgetting gate layer controls the information passing through the memory cell by activating the function, which generates a value from 0 to 1 according to the output of the previous time and the input of the current time, to control whether the information learned at the previous time passes through or partially passes through the memory cell, the formula of which is as follows:
ft＝σ(Wf·[ht-1，xt]+bf)，
where σ is the activation function, ht-1Is the output of the previous time, xtIs the current input, bfIs the amount of the offset that is,ftis the output of a forgetting gate, WfTaking values for the weight;
s2, generating new information needing to be updated; the step comprises two parts, wherein the first part is that an input gate layer determines an updating value through a sigmoid activation function, the second part is that a tanh layer is used for generating a new candidate value which is added into a memory unit as a candidate value generated by a current layer, and the values generated by the two parts are combined for updating:
it＝σ(Wi.[ht-1，xt]+bi)，
wherein, biAnd bcIs the offset, tanh is an activation function, itIs the output of an output gate, WCIs a weight in the tanh layer, WiFor the weight of the activation function sigma,
is an old memory cell;
s3, updating the old memory cell; first multiplying the old memory cell by ftTo delete the unwanted information and then to do so
Adding to obtain a candidate value, wherein the formula is as follows:
wherein, CtIs a new memory cell, Ct-1The memory unit at the previous moment;
s4, determining the output of the model; firstly, an initial output is obtained through a sigmoid layer, and then C is processed by using tanhtScaling the value to be between-1 and 1, and multiplying the value by the output obtained by the sigmoid layer pair by pair to obtain the output of the LSTM model, wherein the formula of the step is as followsThe following:
Ot＝σ(Wo.[ht-1，xt]+bo)，
ht＝Ot*tanh(Ct)，
wherein O istIs the output of the output gate, boIs an offset, WoIs the weight of the activation function.
The mouse ring provided by the invention is matched with a mouse control device connected with target equipment for use, an inertia measurement module, a first data acquisition module and a first wireless communication module are arranged on a ring main body, the inertia measurement module generates inertia measurement data according to the movement of a finger of a user, the inertia measurement data is transmitted to the mouse control device connected with the target equipment through the first wireless communication module after being acquired by the first data acquisition module, and therefore, the mouse control device can generate a mouse instruction according to the inertia measurement data and control the target equipment according to the mouse instruction.
The infrared sensing device provided by the invention is matched with a mouse control device connected with target equipment for use, an inertia measurement module, a second data acquisition module and a second wireless communication module are arranged on a main body of the infrared sensing device, the infrared sensor generates inertia measurement data according to the movement of a finger of a user, the inertia measurement data is acquired by the second data acquisition module and then the horizontal position offset data is sent to the mouse control device connected with the target equipment through the second wireless communication module, so that the mouse control device can generate a mouse instruction according to the horizontal position offset data and control the target equipment according to the mouse instruction.
The invention has the beneficial effects that:
(1) according to the invention, the ring is adopted to realize the function of an electronic touch screen, so that a user can control the target equipment by the finger movement of wearing the mouse ring; because the mouse ring is realized based on the ring, the size of the mouse ring is small, and the mouse ring has the advantages of convenient carrying and beautiful use;
(2) the invention enables the user to realize the function of the electronic touch screen through the finger action, has higher transportability, can expand the existing non-touch screen into the touch screen by utilizing the device of the invention, and reduces the economic cost of the user.
Drawings
FIG. 1 is a schematic diagram of a mouse ring provided by the present invention;
FIG. 2 is a schematic view of an infrared sensing apparatus provided by the present invention;
FIG. 3 is a schematic diagram of a mouse control device provided by the present invention;
FIG. 4 is a schematic diagram of an electronic touch screen system according to the present invention;
FIG. 5 is a data processing flow diagram of an electronic touch screen system according to the present invention;
FIG. 6 is a schematic diagram of an electronic touch screen system according to the present invention;
FIG. 7 is a flowchart illustrating a generation process of a mouse manipulation instruction according to the present invention;
FIG. 8 is a flowchart illustrating a generation process of a mouse moving command according to the present invention.
Detailed Description
For a better understanding of the present disclosure, two examples are given herein.
The first embodiment is as follows: transplantable touch screen device based on IMU and infrared induction
The embodiment provides a portable touch screen device based on an IMU (inertial measurement Unit) and infrared induction, which comprises a mouse ring, an infrared induction device and a mouse control device. Fig. 1 shows a schematic diagram of a mouse ring provided by the invention.
The mouse ring comprises:
the system comprises a ring, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module;
the system comprises an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module, wherein the inertia measurement module, the miniature infrared emitter module, the first data acquisition module, the first wireless communication module, the first rechargeable battery and the first wireless charging module are sequentially connected and are all arranged on a ring;
the inertia measurement module is used for generating inertia measurement data according to the movement of the finger of the user after the ring body is worn on the finger of the user, and sending the inertia measurement data to the first data acquisition module; the miniature infrared transmitter module is used for transmitting infrared beams for receiving by the infrared sensing device.
The first data acquisition module is used for sending the inertia measurement data to a mouse control device connected with target equipment through the first wireless communication module so that the electronic touch screen control device generates a mouse instruction according to the inertia measurement data and controls the target equipment according to the mouse instruction;
the first wireless communication module is used for receiving inertial measurement data sent by a mouse ring and sending the inertial measurement data to the data processing module;
the first rechargeable battery is used for supplying power to each module in the mouse finger ring;
the first wireless charging module is used for charging the rechargeable battery in a wireless charging mode.
The infrared induction device comprises:
the infrared induction device comprises an infrared induction device main body, an infrared inductor, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module;
the infrared sensor, the second data acquisition module, the second wireless communication module, the second rechargeable battery and the second wireless charging module are sequentially connected and are all arranged on the infrared sensing device main body;
the infrared sensor is placed under the display screen, the length of the infrared sensor is larger than that of the display screen, and the placement position of the infrared sensor ensures that the infrared sensor can receive infrared beams emitted by the miniature infrared emitter module of a user when the user wears the mouse ring to click the display screen. When a user wears the mouse ring to generate horizontal offset on a screen, the beam emitted by the miniature infrared emitter module of the mouse ring also generates corresponding horizontal offset, the infrared sensor calculates the movement of the finger of the user according to the position of the received infrared beam to generate horizontal offset data, and sends the horizontal offset data to the second data acquisition module;
the second data acquisition module is used for sending the horizontal position offset data to a mouse control device connected with target equipment through the second wireless communication module so that the mouse control device generates a mouse instruction according to the horizontal position offset data and controls the target equipment according to the mouse instruction;
the second wireless communication module is used for receiving first inertia measurement data sent by a mouse ring and sending the first inertia measurement data to the data processing module;
the second rechargeable battery is used for supplying power to each module in the infrared inductor;
the second wireless charging module is used for charging the rechargeable battery in a wireless charging mode. Fig. 2 is a schematic diagram of an infrared sensing apparatus provided by the present invention.
The mouse control device comprises: the control device comprises a control device main body, and a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body;
the third wireless communication module and the physical communication interface are both connected with the data processing module;
the control device main body is fixedly connected with target equipment through the physical communication interface;
the third wireless communication module is used for receiving inertial measurement data sent by a mouse ring and sending the inertial measurement data to the data processing module; the third wireless communication module is also used for receiving horizontal position offset data sent by an infrared sensing device and sending the horizontal position offset data to the data processing module;
the data processing module is used for generating a mouse instruction according to the inertia measurement data and the horizontal position offset data, and sending the mouse instruction to the target equipment through the physical communication interface so as to control the target equipment;
and the physical communication interface sends a mouse instruction to the target equipment so as to control the target equipment. Fig. 3 is a schematic diagram of a mouse control device provided by the invention.
The mouse ring is worn on the index finger of the right hand of an operator, the inertial measurement module measures the angular velocity information of three axes and the acceleration information of the three axes, and the information is used for controlling the movement of a mouse cursor and click, double click and roller wheel click operations; when an operator moves the right hand to control the mouse cursor, an inertia measurement module on a mouse ring worn by the index finger of the right hand records three-axis acceleration information in the finger moving process, the obtained acceleration is used for carrying out 2-time integration on time to obtain displacement, and the integration is iterated according to the following formula:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2，
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2，
wherein v (t) and v (t-1) respectively represent target speeds at t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the inertia measurement module at t and (t-1), Δ t represents a time interval between t and (t-1), and s (t) and s (t-1) respectively represent target displacements at t and (t-1);
and (5) iterating the formula to obtain the vertical displacement within t time.
The infrared sensor is placed under the display screen, horizontal position offset data is generated according to the movement of a user finger, the finger is positioned at (x, y) at the moment t, and the position of the finger moves to (x + dx, y + dy) at the moment t + dt, so that the infrared sensor senses the shielding of the finger and directly obtains the horizontal position of the finger at the moment; the infrared sensor determines the position information of the right hand and is used for controlling the mouse cursor to move on the horizontal plane.
The control device main body extracts multi-mode features of horizontal shielding information captured by the infrared sensor and information measured by the inertia measurement module on the mouse ring, then inputs the multi-mode features into the LSTM model to obtain a recognition probability result of pressing and hitting by the right-hand index finger, if the probability value is larger than a certain threshold value, a control signal of the mouse is sent to the data processing module, and if the probability is smaller than the certain threshold value, recognition of the action is given up.
The control device main body performs multi-mode feature extraction on horizontal position offset data captured by the infrared sensor and inertial measurement module measurement information on the mouse ring, and feature extraction methods for the inertial measurement module measurement information comprise normalization and amplitude absolute value mean value extraction methods.
The control device main body extracts multi-mode features from horizontal position offset data captured by the infrared sensor and information measured by the inertial measurement module on the mouse ring, inputs the multi-mode features into the LSTM model, and the LSTM model automatically learns the time sequence features of the input multi-mode features and outputs the probability that the features belong to each classification, so that the classification and identification categories to which the input multi-mode features most possibly belong are obtained and serve as the identification probability result of the right-hand index finger pressing.
The control device main body carries out multi-mode feature extraction on horizontal position offset data captured by the infrared sensor and inertial measurement module measurement information on the mouse ring, the feature extraction method for the active section of the inertial measurement module comprises a normalization method and an amplitude absolute value mean value extraction method, and the calculation formula of the normalization extraction method is as follows:
where | xiL represents the absolute value of the frame data in the data of the inertia measurement module, min represents the numerical value with the minimum absolute value in the movable section of the inertia measurement module, and max represents the numerical value with the maximum absolute value in the movable section;
the calculation formula of the amplitude absolute value mean value is as follows:
wherein MAV represents the extracted absolute mean feature, N represents the length of the sliding window, and xiThe amplitude of the frame signal is represented, and the mean value of the absolute values of the amplitudes is the mean value of the absolute values of all data in the sliding window.
The control device main body extracts multi-mode features of horizontal position offset data captured by the infrared sensor and information measured by the inertial measurement module on the mouse ring, and then inputs the multi-mode features into the LSTM model, wherein the LSTM model comprises the following specific steps:
s1, the forgetting gate layer controls the information passing through the memory cell by activating the function, which generates a value from 0 to 1 according to the output of the previous time and the input of the current time, to control whether the information learned at the previous time passes through or partially passes through the memory cell, the formula of which is as follows:
ft＝σ(Wf·[ht-1，xt]+bf)，
where σ is the activation function, ht-iIs the output of the previous time, xtIs the current input, bfIs an offset, ftIs the output of a forgetting gate, WfTaking values for the weight;
s2, generating new information needing to be updated; the step comprises two parts, wherein the first part is that an input gate layer determines an updating value through a sigmoid activation function, the second part is that a tanh layer is used for generating a new candidate value which is added into a memory unit as a candidate value generated by a current layer, and the values generated by the two parts are combined for updating:
it＝σ(Wi.[ht-1，xt]+bi)，
wherein, biAnd bCIs the offset, tanh is an activation function, itIs the output of an output gate, WcIn the tan h layerWeight of (W)iFor the weight of the activation function sigma,
is an old memory cell;
s3, updating the old memory cell; first multiplying the old memory cell by ftTo delete the unwanted information and then to do so
Adding to obtain a candidate value, wherein the formula is as follows:
wherein, CtIs a new memory cell, Ct-1The memory unit at the previous moment;
s4, determining the output of the model; firstly, an initial output is obtained through a sigmoid layer, and then C is processed by using tanhtThe value is scaled to be between-1 and 1, and then the value is multiplied by the output obtained by the sigmoid layer in a pair-by-pair mode, so that the output of the LSTM model is obtained, and the formula of the step is as follows:
Ot＝σ(Wo.[ht-1，xt]+bo)，
ht＝Ot*tanh(Ct)，
wherein O istIs the output of the output gate, boIs an offset, WoIs the weight of the activation function.
Fig. 4 is a schematic diagram of an electronic touch screen system according to the present invention. In fig. 4, a module 1 is a mouse ring, a module 2 is an infrared sensing device, and a module 3 is a mouse control device.
Example two: logic frame diagram of electronic touch screen system
The application provides a logical frame diagram of an electronic touch screen system, and the frame mainly comprises three layers: input layer, processing layer, output layer. These three sections will be described separately below. Fig. 5 shows a data processing flow chart of an electronic touch screen system provided by the present invention. Fig. 6 is a schematic structural diagram of an electronic touch screen system according to the present invention.
The first part, the input layer, mainly obtains the multi-modal information of the user, mainly includes the horizontal position offset information that the infrared inductor obtains, the inertia measurement module, also known as IMU motion sensor, the kinematic information of the index finger of the right hand that obtains, wherein the horizontal position offset information that the infrared inductor obtains is mainly used for controlling the mouse cursor to move about on the horizontal plane, IMU motion sensor measures and obtains the movement information of the goal, is used for controlling movement of the mouse cursor and clicking, double click and pressing operation such as the gyro wheel.
The second part, the processing layer, mainly carries out the integrated processing to received infrared inductor information and IMU motion sensor information, and this layer mainly comprises three major parts: the mouse cursor moving part is controlled, and the mouse ring clicking, double clicking, roller and resetting parts are controlled;
the mouse cursor moving control part mainly comprises an infrared sensor and an IMU motion sensor, wherein the IMU motion sensor is an inertial measurement unit and can measure three-axis acceleration and three-axis angular velocity (x axis, y axis and z axis). The main function of the part is to control the movement of a mouse cursor, and the realization principle is to utilize the information captured by the infrared sensor and the IMU motion sensor, respectively introduce the principle of the infrared sensor and the IMU motion sensor, and finally introduce the fusion of the infrared sensor and the IMU motion sensor.
In the mouse control system, the infrared sensor mainly determines the position information of the right hand so as to convert the position information into horizontal movement of a mouse cursor. For the calculation of the horizontal movement displacement, when a user wears the mouse ring to generate horizontal offset on a screen, the beam emitted by the miniature infrared emitter module of the mouse ring also generates corresponding horizontal offset, and the infrared sensor calculates the movement of the finger of the user according to the position of the received infrared beam to generate horizontal position offset data. When the position of the finger is (x, y) at the time t and the position of the finger moves to (x + dx, y + dy) at the time t + dt, the shielding of the finger is sensed by the infrared sensor, and the current horizontal position of the finger is directly obtained.
In this mouse control system, IMU motion sensor is located the mouse ring that the right hand forefinger wore, and IMU sensor mainly measures triaxial angular velocity information and triaxial acceleration information, and when we moved the right hand and want to control mouse cursor, the IMU sensor on the mouse ring that the right hand forefinger wore can note the triaxial acceleration information of removal in-process, carries out 2 times integrals to the time with the acceleration that obtains and can obtain the displacement, and the following formula of integral mode is iterated:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2,
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2,
wherein v (t) and v (t-1) respectively represent target speeds at t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the IMU sensors at t and (t-1), Δ t represents a time interval between t and (t-1), and s (t) and s (t-1) respectively represent target displacements at t and (t-1);
and (5) iterating the formula to obtain the vertical displacement within the time t.
Fig. 7 shows a flowchart for generating a mouse manipulation instruction according to the present invention.
Thirdly, controlling a mouse ring single click part, a mouse ring double click part, a mouse ring idler wheel part and a mouse ring reset part; the part is mainly realized by a mouse ring module worn at the index finger of the right hand and an infrared sensor positioned below the screen. The partial function realization needs to be assisted by a deep neural network. The LSTM is a time-cycle neural network, can solve the long-term dependence problem of RNN, and is very suitable for processing the time series problem. The quick short-lived left slip of right hand forefinger corresponds the mouse and hits the function left, and the quick short-lived right slip of right hand forefinger corresponds the mouse and hits the function right, and the right hand forefinger double click corresponds the mouse and hits the function double, and the quick short-lived cunning of right hand forefinger corresponds the mouse gyro wheel and slides down the function, and the quick short-lived last slip of right hand forefinger corresponds the mouse gyro wheel and slides up the function, holds the fist in the right side and corresponds reset function (the mouse cursor returns to the screen centre. The control device main body carries out multi-mode feature extraction on horizontal shielding information captured by the infrared sensor and IMU motion sensor information on the mouse ring, then the multi-mode feature extraction is carried out on the horizontal shielding information and the IMU motion sensor information, then the multi-mode feature extraction is input into the LSTM model, a recognition probability result of pressing and hitting of the index finger of the right hand can be obtained, if the probability value is larger than a certain threshold value, a control signal of the mouse is sent to the data processing module, and if the probability is smaller than the certain threshold value, recognition of the action. The feature extraction method for the IMU motion sensor comprises MAV, RMS and the like, wherein the MAV is the mean absolute value of amplitude, and the RMS is the root mean square. FIG. 8 is a flowchart illustrating a generation process of a mouse moving command according to the present invention.
The LSTM (Long Short-Term Memory) can process Long-time sequence information and mainly comprises an input gate, a forgetting gate and an output gate in the internal structure. The working steps of the LSTM model are as follows:
s1, the forgetting gate layer controls the information passing through the memory cell by activating the function, which generates a value from 0 to 1 according to the output of the previous time and the input of the current time, to control whether the information learned at the previous time passes through or partially passes through the memory cell, the formula of which is as follows:
ft＝σ(Wf.[ht-1，xt]+bf)，
where σ is the activation function, ht-iIs the output of the previous time, xtIs the current input, bfIs an offset, ftIs the output of a forgetting gate, WfTaking values for the weight;
s2, generating new information needing to be updated; the step comprises two parts, wherein the first part is that an input gate layer determines an updating value through a sigmoid activation function, the second part is that a tanh layer is used for generating a new candidate value which is added into a memory unit as a candidate value generated by a current layer, and the values generated by the two parts are combined for updating:
it＝σ(Wi.[ht-1，xt]+bi)，
wherein, biAnd bCIs the offset, tanh is an activation function, itBeing output gatesOutput, WcIs a weight in the tanh layer, WiFor the weight of the activation function sigma,
is an old memory cell;
s3, updating the old memory cell; first multiplying the old memory cell by ftTo delete the unwanted information and then to do so
Adding to obtain a candidate value, wherein the formula is as follows:
wherein, CtIs a new memory cell, Ct-1The memory unit at the previous moment;
s4, determining the output of the model; firstly, an initial output is obtained through a sigmoid layer, and then C is processed by using tanhtThe value is scaled to be between-1 and 1, and then the value is multiplied by the output obtained by the sigmoid layer in a pair-by-pair mode, so that the output of the LSTM model is obtained, and the formula of the step is as follows:
Ot＝σ(Wo.[ht-1，xt]+bo)，
ht＝Ot*tanh(Ct)，
wherein O istIs the output of the output gate, boIs an offset, WoIs the weight of the activation function.
And the third part, an output layer, mainly outputs and displays the control instruction obtained by the processing layer, and the layer mainly comprises a data processing module. The quick short-lived left slip of right hand forefinger corresponds the mouse and hits the function left, and the quick short-lived right slip of right hand forefinger corresponds the mouse and hits the function right, and the right hand forefinger double click corresponds the mouse and hits the function double, and the quick short-lived cunning of right hand forefinger corresponds the mouse gyro wheel and slides down the function, and the quick short-lived last slip of right hand forefinger corresponds the mouse gyro wheel and slides up the function, holds the fist in the right side and corresponds reset function (the mouse cursor returns to the screen centre.
It should be noted that the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims (9)
1. The utility model provides a portable touch screen device based on IMU and infrared induction which characterized in that: comprises a mouse ring, an infrared induction device and a mouse control device.
2. An IMU and infrared sensing based portable touch screen device according to claim 1,
the mouse ring comprises:
the intelligent finger ring comprises a finger ring body, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module;
the ring comprises a ring body, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery, a first wireless charging module, a first wireless data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module, wherein the inertia measurement module is connected with the miniature infrared emitter module, the miniature infrared emitter module is connected with the first data acquisition module, the first wireless communication module is connected with the first rechargeable battery, and the inertia measurement module, the miniature infrared emitter module, the first data acquisition module, the first wireless communication module, the first rechargeable battery and the first wireless charging module are all arranged on;
the inertia measurement module is used for generating inertia measurement data according to the movement of the finger of the user after the ring body is worn on the finger of the user, and sending the inertia measurement data to the first data acquisition module; the miniature infrared transmitter module is used for transmitting infrared beams for the infrared sensing device to receive;
the first data acquisition module is used for sending the inertia measurement data to a mouse control device connected with target equipment through the first wireless communication module so that the electronic touch screen control device generates a mouse instruction according to the inertia measurement data and controls the target equipment according to the mouse instruction;
the first wireless communication module is used for receiving the inertia measurement data sent by the first data acquisition module and sending the inertia measurement data to the data processing module;
the first rechargeable battery is used for supplying power to each module in the mouse finger ring;
the first wireless charging module is used for charging the first rechargeable battery in a wireless charging mode.
3. The IMU and infrared induction based implantable touch screen device according to claim 2, wherein the mouse ring is worn on the index finger of the operator's right hand, and the inertial measurement module measures three-axis angular velocity information and three-axis acceleration information for controlling the movement of the mouse cursor and click, double click and roller press operations; when an operator moves the right hand and wants to control the mouse cursor, the inertia measurement module on the mouse ring worn by the index finger of the right hand records the three-axis acceleration information in the finger moving process, the obtained acceleration is used for carrying out 2 times of integration on time to obtain displacement, and the integration is iterated according to the following two integration formulas:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2， (1)
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2， (2)
wherein v (t) and v (t-1) respectively represent target speeds at t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the inertia measurement module at t and (t-1), Δ t represents a time interval between t and (t-1), and s (t) and s (t-1) respectively represent target displacements at t and (t-1);
and (3) iterating the two integral formulas of the formula (1) and the formula (2) to obtain the vertical displacement within the time t.
4. An IMU and infrared sensing based portable touch screen device according to claim 1,
the infrared induction device comprises:
the infrared induction device comprises an infrared induction device main body, an infrared inductor, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module;
the infrared sensor, the second data acquisition module, the second wireless communication module, the second rechargeable battery and the second wireless charging module are all arranged on the infrared sensing device main body;
the infrared sensor is placed under the display screen, the length of the infrared sensor is greater than that of the display screen, and the infrared sensor is placed at a position where the infrared sensor can receive infrared beams emitted by the miniature infrared emitter module of a user when the user wears the mouse ring to click the display screen; when a user wears the mouse ring to generate horizontal offset on the display screen, the beam emitted by the miniature infrared emitter module of the mouse ring also generates corresponding horizontal offset, the infrared sensor calculates horizontal offset data generated by the movement of the finger of the user according to the position of the received infrared beam, and sends the horizontal offset data to the second data acquisition module;
the second data acquisition module is used for sending the horizontal position offset data to a mouse control device connected with target equipment through the second wireless communication module so that the mouse control device generates a mouse instruction according to the horizontal position offset data and controls the target equipment according to the mouse instruction;
the second wireless communication module is used for receiving horizontal position offset data sent by the second data acquisition module and sending the horizontal position offset data to the data processing module;
the second rechargeable battery is used for supplying power to each module in the infrared sensor device;
the second wireless charging module is used for charging the second rechargeable battery in a wireless charging mode.
5. The IMU and IR sensing based portable touch screen device according to claim 4, wherein the IR sensor is placed just below the display screen to generate horizontal position offset data according to the movement of the user's finger, the finger is at (x, y) at time t, and when it moves to (x + dx, y + dy) at time t + dt, the occlusion of the finger is sensed by the IR sensor to directly obtain its current horizontal position; the infrared sensor determines the position information of the right hand and is used for controlling the mouse cursor to move on the horizontal plane.
6. An IMU and infrared sensing based portable touch screen device according to claim 1,
the mouse control device comprises: the control device comprises a control device main body, and a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body;
the third wireless communication module and the physical communication interface are both connected with the data processing module;
the control device main body is fixedly connected with target equipment through the physical communication interface;
the third wireless communication module is used for receiving inertial measurement data sent by a mouse ring and sending the inertial measurement data to the data processing module; the third wireless communication module is also used for receiving horizontal position offset data sent by an infrared sensing device and sending the horizontal position offset data to the data processing module;
the data processing module is used for generating a mouse instruction according to the inertia measurement data and the horizontal position offset data, and sending the mouse instruction to the target equipment through the physical communication interface so as to control the target equipment;
and the physical communication interface sends a mouse instruction to the target equipment so as to control the target equipment.
7. The IMU and infrared induction based portable touch screen apparatus of claim 6,
the control device main body extracts multi-mode features of horizontal position offset data captured by the infrared sensor and information measured by the inertia measurement module on the mouse ring, inputs the multi-mode features into the LSTM model to obtain a recognition probability result of the right-hand index finger pressing, sends a control signal of the mouse to the data processing module if the probability value is larger than a certain threshold value, and abandons the recognition of the action if the probability is smaller than the certain threshold value.
8. The IMU and IR sensing based portable touch screen device according to claim 7, wherein the control device body performs multi-modal feature extraction of horizontal position offset data captured by the IR sensor and inertial measurement module measurement information on the mouse ring, and the feature extraction methods for the inertial measurement module measurement information include normalization and amplitude absolute value mean extraction methods.
9. The IMU and IR sensing based portable touch screen device according to claim 7, wherein the control device body performs multi-modal feature extraction on the horizontal position offset data captured by the IR sensor and the inertial measurement module measurement information on the mouse ring, and then inputs the extracted data into the LSTM model, which automatically learns the time-series features of the input multi-modal features and outputs the probability that the features belong to each class, thereby obtaining the class recognition class to which the input multi-modal features are most likely to belong as the recognition probability result of right-handed index finger pressing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010930016.8A CN112083832B (en) | 2020-09-07 | 2020-09-07 | Implantable touch screen device based on IMU and infrared induction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010930016.8A CN112083832B (en) | 2020-09-07 | 2020-09-07 | Implantable touch screen device based on IMU and infrared induction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112083832A true CN112083832A (en) | 2020-12-15 |
| CN112083832B CN112083832B (en) | 2023-06-30 |
Family
ID=73732472
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010930016.8A Active CN112083832B (en) | 2020-09-07 | 2020-09-07 | Implantable touch screen device based on IMU and infrared induction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112083832B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112929734A (en) * | 2021-02-05 | 2021-06-08 | 维沃移动通信有限公司 | Screen projection method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203299759U (en) * | 2013-05-04 | 2013-11-20 | 罗美华 | Automatic charging energy-saving wireless mouse |
| CN107305420A (en) * | 2016-04-24 | 2017-10-31 | 吴金术 | A kind of computer mouse automatic sensing energy saver |
| CN110928432A (en) * | 2019-10-24 | 2020-03-27 | 中国人民解放军军事科学院国防科技创新研究院 | Ring mouse, mouse control device and mouse control system |
-
2020
- 2020-09-07 CN CN202010930016.8A patent/CN112083832B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203299759U (en) * | 2013-05-04 | 2013-11-20 | 罗美华 | Automatic charging energy-saving wireless mouse |
| CN107305420A (en) * | 2016-04-24 | 2017-10-31 | 吴金术 | A kind of computer mouse automatic sensing energy saver |
| CN110928432A (en) * | 2019-10-24 | 2020-03-27 | 中国人民解放军军事科学院国防科技创新研究院 | Ring mouse, mouse control device and mouse control system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112929734A (en) * | 2021-02-05 | 2021-06-08 | 维沃移动通信有限公司 | Screen projection method and device and electronic equipment |
| CN112929734B (en) * | 2021-02-05 | 2023-09-05 | 维沃移动通信有限公司 | Screen projection method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112083832B (en) | 2023-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110765967B (en) | Action recognition method based on artificial intelligence and related device | |
| CN112789577B (en) | Neuromuscular text input, writing and drawing in augmented reality systems | |
| CN102915111B (en) | A kind of wrist gesture control system and method | |
| CN202584010U (en) | Wrist-mounting gesture control system | |
| CN110928432B (en) | Finger ring mouse, mouse control device and mouse control system | |
| CN102789312B (en) | A kind of user interactive system and method | |
| CN103677289A (en) | Intelligent interactive glove and interactive method | |
| CN111722713A (en) | Multi-mode fused gesture keyboard input method, device, system and storage medium | |
| Jingqiu et al. | An ARM-based embedded gesture recognition system using a data glove | |
| Xiong et al. | A novel HCI based on EMG and IMU | |
| CN103440033A (en) | Method and device for achieving man-machine interaction based on bare hand and monocular camera | |
| CN105919599B (en) | Finger movement detection identifying system and method based on Magnetic Sensor | |
| WO2023178984A1 (en) | Methods and systems for multimodal hand state prediction | |
| Wang et al. | Wearable sensors-based hand gesture recognition for human–robot collaboration in construction | |
| CN113849068A (en) | Gesture multi-mode information fusion understanding and interacting method and system | |
| CN112083832A (en) | Transplantable touch screen device based on IMU and infrared induction | |
| CN106716304A (en) | Control unit and method of interacting with a graphical user interface | |
| Costagliola et al. | Gesture‐Based Computing | |
| Bulugu | Real-time Complex Hand Gestures Recognition Based on Multi-Dimensional Features. | |
| Khan et al. | Gesture recognition using Open-CV | |
| Morajkar et al. | Hand gesture and voice-controlled mouse for physically challenged using computer vision | |
| Pranathi et al. | Tri-Axis motion detection using MEMS for unwired mouse navigation system in the future generation machines | |
| CN103440034A (en) | Method and device for achieving man-machine interaction based on bare hands and monocular camera | |
| Feng et al. | Design and implementation of gesture recognition system based on flex sensors | |
| Elmagrouni et al. | A deep learning framework for hand gesture recognition and multimodal interface control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |