CN112083832B - Implantable touch screen device based on IMU and infrared induction - Google Patents
Implantable touch screen device based on IMU and infrared induction Download PDFInfo
- Publication number
- CN112083832B CN112083832B CN202010930016.8A CN202010930016A CN112083832B CN 112083832 B CN112083832 B CN 112083832B CN 202010930016 A CN202010930016 A CN 202010930016A CN 112083832 B CN112083832 B CN 112083832B
- Authority
- CN
- China
- Prior art keywords
- module
- mouse
- infrared
- data
- wireless communication
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F3/0383—Signal control means within the pointing device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0412—Digitisers structurally integrated in a display
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
- Y02B40/00—Technologies aiming at improving the efficiency of home appliances, e.g. induction cooking or efficient technologies for refrigerators, freezers or dish washers
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Position Input By Displaying (AREA)
Abstract
The invention provides an IMU and infrared induction-based portable touch screen device, which comprises a mouse finger ring, an infrared induction device and a mouse control device. The mouse finger ring comprises a finger ring, an inertia measurement module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module; the infrared sensing device comprises an infrared sensing device main body, an infrared sensor, a micro infrared emission module, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module; the mouse control device includes: the control device comprises a control device main body, and a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body. The electronic touch screen has the advantages of convenience in carrying and attractive use, and has high portability and reduced economic cost for users.
Description
Technical Field
The invention relates to the field of electronic computer input equipment, in particular to a portable touch screen device with multiple induction signals.
Background
Computer displays, also commonly referred to as computer monitors or computer screens, are a display means that display certain electronic document information onto the screen via a specific transmission device and then back to the human eye. At present, computer displays are divided into two major types, namely an LED display screen and a liquid crystal display screen, most of the displays have no input function, a mouse and a keyboard are required to be controlled, and the application of the traditional LED display screen and liquid crystal display screen is limited in the scenes that the keyboard and the mouse are inconvenient to be arranged outside outdoor, market, bank and the like. The electronic touch screen changes the interaction mode of the original mouse and keyboard, is very convenient and fast, but has higher cost, is the hardware of the computer, and has no portability. In order to solve the above problems, a more convenient, quick and efficient man-machine interaction device is needed.
Disclosure of Invention
The invention aims to solve the problems of low efficiency and more equipment needed in the existing man-machine interaction means.
The invention discloses an IMU and infrared induction-based portable touch screen device, which comprises a mouse finger ring, an infrared induction device and a mouse control device.
The mouse finger ring comprises:
the device comprises a ring main body, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module;
the inertial measurement module is connected with the miniature infrared emitter module, the miniature infrared emitter module is connected with the first data acquisition module, the first data acquisition module is connected with the first wireless communication module, the first wireless communication module is connected with the first rechargeable battery, the first rechargeable battery is connected with the first wireless charging module, and the inertial measurement module, the miniature infrared emitter module, the first data acquisition module, the first wireless communication module, the first rechargeable battery and the first wireless charging module are all arranged on the ring main body;
the inertial measurement module is used for generating inertial measurement data according to the movement of the finger of the user after the finger ring main body is worn on the finger of the user, and sending the inertial measurement data to the first data acquisition module; the miniature infrared emitter module is used for emitting infrared beams for being received by the infrared sensing device;
the first data acquisition module is used for transmitting the inertia measurement data to a mouse control device connected with target equipment through the first wireless communication module, so that the electronic touch screen control device generates a mouse instruction according to the inertia measurement data and controls the target equipment according to the mouse instruction;
the first wireless communication module is used for receiving the inertial measurement data sent by the first data acquisition module and sending the inertial measurement data to the data processing module;
the first rechargeable battery is used for supplying power to each module in the mouse finger ring;
the first wireless charging module is used for charging the first rechargeable battery in a wireless charging mode.
The portable touch screen device based on the IMU and the infrared induction is characterized in that the mouse finger ring is worn on the right index finger of an operator, the inertial measurement module measures three-axis angular velocity information and three-axis acceleration information, and the information is used for controlling the movement of a mouse cursor and the clicking, double clicking and roller clicking operations; when an operator moves a right hand to control a mouse cursor, an inertial measurement module on a mouse finger ring worn by a right index finger records triaxial acceleration information in the finger movement process, the obtained acceleration is used for integrating time for 2 times to obtain displacement, and the integration is iterated according to the following two integration formulas:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2， (1)
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2， (2)
wherein v (t) and v (t-1) respectively represent target speeds at times t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the inertial measurement module at times t and (t-1), Δt represents time intervals at times t and (t-1), and s (t) and s (t-1) respectively represent target displacements at times t and (t-1);
and (3) iterating the two integral formulas of the formula (1) and the formula (2) to obtain the vertical displacement in the t time.
The infrared sensing device includes:
the device comprises an infrared sensing device main body, an infrared sensor, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module;
the infrared sensor is connected with the second data acquisition module, the second data acquisition module is connected with the second wireless communication module, the second wireless communication module is connected with the second rechargeable battery, the second rechargeable battery is connected with the second wireless charging module, and the infrared sensor, the second data acquisition module, the second wireless communication module, the second rechargeable battery and the second wireless charging module are all arranged on the infrared sensing device main body;
the infrared sensor is arranged under the display screen, the length of the infrared sensor is longer than that of the display screen, and the position of the infrared sensor is required to ensure that the infrared sensor can receive infrared wave beams emitted by a miniature infrared emitter module of a user when the user clicks the display screen by wearing a mouse finger ring; when the user wears the mouse finger ring to generate horizontal offset on the display screen, the beam emitted by the miniature infrared emitter module of the mouse finger ring also generates corresponding horizontal offset, and the infrared sensor calculates horizontal position offset data generated by the movement of the finger of the user according to the position of the infrared beam received by the infrared sensor and sends the horizontal position offset data to the second data acquisition module;
the second data acquisition module is used for transmitting the horizontal position offset data to a mouse control device connected with target equipment through the second wireless communication module, so that the mouse control device generates a mouse instruction according to the horizontal position offset data and controls the target equipment according to the mouse instruction;
the second wireless communication module is used for receiving the horizontal position offset data sent by the second data acquisition module and sending the horizontal position offset data to the data processing module;
the second rechargeable battery is used for supplying power to each module in the infrared sensor device;
the second wireless charging module is used for charging the second rechargeable battery in a wireless charging mode.
The infrared sensor is arranged under the display screen, horizontal position offset data are generated according to the movement of the fingers of a user, the positions of the fingers are (x, y) at the moment t, and the positions of the fingers move to (x+dx, y+dy) at the moment t+dt, so that the shielding of the fingers is sensed through the infrared sensor, and the horizontal position of the fingers at the moment is directly obtained; the infrared sensor determines the position information of the right hand and is used for controlling the mouse cursor to move on the horizontal plane.
The mouse control device comprises: a control device main body, a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body;
the third wireless communication module and the physical communication interface are both connected with the data processing module;
the control device main body is fixedly connected with the target equipment through the physical communication interface;
the third wireless communication module is used for receiving inertial measurement data sent by the mouse finger ring and sending the inertial measurement data to the data processing module; the third wireless communication module is further used for receiving horizontal position offset data sent by the infrared sensing device and sending the horizontal position offset data to the data processing module;
the data processing module is used for generating a mouse instruction according to the inertia measurement data and the horizontal position offset data, and sending the mouse instruction to the target equipment through the physical communication interface so as to control the target equipment;
and the physical communication interface instructs a mouse to the target equipment so as to control the target equipment.
The control device main body extracts multi-mode characteristics of horizontal position offset data captured by the infrared sensor and measurement information of an inertia measurement module on a mouse finger ring, then inputs the multi-mode characteristics into an LSTM model to obtain a recognition probability result of right index finger pressing, if the probability value is larger than a certain threshold value, a control signal of the mouse is sent to the data processing module, and if the probability value is smaller than the certain threshold value, recognition of the action is abandoned.
The control device main body is used for carrying out multi-mode feature extraction on the horizontal position offset data captured by the infrared sensor and the measurement information of the inertia measurement module on the mouse finger ring, and the feature extraction method of the measurement information of the inertia measurement module comprises normalization and amplitude absolute value average value extraction methods.
The control device main body extracts the multi-modal characteristics of the horizontal position offset data captured by the infrared sensor and the measurement information of the inertia measurement module on the mouse finger ring, then inputs the multi-modal characteristics into the LSTM model, the LSTM model automatically learns the time sequence characteristics of the input multi-modal characteristics, and outputs the probability that the characteristics belong to each category, so that the most likely category recognition category of the input multi-modal characteristics is obtained and is used as the recognition probability result of right index finger pressing.
The control device main body is used for carrying out multi-mode feature extraction on horizontal position offset data captured by an infrared sensor and measurement information of an inertial measurement module on a mouse finger ring, and the feature extraction method of a movable section of the inertial measurement module comprises a normalization and amplitude absolute value average value extraction method, wherein the calculation formula of the normalization extraction method is as follows:
wherein |x i The I represents the absolute value of the frame data in the inertial measurement module data, the min represents the value with the minimum absolute value in the active section of the inertial measurement module, and the max represents the value with the maximum absolute value in the active section;
the calculation formula of the absolute value mean value of the amplitude is as follows:
wherein MAV represents the extracted absolute value average feature, N represents the length of the sliding window, x i The amplitude of the frame signal is represented, and the average value of the absolute values of the amplitude is obtained by taking the average value of the absolute values of all data in a sliding window.
The control device main body extracts multi-mode characteristics of horizontal position offset data captured by an infrared sensor and measurement information of an inertia measurement module on a mouse finger ring, and then inputs the multi-mode characteristics into an LSTM model, wherein the LSTM model comprises the following specific steps of:
s1, the forgetting gate layer controls the information passing through the memory unit through an activation function, generates a value of 0 to 1 according to the output of the last moment and the input of the current moment, and controls whether the information learned at the last moment passes through or partially passes through the memory unit, wherein the formula is as follows:
f t ＝σ(W f ·[h t-1 ，x t ]+b f )，
where σ is the activation function, h t-1 Is the output of the last moment, x t Is the current input, b f Is the offset, f t Is the output of the forgetting gate, W f The weight is taken as a value;
s2, generating new information to be updated; this step comprises two parts, the first part is that an input gate layer decides an updated value through a sigmoid activation function, the second part is that a tanh layer is used for generating a new candidate value, the new candidate value is used as a candidate value generated by a current layer and is added into a memory unit, and the values generated by the two parts are combined for updating:
i t ＝σ(W i .[h t-1 ，x t ]+b i )，
wherein b i And b c Is the offset, tanh is an activation function, i t Is the transfusionOutput from the gate, W C For weights in the tanh layer, W i For the weight of the activation function sigma,
is an old memory cell;
s3, updating the old memory unit; first multiply the old memory cell by f t To delete unnecessary information and then to re-associate with
The candidate values are obtained by addition, and the formula is as follows:
wherein C is t Is a new memory cell, C t-1 A memory unit at the previous time;
s4, determining the output of the model; first get an initial output through the sigmoid layer and then use tanh to get C t The values scale to between-1 and 1, and are multiplied by the output obtained by the sigmoid layer pair by pair, so that the output of the LSTM model is obtained, and the formula of the step is as follows:
O t ＝σ(W o .[h t-1 ，x t ]+b o )，
h t ＝O t *tanh(C t )，
wherein O is t Is the output of the output gate, b o Is the offset, W o Is the weight of the activation function.
The mouse finger ring provided by the invention is matched with the mouse control device connected with the target device, and the inertial measurement module, the first data acquisition module and the first wireless communication module are configured on the finger ring main body, the inertial measurement module generates inertial measurement data according to the movement of the finger of a user, and the first wireless communication module transmits the inertial measurement data to the mouse control device connected with the target device after the inertial measurement data are acquired by the first data acquisition module, so that the mouse control device can generate a mouse instruction according to the inertial measurement data and control the target device according to the mouse instruction.
The infrared sensing device is matched with the mouse control device connected with the target equipment, the inertial measurement module, the second data acquisition module and the second wireless communication module are configured on the infrared sensing device main body, the infrared sensor generates inertial measurement data according to the movement of the finger of the user, the second data acquisition module acquires the inertial measurement data, and the second wireless communication module transmits the horizontal position offset data to the mouse control device connected with the target equipment, so that the mouse control device can generate a mouse instruction according to the horizontal position offset data and control the target equipment according to the mouse instruction.
The beneficial effects of the invention are as follows:
(1) The invention adopts the finger ring to realize the function of the electronic touch screen, so that a user can realize the control of target equipment by the movement of the finger wearing the mouse finger ring; the mouse finger ring is realized based on the finger ring, and the mouse finger ring has the advantages of convenience in carrying and attractive use;
(2) The device can enable a user to realize the function of the electronic touch screen through finger actions, has higher portability, and can expand the traditional non-touch screen into a touch screen, thereby reducing the economic cost of the user.
Drawings
FIG. 1 is a schematic view of a mouse finger ring provided by the present invention;
fig. 2 shows a schematic diagram of an infrared sensing device according to the present invention;
FIG. 3 is a schematic diagram of a mouse control device according to the present invention;
fig. 4 is a schematic diagram of an electronic touch screen system according to the present invention;
FIG. 5 shows a data processing flow chart of an electronic touch screen system provided by the invention;
fig. 6 shows a schematic structural diagram of an electronic touch screen system provided by the present invention;
FIG. 7 is a flowchart showing the generation of a mouse manipulation instruction according to the present invention;
fig. 8 shows a flowchart of generating a mouse movement instruction according to the present invention.
Detailed Description
For a better understanding of the present disclosure, two embodiments are presented herein.
Embodiment one: implantable touch screen device based on IMU and infrared induction
The embodiment provides a portable touch screen device based on an IMU and infrared induction, which comprises a mouse finger ring, an infrared induction device and a mouse control device. Fig. 1 shows a schematic diagram of a mouse finger ring according to the present invention.
The mouse finger ring comprises:
the device comprises a finger ring, an inertia measurement module, a micro infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module;
the device comprises an inertial measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module, wherein the first rechargeable battery and the first wireless charging module are sequentially connected and are arranged on the finger ring;
the inertial measurement module is used for generating inertial measurement data according to the movement of the finger of the user after the finger ring main body is worn on the finger of the user, and sending the inertial measurement data to the first data acquisition module; the miniature infrared emitter module is used for emitting infrared wave beams for being received by the infrared sensing device.
The first data acquisition module is used for transmitting the inertia measurement data to a mouse control device connected with target equipment through the first wireless communication module, so that the electronic touch screen control device generates a mouse instruction according to the inertia measurement data and controls the target equipment according to the mouse instruction;
the first wireless communication module is used for receiving inertial measurement data sent by the mouse finger ring and sending the inertial measurement data to the data processing module;
the first rechargeable battery is used for supplying power to each module in the mouse finger ring;
the first wireless charging module is used for charging the rechargeable battery in a wireless charging mode.
The infrared sensing device comprises:
the device comprises an infrared sensing device main body, an infrared sensor, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module;
the infrared sensor, the second data acquisition module, the second wireless communication module, the second rechargeable battery and the second wireless charging module are sequentially connected and are arranged on the infrared sensing device main body;
the infrared sensor is placed under the display screen, the length of the infrared sensor is larger than that of the display screen, and the placement position of the infrared sensor is used for ensuring that the infrared sensor can receive infrared wave beams sent by the miniature infrared emitter module of the user when the user clicks the display screen by wearing the mouse finger ring. When a user wears a mouse finger ring to generate horizontal offset on a screen, corresponding horizontal offset is generated by a beam emitted by a miniature infrared emitter module of the mouse finger ring, and an infrared sensor calculates the movement of the finger of the user to generate horizontal position offset data according to the position of the infrared beam received by the infrared sensor and sends the horizontal position offset data to a second data acquisition module;
the second data acquisition module is used for transmitting the horizontal position offset data to a mouse control device connected with target equipment through the second wireless communication module, so that the mouse control device generates a mouse instruction according to the horizontal position offset data and controls the target equipment according to the mouse instruction;
the second wireless communication module is used for receiving first inertial measurement data sent by the mouse finger ring and sending the first inertial measurement data to the data processing module;
the second rechargeable battery is used for supplying power to each module in the infrared inductor;
the second wireless charging module is used for charging the rechargeable battery in a wireless charging mode. Fig. 2 shows a schematic diagram of an infrared sensing device provided by the invention.
The mouse control device comprises: a control device main body, a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body;
the third wireless communication module and the physical communication interface are both connected with the data processing module;
the control device main body is fixedly connected with the target equipment through the physical communication interface;
the third wireless communication module is used for receiving inertial measurement data sent by the mouse finger ring and sending the inertial measurement data to the data processing module; the third wireless communication module is further used for receiving horizontal position offset data sent by the infrared sensing device and sending the horizontal position offset data to the data processing module;
the data processing module is used for generating a mouse instruction according to the inertia measurement data and the horizontal position offset data, and sending the mouse instruction to the target equipment through the physical communication interface so as to control the target equipment;
and the physical communication interface instructs a mouse to the target equipment so as to control the target equipment. Fig. 3 shows a schematic diagram of a mouse control device provided by the invention.
The mouse finger ring is worn on the index finger of the right hand of an operator, and the inertia measurement module is used for measuring the three-axis angular velocity information and the three-axis acceleration information, wherein the information is used for controlling the movement of a mouse cursor and the clicking, double clicking and roller clicking operations; when an operator moves a right hand to control a mouse cursor, an inertial measurement module on a mouse finger ring worn by a right index finger records triaxial acceleration information in the finger movement process, the obtained acceleration is used for integrating time for 2 times to obtain displacement, and the integration is iterated according to the following formula:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2，
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2，
wherein v (t) and v (t-1) respectively represent target speeds at times t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the inertial measurement module at times t and (t-1), Δt represents time intervals at times t and (t-1), and s (t) and s (t-1) respectively represent target displacements at times t and (t-1);
and iterating the formula to obtain the vertical displacement in the t time.
The infrared sensor is arranged under the display screen, horizontal position offset data are generated according to the movement of the fingers of a user, the positions of the fingers are (x, y) at the moment t, and the positions of the fingers move to (x+dx, y+dy) at the moment t+dt, so that the shielding of the fingers is sensed through the infrared sensor, and the horizontal position of the fingers at the moment is directly obtained; the infrared sensor determines the position information of the right hand and is used for controlling the mouse cursor to move on the horizontal plane.
The control device main body extracts multi-mode characteristics of horizontal shielding information captured by the infrared sensor and measurement information of an inertia measurement module on a mouse finger ring, then inputs the multi-mode characteristics into an LSTM model to obtain a recognition probability result of right index finger pressing, if the probability value is larger than a certain threshold value, a control signal of the mouse is sent to the data processing module, and if the probability value is smaller than the certain threshold value, recognition of the action is abandoned.
The control device main body is used for carrying out multi-mode feature extraction on the horizontal position offset data captured by the infrared sensor and the measurement information of the inertia measurement module on the mouse finger ring, and the feature extraction method of the measurement information of the inertia measurement module comprises normalization and amplitude absolute value average value extraction methods.
The control device main body extracts the multi-modal characteristics of the horizontal position offset data captured by the infrared sensor and the measurement information of the inertia measurement module on the mouse finger ring, then inputs the multi-modal characteristics into the LSTM model, the LSTM model automatically learns the time sequence characteristics of the input multi-modal characteristics, and outputs the probability that the characteristics belong to each category, so that the most likely category recognition category of the input multi-modal characteristics is obtained and is used as the recognition probability result of right index finger pressing.
The control device main body is used for carrying out multi-mode feature extraction on horizontal position offset data captured by an infrared sensor and measurement information of an inertial measurement module on a mouse finger ring, and the feature extraction method of a movable section of the inertial measurement module comprises a normalization and amplitude absolute value average value extraction method, wherein the calculation formula of the normalization extraction method is as follows:
wherein |x i The I represents the absolute value of the frame data in the inertial measurement module data, the min represents the value with the minimum absolute value in the active section of the inertial measurement module, and the max represents the value with the maximum absolute value in the active section;
the calculation formula of the absolute value mean value of the amplitude is as follows:
wherein MAV represents the extracted absolute value average feature, N represents the length of the sliding window, x i The amplitude of the frame signal is represented, and the average value of the absolute values of the amplitude is obtained by taking the average value of the absolute values of all data in a sliding window.
The control device main body extracts multi-mode characteristics of horizontal position offset data captured by an infrared sensor and measurement information of an inertia measurement module on a mouse finger ring, and then inputs the multi-mode characteristics into an LSTM model, wherein the LSTM model comprises the following specific steps of:
s1, the forgetting gate layer controls the information passing through the memory unit through an activation function, generates a value of 0 to 1 according to the output of the last moment and the input of the current moment, and controls whether the information learned at the last moment passes through or partially passes through the memory unit, wherein the formula is as follows:
f t ＝σ(W f ·[h t-1 ，x t ]+b f )，
where σ is the activation function, h t-i Is the output of the last moment, x t Is the current input, b f Is the offset, f t Is the output of the forgetting gate, W f The weight is taken as a value;
s2, generating new information to be updated; this step comprises two parts, the first part is that an input gate layer decides an updated value through a sigmoid activation function, the second part is that a tanh layer is used for generating a new candidate value, the new candidate value is used as a candidate value generated by a current layer and is added into a memory unit, and the values generated by the two parts are combined for updating:
i t ＝σ(W i .[h t-1 ，x t ]+b i )，
wherein b i And b C Is the offset, tanh is an activation function, i t Is the output of the output gate, W c For weights in the tanh layer, W i For the weight of the activation function sigma,
is an old memory cell;
s3, updating the old memory unit; first multiply the old memory cell by f t To delete unnecessary information and then to re-associate with
The candidate values are obtained by addition, and the formula is as follows:
wherein C is t Is a new memory cell, C t-1 A memory unit at the previous time;
s4, determining the output of the model; first get an initial output through the sigmoid layer and then use tanh to get C t The values scale to between-1 and 1, and are multiplied by the output obtained by the sigmoid layer pair by pair, so that the output of the LSTM model is obtained, and the formula of the step is as follows:
O t ＝σ(W o .[h t-1 ，x t ]+b o )，
h t ＝O t *tanh(C t )，
wherein O is t Is the output of the output gate, b o Is the offset, W o Is the weight of the activation function.
Fig. 4 shows a schematic diagram of an electronic touch screen system provided by the invention. In fig. 4, a module 1 is a mouse finger ring, a module 2 is an infrared sensing device, and a module 3 is a mouse control device.
Embodiment two: logic frame diagram of electronic touch screen system
The application provides a logic frame diagram of electronic touch screen system, this frame mainly divide into three major layers: an input layer, a processing layer and an output layer. These three parts will be described separately below. Fig. 5 shows a data processing flow chart of an electronic touch screen system provided by the invention. Fig. 6 shows a schematic structural diagram of an electronic touch screen system provided by the invention.
The input layer mainly acquires multi-mode information of a user and mainly comprises horizontal position offset information acquired by an infrared sensor, an inertial measurement module, an IMU (inertial measurement unit) motion sensor and acquired right index finger kinematics information, wherein the horizontal position offset information acquired by the infrared sensor is mainly used for controlling a mouse cursor to move left and right on a horizontal plane, and target motion information measured by the IMU motion sensor is used for controlling movement of the mouse cursor and clicking operations such as clicking, double clicking and rolling wheels.
The second part, the processing layer, mainly carry on the comprehensive treatment to infrared inductor information and IMU motion sensor information received, this layer mainly comprises three major parts: the method comprises the steps of controlling a cursor moving part of a mouse, and controlling a single click, double click, a roller and a reset part of a mouse ring;
the mouse cursor moving part is controlled, and the part mainly comprises an infrared sensor and an IMU motion sensor, wherein the IMU motion sensor is an inertial measurement unit and can measure the acceleration of three axes and the angular velocity (x axis, y axis and z axis) of the three axes. The main function of the part is to control the movement of the mouse cursor, and the implementation principle is to utilize the information captured by the infrared sensor and the IMU motion sensor to introduce the principle respectively and finally to introduce the fusion of the two.
In the mouse control system, the infrared sensor mainly determines the position information of the right hand, so that the horizontal movement of the mouse cursor is converted. For calculating the horizontal movement displacement, when the user wears the mouse finger ring to generate horizontal offset on the screen, the beam emitted by the miniature infrared emitter module of the mouse finger ring also generates corresponding horizontal offset, and the infrared sensor calculates the movement of the finger of the user to generate horizontal position offset data according to the position of the infrared beam received by the infrared sensor. The position of the finger at the time t is (x, y), and the position of the finger moves to the position (x+dx, y+dy) at the time t+dt, the shielding of the finger is sensed by the infrared sensor, and the horizontal position of the finger at the moment is directly obtained.
In the mouse control system, an IMU motion sensor is positioned on a mouse finger ring worn by a right index finger, the IMU motion sensor mainly measures three-axis angular velocity information and three-axis acceleration information, when a right hand wants to control a mouse cursor, the IMU motion sensor on the mouse finger ring worn by the right index finger records the three-axis acceleration information in the moving process, the obtained acceleration is used for integrating time for 2 times to obtain displacement, and the integration mode is iterated according to the following formula:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2,
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2,
wherein v (t) and v (t-1) respectively represent target speeds at times t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the IMU sensors at times t and (t-1), Δt represents time intervals at times t and (t-1), and s (t) and s (t-1) respectively represent target displacements at times t and (t-1);
and (5) iterating the formula to obtain the vertical displacement in the t time.
Fig. 7 shows a flowchart of generating a mouse manipulation instruction according to the present invention.
Thirdly, controlling a single click, double click, a roller and a reset part of the mouse ring; this part is mainly realized by a mouse finger ring module worn at the index finger of the right hand and an infrared sensor positioned below the screen. This part of the functionality is implemented by means of a deep neural network. LSTM is a time-cycled neural network that can solve the long-term dependency problem of RNNs, and is well suited to handle time-series problems. The quick and short left slide of the right index finger corresponds to the left click function of the mouse, the quick and short right slide of the right index finger corresponds to the right click function of the mouse, the double click of the right index finger corresponds to the double click function of the mouse, the quick and short lower slide of the right index finger corresponds to the lower slide function of the mouse wheel, the quick and short upper slide of the right index finger corresponds to the upper slide function of the mouse wheel, and the right hand fist corresponds to the reset function (the mouse cursor returns to the center of the screen). The control device main body extracts multi-mode characteristics of horizontal shielding information captured by the infrared sensor and IMU motion sensor information on the mouse finger ring, then inputs the multi-mode characteristics into the LSTM model, and can obtain the recognition probability result of the right index finger pressing, if the probability value is larger than a certain threshold value, a control signal of the mouse is sent to the data processing module, and if the probability value is smaller than a certain threshold value, recognition of the action is abandoned. The feature extraction method for the IMU motion sensor comprises MAV, RMS and the like, wherein the MAV is the mean value of absolute values of amplitude values, and the RMS is the root mean square. Fig. 8 shows a flowchart of generating a mouse movement instruction according to the present invention.
LSTM (Long Short-Term Memory) is capable of handling Long-time series information, mainly with input gates, forget gates, output gates in its internal structure. The LSTM model comprises the following working steps:
s1, the forgetting gate layer controls the information passing through the memory unit through an activation function, generates a value of 0 to 1 according to the output of the last moment and the input of the current moment, and controls whether the information learned at the last moment passes through or partially passes through the memory unit, wherein the formula is as follows:
f t ＝σ(W f .[h t-1 ，x t ]+b f )，
where σ is the activation function, h t-i Is the output of the last moment, x t Is the current input, b f Is the offset, f t Is the output of the forgetting gate, W f The weight is taken as a value;
s2, generating new information to be updated; this step comprises two parts, the first part is that an input gate layer decides an updated value through a sigmoid activation function, the second part is that a tanh layer is used for generating a new candidate value, the new candidate value is used as a candidate value generated by a current layer and is added into a memory unit, and the values generated by the two parts are combined for updating:
i t ＝σ(W i .[h t-1 ，x t ]+b i )，
wherein b i And b C Is the offset, tanh is an activation function, i t Is the output of the output gate, W c For weights in the tanh layer, W i For the weight of the activation function sigma,
is an old memory cell;
s3, updating the old memory unit; first multiply the old memory cell by f t To delete unnecessary information and then to re-associate with
The candidate values are obtained by addition, and the formula is as follows:
wherein C is t Is a new memory cell, C t-1 A memory unit at the previous time;
s4, determining the output of the model; first get an initial output through the sigmoid layer and then use tanh to get C t The values scale to between-1 and 1, and are multiplied by the output obtained by the sigmoid layer pair by pair, so that the output of the LSTM model is obtained, and the formula of the step is as follows:
O t ＝σ(W o .[h t-1 ，x t ]+b o )，
h t ＝O t *tanh(C t )，
wherein O is t Is the output of the output gate, b o Is the offset, W o Is the weight of the activation function.
And the third part, the output layer, mainly outputs and displays the control instruction obtained by the processing layer, and the layer mainly comprises a data processing module. The quick and short left slide of the right index finger corresponds to the left click function of the mouse, the quick and short right slide of the right index finger corresponds to the right click function of the mouse, the double click of the right index finger corresponds to the double click function of the mouse, the quick and short lower slide of the right index finger corresponds to the lower slide function of the mouse wheel, the quick and short upper slide of the right index finger corresponds to the upper slide function of the mouse wheel, and the right hand fist corresponds to the reset function (the mouse cursor returns to the center of the screen).
It is noted that the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The foregoing is merely exemplary of the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are to be included in the scope of the claims of the present invention.
Claims (6)
1. Implantable touch screen device based on IMU and infrared induction, its characterized in that: the device comprises a mouse finger ring, an infrared sensing device and a mouse control device;
the mouse finger ring comprises:
the device comprises a ring main body, an inertia measurement module, a miniature infrared emitter module, a first data acquisition module, a first wireless communication module, a first rechargeable battery and a first wireless charging module;
the inertial measurement module is connected with the miniature infrared emitter module, the miniature infrared emitter module is connected with the first data acquisition module, the first data acquisition module is connected with the first wireless communication module, the first wireless communication module is connected with the first rechargeable battery, the first rechargeable battery is connected with the first wireless charging module, and the inertial measurement module, the miniature infrared emitter module, the first data acquisition module, the first wireless communication module, the first rechargeable battery and the first wireless charging module are all arranged on the ring main body;
the inertial measurement module is used for generating inertial measurement data according to the movement of the finger of the user after the finger ring main body is worn on the finger of the user, and sending the inertial measurement data to the first data acquisition module; the miniature infrared emitter module is used for emitting infrared beams for being received by the infrared sensing device;
the first data acquisition module is used for transmitting the inertia measurement data to a mouse control device connected with target equipment through the first wireless communication module, so that the mouse control device generates a mouse instruction according to the inertia measurement data and controls the target equipment according to the mouse instruction;
the first wireless communication module is used for receiving the inertial measurement data sent by the first data acquisition module and sending the inertial measurement data to the data processing module;
the first rechargeable battery is used for supplying power to each module in the mouse finger ring;
the first wireless charging module is used for charging the first rechargeable battery in a wireless charging mode;
the infrared sensing device includes:
the device comprises an infrared sensing device main body, an infrared sensor, a second data acquisition module, a second wireless communication module, a second rechargeable battery and a second wireless charging module;
the infrared sensor is connected with the second data acquisition module, the second data acquisition module is connected with the second wireless communication module, the second wireless communication module is connected with the second rechargeable battery, the second rechargeable battery is connected with the second wireless charging module, and the infrared sensor, the second data acquisition module, the second wireless communication module, the second rechargeable battery and the second wireless charging module are all arranged on the infrared sensing device main body;
the infrared sensor is arranged under the display screen, the length of the infrared sensor is longer than that of the display screen, and the position of the infrared sensor is required to ensure that the infrared sensor can receive infrared wave beams emitted by a miniature infrared emitter module of a user when the user clicks the display screen by wearing a mouse finger ring; when the user wears the mouse finger ring to generate horizontal offset on the display screen, the beam emitted by the miniature infrared emitter module of the mouse finger ring also generates corresponding horizontal offset, and the infrared sensor calculates horizontal position offset data generated by the movement of the finger of the user according to the position of the infrared beam received by the infrared sensor and sends the horizontal position offset data to the second data acquisition module;
the second data acquisition module is used for transmitting the horizontal position offset data to a mouse control device connected with target equipment through the second wireless communication module, so that the mouse control device generates a mouse instruction according to the horizontal position offset data and controls the target equipment according to the mouse instruction;
the second wireless communication module is used for receiving the horizontal position offset data sent by the second data acquisition module and sending the horizontal position offset data to the data processing module;
the second rechargeable battery is used for supplying power to each module in the infrared sensor device;
the second wireless charging module is used for charging the second rechargeable battery in a wireless charging mode;
the mouse control device comprises: a control device main body, a third wireless communication module, a data processing module and a physical communication interface which are arranged on the control device main body;
the third wireless communication module and the physical communication interface are both connected with the data processing module;
the control device main body is fixedly connected with the target equipment through the physical communication interface;
the third wireless communication module is used for receiving inertial measurement data sent by the mouse finger ring and sending the inertial measurement data to the data processing module; the third wireless communication module is further used for receiving horizontal position offset data sent by the infrared sensing device and sending the horizontal position offset data to the data processing module;
the data processing module is used for generating a mouse instruction according to the inertia measurement data and the horizontal position offset data, and sending the mouse instruction to the target equipment through the physical communication interface so as to control the target equipment;
and the physical communication interface instructs a mouse to the target equipment so as to control the target equipment.
2. The IMU and infrared sensing based portable touch screen device of claim 1, wherein the mouse finger ring is worn on the right index finger of the operator, and the inertial measurement module measures three-axis angular velocity information and three-axis acceleration information for controlling movement of a mouse cursor and single click, double click and roller click operations; when an operator moves a right hand to control a mouse cursor, an inertial measurement module on a mouse finger ring worn by a right index finger records triaxial acceleration information in the finger movement process, the obtained acceleration is used for integrating time for 2 times to obtain displacement, and the integration is iterated according to the following two integration formulas:
v(t)＝v(t-1)+Δt(a(t)+a(t-1))/2， (1)
s(t)＝s(t-1)+Δt(v(t)+v(t-1))/2， (2)
wherein v (t) and v (t-1) respectively represent target speeds at times t and (t-1), a (t) and a (t-1) respectively represent target accelerations recorded by the inertial measurement module at times t and (t-1), Δt represents time intervals at times t and (t-1), and s (t) and s (t-1) respectively represent target displacements at times t and (t-1);
and (3) iterating the two integral formulas of the formula (1) and the formula (2) to obtain the vertical displacement in the t time.
3. The IMU and infrared sensing based portable touch screen device of claim 1, wherein the infrared sensor is disposed under the display screen, generates horizontal position offset data according to the movement of the finger of the user, the position of the finger is (x, y) at time t, and the position of the finger is moved to (x+dx, y+dy) at time t+dt, and the shielding of the finger is sensed by the infrared sensor, so that the horizontal position of the finger at the time is directly obtained; the infrared sensor determines the position information of the right hand and is used for controlling the mouse cursor to move on the horizontal plane.
4. The implantable touch screen device based on IMU and infrared sensing of claim 1,
the control device main body extracts multi-mode characteristics of horizontal position offset data captured by the infrared sensor and measurement information of an inertia measurement module on a mouse finger ring, then inputs the multi-mode characteristics into an LSTM model to obtain a recognition probability result of right index finger pressing, if the probability value is larger than a certain threshold value, a control signal of the mouse is sent to the data processing module, and if the probability value is smaller than the certain threshold value, recognition of the action is abandoned.
5. The IMU and infrared sensing based portable touch screen device of claim 4, wherein the control device body performs multi-modal feature extraction of horizontal position offset data captured by the infrared sensor and inertial measurement module measurement information on the mouse finger ring, and the feature extraction method of the inertial measurement module measurement information includes normalization and amplitude absolute value average extraction methods.
6. The device of claim 4, wherein the control device body extracts the multi-modal characteristics of the horizontal position offset data captured by the infrared sensor and the measurement information of the inertial measurement module on the mouse finger ring, and then inputs the multi-modal characteristics into the LSTM model, the LSTM model automatically learns the timing characteristics of the inputted multi-modal characteristics, and outputs the probability of the characteristics belonging to each category, thereby obtaining the category identification category to which the inputted multi-modal characteristics most likely belong as the identification probability result of the right index finger press.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010930016.8A CN112083832B (en) | 2020-09-07 | 2020-09-07 | Implantable touch screen device based on IMU and infrared induction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010930016.8A CN112083832B (en) | 2020-09-07 | 2020-09-07 | Implantable touch screen device based on IMU and infrared induction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112083832A CN112083832A (en) | 2020-12-15 |
| CN112083832B true CN112083832B (en) | 2023-06-30 |
Family
ID=73732472
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010930016.8A Active CN112083832B (en) | 2020-09-07 | 2020-09-07 | Implantable touch screen device based on IMU and infrared induction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112083832B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112929734B (en) * | 2021-02-05 | 2023-09-05 | 维沃移动通信有限公司 | Screen projection method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203299759U (en) * | 2013-05-04 | 2013-11-20 | 罗美华 | Automatic charging energy-saving wireless mouse |
| CN107305420A (en) * | 2016-04-24 | 2017-10-31 | 吴金术 | A kind of computer mouse automatic sensing energy saver |
| CN110928432A (en) * | 2019-10-24 | 2020-03-27 | 中国人民解放军军事科学院国防科技创新研究院 | Ring mouse, mouse control device and mouse control system |
-
2020
- 2020-09-07 CN CN202010930016.8A patent/CN112083832B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203299759U (en) * | 2013-05-04 | 2013-11-20 | 罗美华 | Automatic charging energy-saving wireless mouse |
| CN107305420A (en) * | 2016-04-24 | 2017-10-31 | 吴金术 | A kind of computer mouse automatic sensing energy saver |
| CN110928432A (en) * | 2019-10-24 | 2020-03-27 | 中国人民解放军军事科学院国防科技创新研究院 | Ring mouse, mouse control device and mouse control system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112083832A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021082749A1 (en) | Action identification method based on artificial intelligence and related apparatus | |
| CN102915111B (en) | A kind of wrist gesture control system and method | |
| CN110928432B (en) | Finger ring mouse, mouse control device and mouse control system | |
| CN202584010U (en) | Wrist-mounting gesture control system | |
| CN106462239A (en) | Finger tracking | |
| Zhang et al. | Recognizing hand gestures with pressure-sensor-based motion sensing | |
| CN110610158A (en) | Human body posture identification method and system based on convolution and gated cyclic neural network | |
| CN111722713A (en) | Multi-mode fused gesture keyboard input method, device, system and storage medium | |
| Xiong et al. | A novel HCI based on EMG and IMU | |
| Zhu et al. | Control with gestures: A hand gesture recognition system using off-the-shelf smartwatch | |
| Coffen et al. | Tinydl: Edge computing and deep learning based real-time hand gesture recognition using wearable sensor | |
| CN104571837A (en) | Method and system for realizing human-computer interaction | |
| CN112083832B (en) | Implantable touch screen device based on IMU and infrared induction | |
| CN105919599B (en) | Finger movement detection identifying system and method based on Magnetic Sensor | |
| Yuan et al. | Smartphone-based activity recognition using hybrid classifier | |
| Hernandez et al. | Machine learning techniques for motion analysis of fatigue from manual material handling operations using 3D motion capture data | |
| CN113849068A (en) | Gesture multi-mode information fusion understanding and interacting method and system | |
| Li et al. | Hand gesture recognition and real-time game control based on a wearable band with 6-axis sensors | |
| Sharma et al. | Trbaggboost: An ensemble-based transfer learning method applied to Indian Sign Language recognition | |
| Wang et al. | Wearable sensors-based hand gesture recognition for human–robot collaboration in construction | |
| Pan et al. | State-of-the-art in data gloves: A review of hardware, algorithms, and applications | |
| US8749488B2 (en) | Apparatus and method for providing contactless graphic user interface | |
| KR101930942B1 (en) | Sign language recognition system and method | |
| Zhang et al. | Integrating intention-based systems in human-robot interaction: a scoping review of sensors, algorithms, and trust | |
| Avadut et al. | A deep learning based iot framework for assistive healthcare using gesture based interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |