CN106952305B - Information processing method and device, electronic equipment and wearable equipment - Google Patents
Information processing method and device, electronic equipment and wearable equipment Download PDFInfo
- Publication number
- CN106952305B CN106952305B CN201710184743.2A CN201710184743A CN106952305B CN 106952305 B CN106952305 B CN 106952305B CN 201710184743 A CN201710184743 A CN 201710184743A CN 106952305 B CN106952305 B CN 106952305B
- Authority
- CN
- China
- Prior art keywords
- user
- tympanic membrane
- image
- preset condition
- membrane deformation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/10—Earpieces; Attachments therefor ; Earphones; Monophonic headphones
- H04R1/1091—Details not provided for in groups H04R1/1008 - H04R1/1083
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
Abstract
The invention discloses an information processing method and device, electronic equipment and wearable equipment, and relates to the field of intelligent equipment. The information processing method comprises the following steps: acquiring an eardrum image acquired within an ear canal of a user; judging whether the deformation amount of the eardrum of the user meets a first preset condition according to the eardrum image; generating at least one trigger signal when the amount of drum membrane deformation of the user meets a first preset condition; wherein the trigger signal is used to cause the user to perform a swallowing action. The ear discomfort caused by the air pressure change can be relieved by adapting to the actual state of a user in real time based on the combination of the image acquisition in the ear canal and the image processing of the tympanic membrane image, so that the ear discomfort caused by the air pressure change of the environment can be relieved by the user, and the comfort of a human body in the air pressure change environment is improved.
Description
Technical Field
The invention relates to the field of intelligent equipment, in particular to an information processing method and device, electronic equipment and wearable equipment.
Background
Most people experience more or less ear discomfort during the ride of the aircraft. The reason is that the air pressure of the airplane is constantly changed in the processes of taking off, flying and landing, the inside of the ear chamber is influenced by the external air pressure, and the eardrum separating the middle ear chamber and the external auditory canal is deformed under the action of the atmospheric pressure, so that the ear discomfort of people is caused. At present, prompting information such as pictures, texts, voice, vibration and the like is generally adopted to prompt passengers to adjust themselves, for example, the passengers are prompted to swallow by broadcasting to relieve discomfort of human ears when taking an airplane. However, the prompt information usually cannot be accurately adapted to the actual air pressure change, and the actual responses of different people to the air pressure change are different, so that the problem that the human ears are uncomfortable due to the air pressure change cannot be effectively solved in the existing mode.
Disclosure of Invention
The invention provides an information processing method and device, electronic equipment and wearable equipment, which can adapt to environmental air pressure change and help a human body to relieve ear discomfort.
In a first aspect, the present invention provides an information processing method, including:
acquiring an eardrum image in an ear canal of a user;
judging whether the deformation amount of the eardrum of the user meets a first preset condition according to the eardrum image;
generating at least one trigger signal when the amount of drum membrane deformation of the user meets a first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
In one possible implementation, the generating at least one trigger signal includes:
applying an electrical stimulation signal to a body surface electrode to form a swallowing stimulus to the user when the body surface electrode is conductively configured at a predetermined body surface location of the user;
alternatively, the first and second electrodes may be,
sending a control signal to an external device to cause the external device to apply an electrical stimulation signal to the connected body surface electrodes upon receiving the control signal to form a swallowing stimulus for the user when the body surface electrodes are conductively configured at a predetermined body surface location of the user.
In one possible implementation, the generating at least one trigger signal when the amount of tympanic membrane deformation of the user satisfies a first preset condition includes:
acquiring external auditory canal air pressure of the user;
and generating the at least one trigger signal when the deformation quantity of the eardrum of the user meets a first preset condition and the variation quantity of the external ear canal air pressure of the user in unit time meets a second preset condition.
In one possible implementation, the method further includes:
when the tympanic membrane deformation of the user does not meet the first preset condition, acquiring the external auditory canal air pressure of the user;
judging whether the variation of the external auditory canal air pressure of the user in unit time meets a second preset condition or not;
and when the variation of the external ear canal air pressure of the user in unit time meets the second preset condition, generating the at least one trigger signal.
In one possible implementation, the acquiring an image of an eardrum within an ear canal of a user includes:
acquiring at least two speckle images acquired at least two different times for the same area containing the eardrum within the ear canal of the user;
accordingly, the determining whether the amount of drum membrane deformation of the user satisfies a first preset condition according to the drum membrane image includes:
determining a sample image with the tympanic membrane deformation amount as a reference value and a target image with the tympanic membrane deformation amount to be measured in the at least two speckle images;
comparing the sample image and the target image to determine a speckle field area at a location of maximum deformation in the sample image and the target image;
calculating the tympanic deformation amount based on a correlation analysis of the speckle images to determine that the tympanic deformation amount satisfies the first preset condition when the tympanic deformation amount is greater than a preset threshold, the tympanic deformation amount being a spatial displacement between speckle field areas at a maximum deformation position in the sample image and the target image.
In a second aspect, the present invention also provides an information processing apparatus comprising:
the acquisition module is used for acquiring an eardrum image in an ear canal of a user;
the judging module is used for judging whether the tympanic membrane deformation quantity of the user meets a first preset condition according to the tympanic membrane image;
a generating module, configured to generate at least one trigger signal when the amount of drum membrane deformation of the user satisfies a first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
In a third aspect, the present invention also provides an information processing apparatus comprising:
a processing component;
storage means for storing instructions executable by the processing means;
wherein the processing component is configured to:
acquiring an eardrum image in an ear canal of a user;
judging whether the deformation amount of the eardrum of the user meets a first preset condition according to the eardrum image;
generating at least one trigger signal when the amount of drum membrane deformation of the user satisfies a first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
In a fourth aspect, the present invention further provides a wearable device, including a supporting component, a camera component, and a data transmitting module; wherein the content of the first and second substances,
the support member is wearable on a user's head or ear;
the camera shooting part is arranged on the supporting part so that a lens of the camera shooting part can be aligned with an eardrum in an ear canal of a user when the supporting part is worn by the user;
the data sending module is connected with the camera part and used for sending the eardrum image collected in the ear canal of the user and collected by the camera part to an information processing device so as to enable the information processing device to:
receiving an eardrum image from the data transmission module;
judging whether the deformation amount of the eardrum of the user meets a first preset condition according to the eardrum image;
generating at least one trigger signal when the amount of drum membrane deformation of the user meets a first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
In one possible implementation manner, the wearable device further comprises a body surface electrode, wherein the body surface electrode is fixed on the supporting component through an electrode support, so that the body surface electrode can be located at a preset body surface position of a user when the supporting component is worn by the user;
the information processing apparatus generates a first signal when an amount of drum membrane deformation of the user satisfies a first preset condition; the wearable device further comprises a receiving module for receiving the first signal, and an electric signal generating module connected with the receiving module;
the electric signal generating module is used for applying an electric stimulation signal to the connected body surface electrodes when the receiving module receives the first signal so as to apply swallowing stimulation to the user when the body surface electrodes are configured at the preset body surface positions of the user in an electrically conductive mode.
In one possible implementation, the information processing apparatus is provided on the support member as a part of the wearable device.
According to the technical scheme, based on the combination of the image acquisition in the ear canal and the image processing of the tympanic membrane image, the swallowing action of the user can be prompted when the tympanic membrane deformation quantity meets the first preset condition, so that the ear discomfort caused by the air pressure change can be relieved by adapting to the actual state of the user in real time, and even the further deformation of the tympanic membrane can be avoided by the swallowing action before the user feels the ear discomfort, so that the ear discomfort caused by the environmental air pressure change can be greatly relieved for the user, and the comfort of the human body in the air pressure change environment can be improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow chart of an information processing method provided by an embodiment of the invention;
fig. 2 is a schematic structural diagram of a wearable device provided in an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating an implementation manner of an information processing method according to another embodiment of the present invention;
fig. 4 is a schematic diagram illustrating an implementation manner of an information processing method according to another embodiment of the present invention;
FIG. 5 is a flowchart illustrating steps of a method for processing information according to another embodiment of the present invention;
FIG. 6 is a flowchart illustrating steps of a method for processing information according to another embodiment of the present invention;
fig. 7 is a block diagram of an information processing apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It is noted that the methods provided by the present invention are all used for non-therapeutic purposes.
Fig. 1 is a flowchart of an information processing method according to an embodiment of the present invention. Referring to fig. 1, the information processing method of the present embodiment includes:
step 101: an image of the tympanic membrane within the ear canal of the user is acquired.
Step 102: whether the amount of drum membrane deformation of the user meets a first preset condition is judged according to the drum membrane image.
Step 103: at least one trigger signal is generated when the amount of drum membrane deformation of the user meets a first preset condition. Wherein the trigger signal is used for enabling the user to perform a swallowing action.
It should be noted that the information processing method of the present embodiment can be applied to any electronic device with an image processing function to implement a function of relieving the user's ear discomfort, alone or in combination with other devices, and specific examples will be described in detail later.
It should be noted that, in the present embodiment, the eardrum image can be acquired by, for example, any image pickup component that can be used for imaging the surface of the eardrum in the external auditory canal, such as a miniature camera, a fiber optic probe, an infrared sensor, an otoscope equipped with an image sensor, and the like, the imaging accuracy of which can be adaptively configured according to the accuracy requirement for determining the deformation of the eardrum, and which can be equipped with an illumination device to improve the imaging quality. On this basis, the eardrum image may be acquired in a manner such as receiving an electrical signal of the eardrum image from the image pickup part via a signal transmission line, or receiving the eardrum image transmitted by the image pickup part via a communication connection based on a wire or a wireless, or the like. The acquired tympanic membrane image can be an original signal directly obtained by acquisition, or an image signal obtained by processing the original signal, such as amplification, noise reduction, interception, compression and the like; at least part of data irrelevant to the determination of the tympanic membrane deformation amount can be removed from the original signal so as to improve the transmission efficiency and the processing speed.
It should be noted that the amount of deformation of the eardrum in this embodiment refers to a physical quantity describing the magnitude of the deformation of the eardrum, and the first preset condition in this embodiment refers to a defining criterion for determining whether to enable the user to swallow based on the image of the eardrum, and both of them can be preset according to actual requirements when implemented. For example, image recognition may be performed on a plurality of acquired tympanic membrane images to determine at least one tympanic membrane image without deformation of the tympanic membrane, and the image features of the tympanic membrane without deformation are extracted, so that the first preset condition is set as: the matching rate of the extracted image characteristics of the eardrum in the eardrum image compared with the image characteristics of the eardrum without deformation is less than or equal to 70%, so that the judgment result of the eardrum deformation is quickly obtained based on the image processing of the eardrum image. As can be seen from the above example, determining whether the amount of drum membrane deformation of the user satisfies the first preset condition does not necessarily include calculating the amount of drum membrane deformation according to the drum membrane image, but may have a corresponding form according to the setting manner.
It is further noted that the at least one triggering signal may comprise a prompt signal directly output to the user (e.g., a light, an image, a voice, a text, a vibration or a combination thereof for communicating a prompt to the user), or a signal for triggering a user prompt process (for example, an image data signal transmitted to the display device, a start voltage signal output to the light emitting diode, or a wireless signal sent to the smartphone to trigger the user prompt process, etc.), or a human body stimulation signal for prompting the user to perform a swallowing action (for example, an image signal for prompting the human body to secrete saliva through conditioned reflex, or an electrical stimulation signal applied to the human body at a predetermined body surface position such as a swallowing nerve, a relevant acupoint, etc., or the like), or a signal for triggering a process for prompting the user to perform a swallowing action (for example, a wired signal for controlling the electrical stimulation device to output the electrical stimulation signal to the body surface electrode). Due to triggering of swallowing action by the user
It can be seen that, according to the embodiment of the present invention, based on the combination between the image acquisition in the ear canal and the image processing of the tympanic membrane image, the user can be prompted to perform the swallowing action when the tympanic membrane deformation amount satisfies the first preset condition, so that the user can adapt to the actual state of the user in real time to relieve the ear discomfort caused by the change of the air pressure, and even before the user feels the ear discomfort, the further deformation of the tympanic membrane can be avoided by the swallowing action, thereby greatly helping the user to relieve the ear discomfort caused by the change of the ambient air pressure, and improving the comfort of the human body when the human body is in the environment with the change of the air pressure.
Fig. 2 is a schematic structural diagram of a wearable device provided in an embodiment of the present invention, and the wearable device may be used to implement the information processing method. The wearable device is embodied as a headset in the present embodiment, and referring to fig. 2, the wearable device includes a support 21a, two left and right headset units 21b, a micro camera 22, an information processing apparatus 23, a body surface electrode 24, and an electrode support 25, wherein:
the bracket 21a is in an arc shape which is adapted to the curve of the head of a human body on the whole, and can be stretched to adapt to the wearing requirements of different users; the two earphone units 21b are mounted at both ends of the holder 21a, respectively, and connected to each other by a connection line hidden in the inner cavity of the holder 21 a. The support 21a and the two earphone units 21b constitute a support part of the wearable device, which not only plays a role of fixing and supporting other parts, but also can be worn on the head by the user, providing a position and a view angle required for collecting the tympanic membrane image.
The micro camera 22 is provided inside the earphone unit 21b of the left ear as an image pickup part of the wearable device, and its arrangement in a relative position and a relative angle enables the lens of the micro camera 22 to be directed to the eardrum in the ear canal of the user after the user wears the support part. Not shown in fig. 2, the housing of the miniature camera 22 is provided with a light guide fiber for providing illumination, which can emit illumination light to image the eardrum in the ear canal after the user wears the support member; the micro camera 22 includes an image sensor, and can receive the light reflected by the eardrum and convert the light into an image signal, and then transmit the image signal to the information processing device 23 through a connecting wire hidden inside the bracket 21a for processing.
The information processing device 23 is disposed inside the support 21a, includes an image signal interface, a processing unit, a storage unit, an electrical stimulation signal output port, and the like, can receive the image signal from the micro camera 22 through the image signal interface, and performs the above-mentioned flow of steps 101 to 103 based on the cooperation of the processing unit and the storage unit. For example, the storage component has stored therein instructions executable by the processing component such that the processing component is capable of: receiving an image signal to acquire an eardrum image; processing the acquired tympanic membrane image based on a white light digital speckle method, and calculating the deformation quantity (space displacement at the maximum deformation position) of the tympanic membrane through randomly distributed spots on the tympanic membrane; when the deformation amount of the tympanic membrane is greater than a preset threshold (e.g., 0.1mm to 2mm), an electrical stimulation signal is output to the body surface electrode 24 through the electrical stimulation signal output port and the connection line.
The body surface electrodes 24 are secured to the support member by electrode supports 25 to enable the body surface electrodes to be positioned at a predetermined body surface location of the user when the support member is worn by the user (e.g., the body surface electrodes 24 shown in fig. 2 are positioned in a manner such that they are positioned substantially at the wind pool point of the user's body surface when the user wears the headset). Any electrical stimulation method capable of prompting the human body to swallow can be preset: the number of body surface electrodes, the predetermined body surface location to which each body surface electrode corresponds, the electrical stimulation signals corresponding to each body surface electrode, and the signal timing for applying the electrical stimulation signals to the body surface electrodes, among other things. For example, the wearable device shown in fig. 2 is provided with a body surface electrode 24 corresponding to the Fengchi acupoint, and the swallowing action of the human body can be promoted by applying pulse signals with fixed frequency to the body surface electrode 24. It should be noted that electrical stimulation methods such as nerve stimulation and electro-acupuncture, which can promote swallowing of a human body, are well known to those skilled in the art and will not be described herein.
It should be noted that the processing Unit may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, or a plurality of integrated circuits for controlling program execution, and the like.
The Memory component may include a Read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed, and so forth.
The information processing apparatus 23 in the present embodiment can implement the information processing method as an electronic device having an image processing function, and a wearable device having the information processing apparatus 23 as an internal component can prompt a user to perform a swallowing action when the amount of deformation of the eardrum of the user is too large in combination with the information processing method after being worn by the user. As an example, the wearable device of the present embodiment is used as follows:
referring to fig. 2 and 3, after the user wears the headset shown in fig. 2, the miniature camera 22 is inserted into the ear canal of the user's left ear with its lens directed toward the tympanic membrane. Meanwhile, after wiping the wind pool hole, the user can electrically attach the body surface electrode 24 to the wind pool hole by adjusting the electrode support 25, so as to complete the preparation before starting. After the power of the earphone is turned on, the information processing device 23 starts to be in a working state, collects images in an initialization stage to identify the eardrum, confirms that all the components are working normally and the earphone is worn correctly if the identification is successful (for example, the eardrum is identified in the images), and prompts the user that the working state is abnormal if the identification is failed (for example, the eardrum is not identified in the images). In the normal working phase after the initialization phase is finished, the information processing device 23 periodically collects tympanic membrane images and outputs pulse signals with fixed frequency to the body surface electrodes when the deformation of the tympanic membrane meets a first preset condition so as to form swallowing stimulation to the user, so that the user is prompted to swallow to open the eustachian tube and balance the air pressure on two sides of the tympanic membrane. In a use scenario, for example, when an airplane is taken, the wearable device shown in fig. 2 may help a user keep the air pressure balance on both sides of the eardrum by monitoring the deformation state of the eardrum of the user in real time, and alleviate the ear discomfort caused by an excessive difference between the air pressures on both sides of the eardrum.
It can be seen that the wearable device of this embodiment can adapt to the actual state of the user in real time to alleviate the ear discomfort caused by the change of air pressure, and even can avoid the further deformation of the eardrum by the swallowing action before the user feels the ear discomfort, can help the user to alleviate the ear discomfort caused by the change of ambient air pressure, and improve the comfort of the human body when being in the environment of air pressure change.
It is understood that the structure of the wearable device shown in fig. 2 is merely an example, and the number, type, structure, and arrangement of each component part can be adjusted within the achievable range, and a part of the components can be removed or other components can be added within the achievable range. For example, a micro-camera may be symmetrically added to the earphone unit 21b of the right ear, and a corresponding connection line is provided to connect the micro-camera to another image signal interface of the information processing device 23, so as to help maintain the balance of air pressure on both sides of the tympanic membrane in the ear canal of the right ear in the same manner as the left ear. As another example, the information processing device 23 may be integrated with the original circuit structure of the earphone, for example, all circuit elements may be disposed on the same circuit substrate, or the executable instructions corresponding to the information processing method may be configured based on the original processing component and storage component of the earphone, and corresponding circuit connection lines and signal interfaces may be added.
Fig. 4 is a schematic diagram of an implementation manner of an information processing method according to another embodiment of the present invention. Referring to fig. 4, the electronic device implementing the information processing method in this embodiment is specifically a terminal device 41, and the terminal device 41 may receive the eardrum image from the wearable device 42, and may issue a user prompt according to the processing of the eardrum image, and prompt the user to swallow when the amount of eardrum deformation of the user meets a first preset condition.
Specifically, the wearable device 42 in the present embodiment includes a support member (including both left and right earphone heads) having the same outer shape as the earphone housing of an earphone or an in-ear earphone, and thus can be worn at the ear in the same manner as the earphone or the in-ear earphone. Not shown in fig. 4, the wearable device 42 has a camera component disposed on the support component in such a way that when the user wears the support component on the ear, the lens of the camera component can be inserted with the earphone head into the ear canal of the user and aimed at the eardrum in the ear canal of the user. Alternative implementations of the image capture component have been described previously and will not be described in detail herein. Furthermore, as shown in fig. 4, the wearable device 42 further includes two data transmitting modules 421 respectively disposed on the left and right earpieces, and the two data transmitting modules 421 may establish a communication connection with the terminal device 41 in advance in a wired or wireless manner, so as to transmit the tympanic membrane image captured by the camera to the electronic device.
The terminal device 41 in this embodiment may specifically be an electronic device such as a smart phone, a tablet computer, a notebook computer, a Personal Digital Assistant (PDA), etc., may include any one of the above processing components and any one of the above storage components, and receives the tympanic membrane image from the wearable device 42 by way of wired communication and/or wireless communication. After receiving the tympanic membrane image, the terminal device 41 may perform a determination whether the tympanic membrane deformation amount satisfies a first preset condition, and generate at least one trigger signal to prompt the user to perform a swallowing action when the determination result is yes.
In one example, the terminal device 41 transmits the eardrum images arranged in time series to the cloud server through a network connection to acquire a change of the eardrum deformation amount over time using cloud computing resources of the cloud server, thereby reminding the user of a swallowing action in the form of a popup message and a cue tone when the eardrum deformation amount exceeds a threshold set by the user in accordance with a trigger policy corresponding to a first preset condition, and transmitting a first signal to an electro-stimulation device (not shown in fig. 4) when a change amount of the eardrum deformation amount per unit time exceeds a threshold set by the user, so that the electro-stimulation device applies an electro-stimulation signal to a connected body surface electrode upon receiving the first signal to apply a swallowing stimulation to the user when the body surface electrode is conductively configured at a predetermined body surface position of the user. It can be seen that the triggering strategy mainly includes a determination condition related to the deformation amount of the tympanic membrane, and a triggering action corresponding to each determination result; in all possible implementations, the content in the trigger policy may originate from factory settings, user settings, push from a server, and so on. The electrostimulation device and the wearable device 42 may be independent devices or combined devices. For example, an electrical signal generation module may be provided in the wearable device 42 to apply an electrical stimulation signal to the connected body surface electrodes upon receiving the first signal, i.e., to integrate the electrical stimulation device into the wearable device 42. The setting manner of the body surface electrodes connected to the electrical signal generating module on the wearable device may refer to the setting manner of the body surface electrodes 24 in fig. 2, which is not described herein again.
Other forms of wearable devices are available, according to the examples of wearable devices provided above. For example, the micro camera 22 in fig. 2 may be combined with the support 21a, the housing of the left and right earphone units 21b, and the data transmission module 421 to form a simplified wearable device, that is, the mechanical structure of the headset may be used to carry the camera and the data transmission module, so as to implement the workflow shown in fig. 4. For another example, the wearable device 42 shown in fig. 4 may further include the information processing apparatus 23, the body surface electrodes 24, and the electrode holder 25, so as to implement the functions that can be implemented by the wearable device shown in fig. 2. Furthermore, the wearable device of the present invention is not limited to the form of an earphone, but may also have the form of an ear plug, an earbob, a headband, glasses, and the like. However, based on the design of the electronic device implementing the information processing method as a part of the headset, the electronic device may directly output a trigger signal in the form of audio to enable the user to perform swallowing. For example, the at least one trigger signal may include an audio prompt signal transmitted from the information processing device 23 to the two earphone units 21b through the audio output port thereof, so as to prompt the user to perform a swallowing action by sound. It can be seen that based on the above design, the audio playing function of the earphone can be integrated into the process of eliminating the ear discomfort of the user, and high integration between the functions is realized.
According to the example of the electrical stimulation method provided above, it is inferred that step 103 in the information processing method may include: applying an electrical stimulation signal to a body surface electrode to form a swallowing stimulus to the user when the body surface electrode is conductively configured at a predetermined body surface location of the user; alternatively, a control signal is sent to an external device to cause the external device to apply an electrical stimulation signal to the connected body surface electrodes upon receiving the control signal to form a swallowing stimulus for the user when the body surface electrodes are conductively configured at a predetermined body surface location of the user. That is, when the electronic device implementing the information processing method is equipped with the generation device of the electrical stimulation signal and the body surface electrodes, the electrical stimulation signal may be directly applied to a predetermined body surface position of the user by the electronic device; in other cases, the electronic device can control the external device to complete the process through the control signal, so as to simplify the structure of the electronic device and optimize the use process.
In addition to any of the above information processing methods, the method may further include the following steps shown in fig. 5:
step 501: when the tympanic membrane deformation amount of the user does not meet the first preset condition, the external auditory canal air pressure of the user is acquired.
Step 502: and judging whether the variation of the external auditory canal air pressure of the user in unit time meets a second preset condition.
Step 503: and generating at least one trigger signal when the variation of the external auditory canal air pressure of the user in unit time meets a second preset condition.
The method for acquiring the external auditory canal air pressure of the user may include: arranging an air pressure sensor at a position where the wearable device is inserted into an ear canal, and transmitting an acquired air pressure signal to electronic equipment for realizing the information processing method through a signal connecting wire; alternatively, the electronic device for implementing the information processing method receives the ambient air pressure transmitted by an external device in a wireless communication manner, and the external device may be, for example, an air pressure monitor in the cabin of the aircraft. Moreover, the second preset condition refers to a defining standard for determining whether the user needs to swallow based on the external auditory canal air pressure, and can be preset according to actual requirements when implemented. For example, the second preset condition may include: the variation of the external auditory canal air pressure in unit time exceeds a preset value (for example, exceeds 20-200 hPa/s). In addition, examples of the at least one trigger signal are described in detail in the foregoing, and are not described herein again.
Based on the above process, the information processing method of the embodiment can assist in triggering the swallowing action based on monitoring of the external auditory canal air pressure when the tympanic membrane image is abnormal and the swallowing action of the human body cannot be normally triggered, and is beneficial to improving the reliability and the application range of the product.
Further, the above step 103: generating at least one trigger signal when the amount of drum deformation of the user meets a first preset condition may specifically comprise the following steps not shown in the figures:
acquiring external auditory canal air pressure of the user;
and generating the at least one trigger signal when the deformation quantity of the eardrum of the user meets a first preset condition and the variation quantity of the external ear canal air pressure of the user in unit time meets a second preset condition.
Based on the above process, the information processing method of the present invention can generate the trigger signal when both the first preset condition and the second preset condition are satisfied, that is, both the deformation amount of the tympanic membrane and the variation amount of the external ear canal air pressure in a unit time length are taken as necessary conditions for generating the trigger signal, which is helpful for reducing the situation of error output.
In any of the above information processing methods, the step 101: acquiring an image of the tympanic membrane within the ear canal of the user may specifically include: at least two speckle images acquired at least two different times for the same area containing the eardrum within the ear canal of the user are acquired. Accordingly, referring to fig. 6, the above step 102: judging whether the tympanic membrane deformation amount of the user meets a first preset condition according to the tympanic membrane image, including:
step 601: a sample image in which the amount of tympanic membrane deformation is a reference value and an object image in which the amount of tympanic membrane deformation is to be measured are determined among the at least two speckle images.
Step 602: the sample image and the target image are compared to determine the speckle field area at the location of maximum deformation in the sample image and the target image.
Step 603: the amount of drum distortion, which is the spatial displacement between the speckle field areas at the maximum distortion position in the sample image and the target image, is calculated based on the correlation analysis of the speckle images to determine that the amount of drum distortion satisfies a first preset condition when the amount of drum distortion is greater than a preset threshold.
It should be noted that the acquisition and comparison of speckle images may be based on, for example, laser speckle or white light speckle, while the sample image may be determined only once throughout the monitoring process and used for multiple calculations of the amount of tympanic membrane deformation. Based on the process, the information processing method provided by the embodiment of the invention can realize real-time, quick and nondestructive tympanic membrane deformation measurement, saves a plurality of processing flows of irrelevant image information compared with the traditional imaging mode, and is beneficial to improving the reaction speed of products.
Based on the same inventive concept, fig. 7 is a block diagram of an information processing apparatus according to an embodiment of the present invention. Referring to fig. 7, the information processing apparatus of the embodiment of the present invention includes:
an acquisition module 71 for acquiring an image of the tympanic membrane in the ear canal of the user;
a judging module 72 for judging whether the amount of deformation of the eardrum of the user satisfies a first preset condition according to the eardrum image;
a generating module 73, configured to generate at least one trigger signal when the amount of deformation of the eardrum of the user satisfies a first preset condition; wherein the trigger signal is used for enabling the user to perform a swallowing action.
It should be noted that the information processing apparatus of the present embodiment can be applied to any electronic device having an image processing function to implement a function of alleviating user's ear discomfort, alone or in combination with other devices. Specific examples of the electronic device have been described in detail in the foregoing, and are not described herein again.
It should be noted that, in the present embodiment, the eardrum image can be acquired by, for example, any image pickup component that can be used for imaging the surface of the eardrum in the external auditory canal, such as a miniature camera, a fiber optic probe, an infrared sensor, an otoscope equipped with an image sensor, and the like, the imaging accuracy of which can be adaptively configured according to the accuracy requirement for determining the deformation of the eardrum, and which can be equipped with an illumination device to improve the imaging quality. On this basis, the eardrum image may be acquired in a manner such as receiving an electrical signal of the eardrum image from the image pickup part via a signal transmission line, or receiving the eardrum image transmitted by the image pickup part via a communication connection based on a wire or a wireless, or the like. The acquired tympanic membrane image can be an original signal directly obtained by acquisition, or an image signal obtained by processing the original signal, such as amplification, noise reduction, interception, compression and the like; at least part of data irrelevant to the determination of the tympanic membrane deformation amount can be removed from the original signal so as to improve the transmission efficiency and the processing speed.
It should be noted that the amount of deformation of the eardrum in this embodiment refers to a physical quantity describing the magnitude of the deformation of the eardrum, and the first preset condition in this embodiment refers to a defining criterion for determining whether to enable the user to swallow based on the image of the eardrum, and both of them can be preset according to actual requirements when implemented. For example, image recognition may be performed on a plurality of acquired tympanic membrane images to determine at least one tympanic membrane image without deformation of the tympanic membrane, and the image features of the tympanic membrane without deformation are extracted, so that the first preset condition is set as: the matching rate of the extracted image characteristics of the eardrum in the eardrum image compared with the image characteristics of the eardrum without deformation is less than or equal to 70%, so that the judgment result of the eardrum deformation is quickly obtained based on the image processing of the eardrum image. As can be seen from the above example, determining whether the amount of drum membrane deformation of the user satisfies the first preset condition does not necessarily include calculating the amount of drum membrane deformation according to the drum membrane image, but may have a corresponding form according to the setting manner.
It is further noted that the at least one triggering signal may comprise a prompt signal directly output to the user (e.g., a light, an image, a voice, a text, a vibration or a combination thereof for communicating a prompt to the user), or a signal for triggering a user prompt process (for example, an image data signal transmitted to the display device, a start voltage signal output to the light emitting diode, or a wireless signal sent to the smartphone to trigger the user prompt process, etc.), or a human body stimulation signal for prompting the user to perform a swallowing action (for example, an image signal for prompting the human body to secrete saliva through conditioned reflex, or an electrical stimulation signal applied to the human body at a predetermined body surface position such as a swallowing nerve, a relevant acupoint, etc., or the like), or a signal for triggering a process for prompting the user to perform a swallowing action (for example, a wired signal for controlling the electrical stimulation device to output the electrical stimulation signal to the body surface electrode). Due to triggering of swallowing action by the user
It can be seen that, according to the embodiment of the present invention, based on the combination between the image acquisition in the ear canal and the image processing of the tympanic membrane image, the user can be prompted to perform the swallowing action when the tympanic membrane deformation amount satisfies the first preset condition, so that the user can adapt to the actual state of the user in real time to relieve the ear discomfort caused by the change of the air pressure, and even before the user feels the ear discomfort, the further deformation of the tympanic membrane can be avoided by the swallowing action, thereby greatly helping the user to relieve the ear discomfort caused by the change of the ambient air pressure, and improving the comfort of the human body when the human body is in the environment with the change of the air pressure.
In one possible implementation, the generating module 73 is specifically configured to apply electrical stimulation signals to the body surface electrodes to form swallowing stimuli to the user when the body surface electrodes are conductively configured at predetermined body surface locations of the user; alternatively, the generating module 73 is specifically configured to send a control signal to the external device, so that the external device applies an electrical stimulation signal to the connected body surface electrodes when receiving the control signal, so as to form swallowing stimulation for the user when the body surface electrodes are conductively configured on a predetermined body surface position of the user.
In one possible implementation, the generating module 73 includes:
an acquisition unit for acquiring external auditory canal air pressure of a user;
the generating unit is used for generating at least one trigger signal when the deformation quantity of the eardrum of the user meets a first preset condition and the variation quantity of the external ear canal air pressure of the user in unit time length meets a second preset condition.
In one possible implementation, the apparatus further includes:
the second acquisition module is used for acquiring the external auditory canal air pressure of the user when the tympanic membrane deformation of the user does not meet the first preset condition;
the second judgment module is used for judging whether the variation of the external auditory canal air pressure of the user in unit time meets a second preset condition or not;
and the second generation module is used for generating at least one trigger signal when the variation of the external auditory canal air pressure of the user in unit time meets a second preset condition.
In a possible implementation, the acquisition module 71 is specifically configured to acquire at least two speckle images acquired at least two different times for the same area containing the eardrum within the ear canal of the user; accordingly, the determination module 72 includes:
a determination unit for determining a sample image in which an amount of tympanic membrane deformation is a reference value, and an object image in which the amount of tympanic membrane deformation is to be measured, among the at least two speckle images;
a comparison unit for comparing the sample image and the target image to determine a speckle field area at a maximum deformation position in the sample image and the target image;
and a calculation unit for calculating a drum deformation amount based on the correlation analysis of the speckle images to determine that it satisfies a first preset condition when the drum deformation amount is greater than a preset threshold, the drum deformation amount being a spatial displacement between the speckle field areas at a maximum deformation position in the sample image and the target image.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (9)
1. An information processing method characterized by comprising:
acquiring at least two speckle images acquired at least two different times for the same area containing the eardrum within the ear canal of the user;
determining a sample image with the tympanic membrane deformation amount as a reference value and a target image with the tympanic membrane deformation amount to be measured in the at least two speckle images;
comparing the sample image and the target image to determine a speckle field area at a location of maximum deformation in the sample image and the target image;
calculating the tympanic membrane deformation amount based on the correlation analysis of the speckle images to determine that the tympanic membrane deformation amount satisfies a first preset condition when the tympanic membrane deformation amount is greater than a preset threshold, the tympanic membrane deformation amount being a spatial displacement between speckle field areas at a maximum deformation position in the sample image and the target image;
generating at least one trigger signal when the amount of drum membrane deformation of the user satisfies the first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
2. The method of claim 1, wherein the generating at least one trigger signal comprises:
applying an electrical stimulation signal to a body surface electrode to form a swallowing stimulus to the user when the body surface electrode is conductively configured at a predetermined body surface location of the user;
alternatively, the first and second electrodes may be,
sending a control signal to an external device to cause the external device to apply an electrical stimulation signal to the connected body surface electrodes upon receiving the control signal to form a swallowing stimulus for the user when the body surface electrodes are conductively configured at a predetermined body surface location of the user.
3. The method according to claim 1, wherein generating at least one trigger signal when the amount of tympanic membrane deformation of the user satisfies the first preset condition comprises:
acquiring external auditory canal air pressure of the user;
and generating the at least one trigger signal when the deformation amount of the eardrum of the user meets the first preset condition and the variation amount of the external ear canal air pressure of the user in unit time meets a second preset condition.
4. The method of claim 1, further comprising:
when the tympanic membrane deformation of the user does not meet the first preset condition, acquiring the external auditory canal air pressure of the user;
judging whether the variation of the external auditory canal air pressure of the user in unit time meets a second preset condition or not;
and when the variation of the external ear canal air pressure of the user in unit time meets the second preset condition, generating the at least one trigger signal.
5. An information processing apparatus characterized by comprising:
an acquisition module for acquiring at least two speckle images acquired at least two different times for a same area containing the eardrum within the ear canal of the user;
the judging module is used for determining a sample image with the tympanic membrane deformation as a reference value and a target image with the tympanic membrane deformation to be measured in the at least two speckle images;
comparing the sample image and the target image to determine a speckle field area at a location of maximum deformation in the sample image and the target image;
calculating the tympanic membrane deformation amount based on the correlation analysis of the speckle images to determine that the tympanic membrane deformation amount satisfies a first preset condition when the tympanic membrane deformation amount is greater than a preset threshold, the tympanic membrane deformation amount being a spatial displacement between speckle field areas at a maximum deformation position in the sample image and the target image;
a generating module, configured to generate at least one trigger signal when the amount of drum membrane deformation of the user satisfies the first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
6. An information processing apparatus characterized by comprising:
a processing component;
storage means for storing instructions executable by the processing means;
wherein the processing component is configured to:
acquiring at least two speckle images acquired at least two different times for the same area containing the eardrum within the ear canal of the user;
determining a sample image with the tympanic membrane deformation amount as a reference value and a target image with the tympanic membrane deformation amount to be measured in the at least two speckle images;
comparing the sample image and the target image to determine a speckle field area at a location of maximum deformation in the sample image and the target image;
calculating the tympanic membrane deformation amount based on the correlation analysis of the speckle images to determine that the tympanic membrane deformation amount satisfies a first preset condition when the tympanic membrane deformation amount is greater than a preset threshold, the tympanic membrane deformation amount being a spatial displacement between speckle field areas at a maximum deformation position in the sample image and the target image;
generating at least one trigger signal when the amount of drum membrane deformation of the user satisfies the first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
7. A wearable device is characterized by comprising a supporting component, a camera component and a data transmission module; wherein the content of the first and second substances,
the support member is wearable on a user's head or ear;
the camera shooting part is arranged on the supporting part so that a lens of the camera shooting part can be aligned with an eardrum in an ear canal of a user when the supporting part is worn by the user;
the data sending module is connected with the camera part and used for sending the eardrum image collected in the ear canal of the user and collected by the camera part to an information processing device so as to enable the information processing device to:
receiving an eardrum image from the data transmission module, including receiving at least two speckle images from the data transmission module acquired at least two different times for a same eardrum-containing region within the ear canal of the user;
determining a sample image with the tympanic membrane deformation amount as a reference value and a target image with the tympanic membrane deformation amount to be measured in the at least two speckle images;
comparing the sample image and the target image to determine a speckle field area at a location of maximum deformation in the sample image and the target image;
calculating the tympanic membrane deformation amount based on the correlation analysis of the speckle images to determine that the tympanic membrane deformation amount satisfies a first preset condition when the tympanic membrane deformation amount is greater than a preset threshold, the tympanic membrane deformation amount being a spatial displacement between speckle field areas at a maximum deformation position in the sample image and the target image;
generating at least one trigger signal when the amount of drum membrane deformation of the user satisfies the first preset condition;
wherein the trigger signal is used to cause the user to perform a swallowing action.
8. The wearable device according to claim 7, further comprising a body surface electrode fixed to the support member by an electrode holder to enable the body surface electrode to be positioned at a predetermined body surface location of a user when the support member is worn by the user;
the information processing apparatus generates a first signal when the amount of drum membrane deformation of the user satisfies the first preset condition; the wearable device further comprises a receiving module for receiving the first signal, and an electric signal generating module connected with the receiving module;
the electric signal generating module is used for applying an electric stimulation signal to the connected body surface electrodes when the receiving module receives the first signal so as to apply swallowing stimulation to the user when the body surface electrodes are configured at the preset body surface positions of the user in an electrically conductive mode.
9. Wearable device according to claim 7 or 8, characterized in that the information processing means is provided on the support part as part of the wearable device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710184743.2A CN106952305B (en) | 2017-03-24 | 2017-03-24 | Information processing method and device, electronic equipment and wearable equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710184743.2A CN106952305B (en) | 2017-03-24 | 2017-03-24 | Information processing method and device, electronic equipment and wearable equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106952305A CN106952305A (en) | 2017-07-14 |
| CN106952305B true CN106952305B (en) | 2020-01-24 |
Family
ID=59473327
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710184743.2A Active CN106952305B (en) | 2017-03-24 | 2017-03-24 | Information processing method and device, electronic equipment and wearable equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106952305B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107889010A (en) * | 2017-11-16 | 2018-04-06 | 魏磊 | Earphone warning system |
| CN107995548B (en) * | 2017-11-29 | 2019-05-03 | 宣城良知知识产权服务有限公司 | A kind of platform for protecting children's ear-drum |
| CN111491230A (en) * | 2020-03-04 | 2020-08-04 | 张广德 | Bluetooth headset unilateral audio data driving system and corresponding terminal |
| CN115206072A (en) * | 2021-04-09 | 2022-10-18 | Oppo广东移动通信有限公司 | Reminding method and device, wearable device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3733166B2 (en) * | 1996-03-11 | 2006-01-11 | キヤノン株式会社 | Multicolor image output apparatus and method |
| CN101664303A (en) * | 2009-10-10 | 2010-03-10 | 薛令军 | Visual acoustic immitance tester |
| CN101848851A (en) * | 2007-11-09 | 2010-09-29 | 三菱电机株式会社 | Elevator atmospheric pressure controller |
| CN102066223A (en) * | 2008-06-13 | 2011-05-18 | 三菱电机株式会社 | Elevator controller and elevator apparatus |
| CN205193801U (en) * | 2015-12-10 | 2016-04-27 | 佛山博智医疗科技有限公司 | Otology detects data management unit |
| CN106361347A (en) * | 2016-09-08 | 2017-02-01 | 佛山博智医疗科技有限公司 | Device and method for detecting vibration shape of tympanic membrane |
-
2017
- 2017-03-24 CN CN201710184743.2A patent/CN106952305B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3733166B2 (en) * | 1996-03-11 | 2006-01-11 | キヤノン株式会社 | Multicolor image output apparatus and method |
| CN101848851A (en) * | 2007-11-09 | 2010-09-29 | 三菱电机株式会社 | Elevator atmospheric pressure controller |
| CN102066223A (en) * | 2008-06-13 | 2011-05-18 | 三菱电机株式会社 | Elevator controller and elevator apparatus |
| CN101664303A (en) * | 2009-10-10 | 2010-03-10 | 薛令军 | Visual acoustic immitance tester |
| CN205193801U (en) * | 2015-12-10 | 2016-04-27 | 佛山博智医疗科技有限公司 | Otology detects data management unit |
| CN106361347A (en) * | 2016-09-08 | 2017-02-01 | 佛山博智医疗科技有限公司 | Device and method for detecting vibration shape of tympanic membrane |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106952305A (en) | 2017-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106952305B (en) | Information processing method and device, electronic equipment and wearable equipment | |
| US10856071B2 (en) | System and method for improving hearing | |
| CN110062318B (en) | Hearing aid system | |
| JP6657337B2 (en) | Method of operating hearing aid and hearing aid | |
| US20200016402A1 (en) | Input Selection For An Auditory Prosthesis | |
| US20100074460A1 (en) | Self-steering directional hearing aid and method of operation thereof | |
| US11223915B2 (en) | Detecting user's eye movement using sensors in hearing instruments | |
| JP2022534833A (en) | Audio profiles for personalized audio enhancements | |
| US9956407B2 (en) | Tonal deafness compensation in an auditory prosthesis system | |
| CN111669691B (en) | Hearing device comprising a sensor configuration detector | |
| EP3876828B1 (en) | Physical therapy and vestibular training systems with visual feedback | |
| CN104822119B (en) | Equipment for determining cochlea dead region | |
| US20220361787A1 (en) | Ear-worn device based measurement of reaction or reflex speed | |
| JP2022549548A (en) | A method and system for adjusting the level of haptic content when presenting audio content | |
| US20220053278A1 (en) | Systems and methods for adjustment of auditory prostheses based on tactile response | |
| EP3614695A1 (en) | A hearing instrument system and a method performed in such system | |
| CN106331972B (en) | Method and apparatus for placing an in-the-ear communication device in the ear canal of a user | |
| CN114009061A (en) | Mitigating crosstalk in tissue-conducting audio systems | |
| EP3685596B1 (en) | Remote hearing test system and associated methods for establishing an auditory profile and adjusting hearing aids using such a system | |
| US11272297B2 (en) | Intra-operative determination of vibratory coupling efficiency | |
| CN113825078A (en) | Hearing system with a hearing instrument and method for operating such a hearing system | |
| JP2022548811A (en) | Method and system for controlling haptic content | |
| CN113195043A (en) | Evaluating responses to sensory events and performing processing actions based thereon | |
| Stenfelt | A model for prediction of own voice alteration with hearing aids | |
| US20230320669A1 (en) | Real-time in-ear electroencephalography signal verification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |