JP2024065074A - DETECTION SYSTEM, DETECTION METHOD, AND PROGRAM - Google Patents

Abstract

[Problem] To reduce the burden on a user when detecting the movement of a moving object near the auricle. [Solution] One aspect of the present invention is a detection system that includes a vibrator that contacts the ear cartilage and vibrates the ear cartilage, and a vibration detector that detects the vibrations resulting from the vibration of the ear cartilage. Detection using such a detection system does not impose a burden on the user that would occur if the ear canal were blocked, so the detection system can reduce the burden on the user when detecting the movement of a moving object near the auricle. [Selected Figure] Figure 1

Description

This application claims priority to U.S. Provisional Patent Application No. 63/381,326, filed October 28, 2022, the contents of which are incorporated herein by reference.

The present invention relates to a detection system, a detection method, and a program.

In recent years, human-computer interaction using wearable devices has been attracting attention. As a result, wearable devices are becoming smaller day by day, leaving less space on the body available for user input. As a result, technology has been proposed that uses acoustic devices to sense the movement of moving objects near the auricle. Note that "near the auricle" means a position closer to the auricle than the tip of the nose, the top of the head, the shoulders, or the clavicle.

Y. Zhang, J. Zhou, G. Laput, and C. Harrison, “Skintrack: Using the body as an electrical waveguide for continuous finger tracking on the skin,” Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, p.1491-1503, CHI ’16, ACM, New York, NY, USA, 2016.

However, the technologies proposed so far have required users to wear earphones in their ears, which means the device blocks the user's ears. This has led to the problem that users can become fatigued when wearing the device for long periods of time.

In view of the above circumstances, the present invention aims to provide a technology that reduces the burden on a user when detecting the movement of a moving object near the auricle.

One aspect of the present invention is a detection system that includes a vibrator that contacts the ear cartilage and vibrates the ear cartilage, and a vibration detector that detects vibrations resulting from the vibration of the ear cartilage.

One aspect of the present invention is a detection system that includes eyeglasses, and includes a vibrator that contacts the ear cartilage and vibrates the ear cartilage, and a vibration detector that detects vibrations resulting from the vibration of the ear cartilage, the vibrator being located inside the end of the eyeglasses, and the vibration detector being located on the temple of the eyeglasses.

One aspect of the present invention is a detection method performed by a detection system that includes a vibrator that contacts the ear cartilage and vibrates the ear cartilage, and a vibration detector that detects vibrations resulting from the vibration of the ear cartilage, the detection method having an excitation step in which the vibrator excites vibrations, and a detection step in which the vibration detector detects vibrations resulting from the vibrations.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned detection system.

This invention makes it possible to reduce the burden on the user when detecting the movement of a moving object near the auricle.

FIG. 1 is an explanatory diagram illustrating a detection system according to an embodiment. 1A and 1B are diagrams illustrating an example of glasses including a vibrator and a vibration detector according to an embodiment. FIG. 11 is a diagram showing a first example of an experimental result in the embodiment. FIG. 11 is a diagram showing a second example of an experimental result in the embodiment. FIG. 11 is a diagram showing a third example of an experimental result in the embodiment. FIG. 11 is a diagram showing a fourth example of an experimental result in the embodiment. FIG. 11 is a diagram showing a fifth example of an experimental result in the embodiment. FIG. 2 is a diagram showing an example of a hardware configuration of an analysis apparatus according to an embodiment. 1 is a flowchart showing an example of a flow of a process executed by a detection system in an embodiment.

(Embodiment)
1 is an explanatory diagram illustrating a detection system 100 according to an embodiment. The detection system 100 is a system that senses the movement of a moving object in the auricle of a user 9. For example, the detection system 100 is a system that tracks the movement of a detection target such as a finger or hand when the user 9 moves the detection target in the vicinity of the auricle.

Such tracking results can be used to control devices such as smartphones. Specifically, if the trajectory of the movement of a detection target such as a finger or hand is associated in advance with a control signal to the device, the control signal to the device can be determined based on the tracking results, making it possible to control the device.

Furthermore, the detection system 100 is not limited to sensing the movement of the detection target, such as the movement of a finger or hand, but is also capable of sensing changes in the shape of the auricle. For example, assume that the user 9 touches his or her auricle with a finger or hand, changing the shape of the auricle. In such a case, the detection system 100 can detect the change in the shape of the auricle. Therefore, if the change in the shape of the auricle is associated in advance with a control signal for the device, the detection system 100 can control the device based on the change in the shape of the auricle.

Now, we will explain the detection system 100 in more detail. The detection system 100 includes a vibrator 1, a vibration detector 2, and an analysis device 3.

The vibrator 1 comes into contact with the ear cartilage and vibrates the ear cartilage. Specifically, the ear cartilage is the ear cartilage near the ear canal opening at the base of the ear. Because it comes into contact with the ear cartilage, the vibrator 1 does not block the ear canal. Because it comes into contact with the ear cartilage, the vibrator 1 excites the ear cartilage to vibrate. More specifically, the vibration generated by the vibrator 1 is an elastic wave that propagates through an elastic body. The elastic body may be a gas, a solid, or a liquid.

In addition, because ear cartilage is a softer tissue than other parts of the body, exciting vibrations in it is preferable for the detection system 100 to be able to excite vibrations with less power.

The frequency of the vibrations excited by the vibrator 1 in the ear cartilage may be a frequency in the ultrasonic band, a frequency in the audible band, or a frequency in the low-frequency band of 0 Hz to 20 Hz. The wider the frequency band, the greater the amount of information that can be obtained, and therefore the greater the sensing accuracy. Therefore, the greater the sensing accuracy is when, for example, a frequency in the audible band is used, rather than when only an ultrasonic frequency is used.

When the frequency of the vibration excited by vibrator 1 in the ear cartilage is an ultrasonic frequency band frequency, the user does not perceive the vibration excited by vibrator 1 as sound with his/her ears, unlike when an audible frequency band frequency is used. Therefore, when the frequency of the vibration excited by vibrator 1 in the ear cartilage is an ultrasonic frequency band frequency, it has the effect of not interfering with the user's ear-based activities, such as conversation, phone calls, and listening to music, as much as when an audible frequency band frequency is used.

The contact of the vibrator 1 with the ear cartilage also means contact via a solid or liquid. For example, applying gel to the ear cartilage and placing the vibrator 1 on top of the gel also means that the vibrator 1 is in contact with the ear cartilage. In other words, the state of contact of the vibrator 1 with the ear cartilage means a state in which there is no layer of gas at room temperature and pressure between the ear cartilage and the vibrator 1.

When the ear cartilage vibrates, the vibration causes the medium in contact with the ear cartilage, such as the air, to vibrate. The vibration of the medium then propagates to the medium surrounding it. In this way, the vibration of the ear cartilage propagates through the medium near the ear cartilage. The medium may be air, for example, or it may be a part of a living body, such as a finger or hand.

The vibration detector 2 detects the vibrations that have propagated through the medium in this manner. The vibrations that have propagated through the medium described above are vibrations that arise as a result of the vibration of the ear cartilage. Therefore, it can be said that the vibration detector 2 detects vibrations that arise as a result of the vibration of the ear cartilage caused by the vibrator 1.

Vibrations of the ear cartilage propagate through multiple types of media, such as air and parts of a living body, before reaching the vibration detector 2. Therefore, the vibrations detected by the vibration detector 2 contain information about the media on the propagation path of the vibration. For example, if an object that was on the propagation path moves and is no longer on the propagation path, the vibrations detected by the vibration detector 2 contain information about this change.

Therefore, if the position of an object on the vibration propagation path changes, the vibration detected by the vibration detector 2 also changes, and this change in vibration represents a change in the position or speed of the object on the propagation path. If we provide a mathematical explanation for the change in the vibration detected by the vibration detector 2, the change in the vibration detected by the vibration detector 2 means a change in the value of the propagation function of the vibration propagating from the ear cartilage to the vibration detector 2.

In this way, the change in vibration detected by the vibration detector 2 represents a change in the position or speed of an object on the vibration propagation path, so the movement of a moving object near the auricle is detected by detection by the vibration detector 2. Note that "Modulated Signal" in Figure 1 represents the fact that the excited vibration (i.e., the signal) of the vibrator 1 is modulated by the finger.

The vibration detector 2 outputs the detection results to a predetermined output destination. The predetermined output destination is, for example, the analysis device 3.

<Installation of vibrator 1 and vibration detector 2>
Here, we will explain examples of mounting techniques and more specific examples of mounting positions for the vibrator 1 and the vibration detector 2. The vibration detector 2 may be located in any position as long as it is in a position where it can detect vibrations resulting from vibration of the ear cartilage by the vibrator 1. In addition, the vibration detector 2 may be located in a position that is not approximately the same as the vibrator 1, but is in a position where it can detect vibrations resulting from vibration of the ear cartilage by the vibrator 1.

If the vibration detector 2 is located in approximately the same position as the vibrator 1, even if the vibration excited by the vibration of the ear cartilage changes due to the movement of the object, the vibration detector 2 will detect the vibration of the ear cartilage itself more strongly than the change. As a result, the accuracy of detecting the movement of the object will decrease. If the vibration detector 2 and the vibrator 1 are not located in approximately the same position, this decrease in accuracy is mitigated.

The vibrator 1 may be located at a forward position, for example, in front of the auricle, and the vibration detector 2 may be located at a rear position, for example, behind the auricle. Note that the forward position (forward position) and rear position (rear position) of the auricle are closer to the nose than the auricle, and the rear position is farther from the nose than the auricle. For example, in the X-axis of FIG. 2 described below, with the position of the auricle as the origin of the X-axis, a negative position on the X-axis is an example of a rear position, and a positive position on the X-axis is an example of a forward position.

For example, the vibrator 1 may be attached in a state where it is adhered to the ear cartilage by a substance that adheres one object to another, such as tape or adhesive. Note that tape and adhesive are not gas layers. Therefore, the state of the vibrator 1 adhered to the ear cartilage by tape or adhesive is an example of the state of contact of the vibrator 1 with the ear cartilage. In such a case, the vibration detector 2 may be attached to a part of the human body that can detect vibrations resulting from vibrations of the ear cartilage by a substance that adheres one object to another, such as tape or adhesive.

The position at which vibrations resulting from vibration of the ear cartilage can be detected depends on the strength of the vibrations excited in the ear cartilage by vibrator 1. The stronger the output strength of vibrator 1, the stronger the vibrations excited in the ear cartilage. The stronger the vibrations excited in the ear cartilage, the easier it is for vibration detector 2 to detect the vibrations. Therefore, the stronger the vibrations excited in the ear cartilage, the greater the distance between vibrator 1 and vibration detector 2 can be.

Therefore, a position where vibrations resulting from vibrations of the ear cartilage can be detected is, for example, near the auricle. Near the auricle means a position closer to the auricle than the tip of the nose, the top of the head, the shoulders, or the clavicle.

For example, the vibrator 1 may be attached to a headgear such as a hat that can be worn so that the vibrator 1 is in contact with the ear cartilage. The hat may be, for example, a knitted hat. The vibration detector 2 may be provided on the headgear so that it is located near the auricle when the headgear is worn so that the vibrator 1 is in contact with the ear cartilage. Note that just because the vibrator 1 is attached to the headgear, the vibration detector 2 does not necessarily have to be attached to the headgear as well; it may be attached to the position of the attachment target using a substance that bonds objects together, such as tape or adhesive.

For example, the vibrator 1 may be attached to an ear hook having a portion to which the vibrator 1 can be attached, and the ear hook may be shaped so that the vibrator 1 comes into contact with the ear cartilage when worn when the vibrator 1 is attached to the portion.

In such a case, the vibration detector 2 may be attached to a position on the ear hook that is different from the mounting position of the vibrator 1. The vibration detector 2 does not need to be attached to a position on the ear hook that is different from the mounting position of the vibrator 1, and may be attached to the mounting target position using a substance that bonds objects together, such as tape or adhesive.

For example, the vibrator 1 and the vibration detector 2 may be provided in an instrument such as glasses. In such a case, the instrument including the vibrator 1 and the vibration detector 2 is provided in the detection system 100. Below, an example of an instrument including the vibrator 1 and the vibration detector 2 will be described with reference to FIG. 2, taking the instrument as glasses.

Figure 2 is a diagram showing an example of eyeglasses 10 equipped with a vibrator 1 and a vibration detector 2 according to an embodiment. In many cases, eyeglasses are shaped so that the left and right sides (i.e., the structure on the right eye side and the structure on the left eye side) are approximately symmetrical with respect to the bridge. In the example of Figure 2, for ease of explanation, only the structure on the right eye side of eyeglasses 10 is shown.

The structure of the right eye side of the glasses 10 may be the same as that shown in FIG. 2, or if the left eye side is provided with a vibrator 1 and a vibration detector 2, the structure of the right eye side does not have to include a vibrator 1 and a vibration detector 2. Also, if the structure of the right eye side includes a vibrator 1 and a vibration detector 2, the left eye side may or may not include a vibrator 1 and a vibration detector 2.

In recent years, many glasses, such as smart glasses, are attached to only one of the right and left eyes. The glasses 10 also do not necessarily have to have a structure for the right eye and a structure for the left eye, and may only have one of them.

In the example of FIG. 2, the glasses 10 are equipped with glasses lenses 101. The glasses lenses 101 of the glasses 10 may or may not have prescription lenses. Glasses 10 equipped with non-prescription glasses lenses 101 are so-called fashion glasses.

As shown in FIG. 2, in the eyeglasses 10, the vibrator 1 is attached (i.e., located) on the inside of the end piece 102 (i.e., the side of the end piece 102 closer to the base of the ear). Therefore, when the eyeglasses 10 are worn, the vibrator 1 comes into contact with the ear cartilage. As shown in FIG. 2, in the eyeglasses 10, the vibration detector 2 is attached to the temple 103. Note that the inside of the end piece 102 is the side of the end piece 102 closer to the base of the ear as described above, in other words, the end piece 102 is oriented to come into contact with the back of the auricle.

The glasses 10 may be attached to headwear. As described above, the glasses 10 are an example of an instrument equipped with the vibrator 1 and the vibration detector 2, but headwear equipped with the vibrator 1 and the vibration detector 2 is also an example of an instrument equipped with the vibrator 1 and the vibration detector 2, and an ear hook equipped with the vibrator 1 and the vibration detector 2 is also an example of an instrument equipped with the vibrator 1 and the vibration detector 2.

The ear hook equipped with the vibrator 1 and the vibration detector 2 may be, for example, the glasses 10 of FIG. 2 that does not have the eyeglass lens 101. The glasses 10 that do not have the eyeglass lens 101 may have only a right eye structure, only a left eye structure, or both a right eye and a left eye structure.

In addition, the glasses 10 that do not have the eyeglass lenses 101 only need to have the vibrator 1, the vibration detector 2, the end pieces 102, and the temples 103, and do not need to have a support for supporting the eyeglass lenses, such as a rim or clings.

Note that FIG. 2 shows the X-axis. The X-axis is a coordinate axis with origin O at the position of the auricle, and is a coordinate axis along the temple 103 of the glasses. As described above, a positive position on the X-axis is an example of a forward position, and a negative position on the X-axis is an example of a rearward position. Therefore, the arrangement of the vibrator 1 and vibration detector 2 in the glasses 10 shown in FIG. 2 is an example of an arrangement in which the vibration detector 2 is located at a forward position in front of the auricle, and the vibrator 1 is located at a rearward position behind the auricle.

Returning to the explanation of FIG. 1, the analysis device 3 includes a control unit 31 including a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus.

The control unit 31 executes an analysis process on the detection result of the vibration detector 2. The analysis process is a process predetermined by a user or the like according to subsequent processing, and is a process of converting the detection result of the vibration detector 2 into data in a format that can be processed by the subsequent processing. The analysis process is, for example, a process of estimating the position of the detection target, such as a finger or hand, according to the detection result of the vibration detector 2. If the position can be estimated, the movement of the detection target can be detected by performing position estimation at a predetermined time interval.

The process of estimating the position of the detection target by analysis processing is, for example, processing of executing a trained machine learning model that estimates the position of the detection target based on the detection result of the vibration detector 2. The trained machine learning model that estimates the position of the detection target based on the detection result of the vibration detector 2 is, for example, the result of learning using the detection result of the vibration detector 2 and the position of the detection target, which is executed on the mathematical model of the learning target until a learning end condition is satisfied. The mathematical model of the learning target is a mathematical model that estimates the position of the detection target based on the detection result of the vibration detector 2.

The learning end condition is a predetermined condition regarding the end of learning. The learning end condition may be any condition regarding the end of learning, and may be, for example, a condition that the mathematical model to be learned has been updated a predetermined number of times, or a condition that the change in the mathematical model to be learned due to the update is smaller than a predetermined change.

The learning technology of the mathematical model of the learning target may be any machine learning technology that improves the accuracy of estimating the position of the detection target based on the detection results of the vibration detector 2. Therefore, the learning technology of the mathematical model of the learning target may be, for example, a support vector machine.

The technology for learning the mathematical model to be learned may be, for example, a technology using a generative adversarial network, a technology using an attention mechanism, or a technology using a diffusion model (extended model). However, the use of a support vector machine is preferable in that it requires a smaller amount of training data compared to other machine learning technologies.

<Experimental Results>
Here, an example of the experimental results of an experiment using the detection system 100 is shown. In the experiment, a chirp sine wave of 20 to 44 kHz with a sampling frequency of 96 kHz, 8192 samples, and 24 bits was used as the transmission signal. The transmission signal is an example of vibration excited by the vibrator 1. In the experiment, the glasses 10 of Fig. 2 were used, and the vibrator 1 and vibration detector 2 were provided only on the left eye side. In the experiment, the vibrator 1 was a bone conduction speaker.

In the experiment, the position of the finger on the auricle was changed to obtain changes in the frequency characteristics of the signal detected by the vibration detector 2. Therefore, 100 measurements were made continuously in the experiment. One measurement is defined as a measurement of a signal using one repeatedly transmitted chirp sine wave. Therefore, 100 measurements obtained information on frequency characteristics at 100 points. The significance of making continuous measurements is that the effects of noise can be reduced. The time length required for one frequency analysis was 8192/96000 (approximately 0.0853) seconds, which corresponds to 8192 samples of the transmitted signal. Note that, although 100 measurements were made in the illustrated experiment, it is not necessary to make 100 measurements, and even if measurements are made fewer than 100 times, qualitatively similar results to those in the examples of Figures 3 to 7 can be obtained.

Figure 3 shows a first example of experimental results in the embodiment. Figure 3 shows an example of the frequency characteristics of the response waveform for the 45th sample (i.e., 3.84 seconds after the start of measurement). The response waveform is the waveform of the signal detected by the vibration detector 2.

Figure 3 shows five images showing finger positions (each image in Figure 3(a)) and the frequency characteristics of the response waveform obtained in the scene shown by each image in Figure 3(a) (results shown in Figure 3(b)). The finger positions in the experiment were five positions, from "Position 1" to "Position 5", as shown in Figure 3(a).

In the graph showing the frequency characteristics of the response waveform shown in Figure 3(b), the horizontal axis indicates frequency and the vertical axis indicates the amplitude of the response waveform. Note that the results corresponding to "Position 1" to "Position 5" shown in Figure 3(b) are the results obtained at each finger position after the finger position was changed from "Position 1" to "Position 2", from "Position 2" to "Position 3", from "Position 3" to "Position 4", and from "Position 4" to "Position 5" after the chirp sine wave was transmitted 15 times (i.e., 1.28 seconds).

The results in FIG. 3 show that the frequency characteristics of the response waveform change depending on the finger position. Note that the frequency characteristics of the response waveform are an example of information obtained from the detection results of the vibration detector 2, and the process of obtaining the frequency characteristics of the response waveform may be included in the analysis process by the analysis device 3. Therefore, the analysis process may include, for example, a process of performing a Fourier transform.

In such a case, the analysis process may include, after performing a Fourier transform, a process of estimating the position of a detection target such as a finger or hand using the result of the Fourier transform. The process of estimating the position of a detection target such as a finger or hand using the result of the Fourier transform (i.e., the frequency characteristics of the response waveform) may be performed using, for example, a trained mathematical model.

In such a case, the trained mathematical model is a mathematical model that estimates the position of the detection target based on the frequency characteristics of the response waveform. Therefore, in training the mathematical model that estimates the position of the detection target based on the frequency characteristics of the response waveform, training is performed using the frequency characteristics of the response waveform and information indicating the position of the detection target.

Figure 4 shows a second example of experimental results in the embodiment. Figure 4 shows five images showing the hand position (each image in (a) of Figure 4), the frequency characteristics of the response waveform obtained in the scene shown in each image in (a) of Figure 4 (results shown in (b) of Figure 4), and the cross-correlation obtained in the scene shown in each image in (a) of Figure 4 (results shown in (c) of Figure 4). Specifically, the cross-correlation was the correlation between the transmitted signal and the signal detected by the vibration detector 2.

The hand positions in the experiment were five, from "Position 1" to "Position 5," as shown in Figure 4(a).

In the graph showing the frequency characteristics of the response waveform shown in Figure 4(b), the horizontal axis indicates frequency and the vertical axis indicates the amplitude of the response waveform. Note that the results corresponding to "Position 1" to "Position 5" shown in Figure 4(b) were obtained at each hand position after the hand position was changed from "Position 1" to "Position 2", from "Position 2" to "Position 3", from "Position 3" to "Position 4", and from "Position 4" to "Position 5" after the chirp sine wave was transmitted 17 times (i.e., 1.28 seconds).

In the graph showing the cross-correlation shown in FIG. 4(c), the horizontal axis indicates time and the vertical axis indicates cross-correlation. Note that the results corresponding to "Position 1" to "Position 5" shown in FIG. 4(c) were obtained at each hand position after changing the hand position from "Position 1" to "Position 2", "Position 2" to "Position 3", "Position 3" to "Position 4", and "Position 4" to "Position 5" every time the chirp sine wave was transmitted 17 times (i.e., 1.28 seconds). In addition, for all cross-correlation, the difference between the 16th sample and all other frames was used in order. This resulted in a result that suppressed the influence of direct waves (i.e., vibrations excited by the vibrator 1 that were detected by the vibration detector 2 without passing through the vibration of the pinna).

The results in Figure 4 show that cross-correlation changes depending on the hand position. The results in Figure 4 also show that while cross-correlation shows a clear shift in the peak in response to changes in hand position, the frequency characteristics can show a lot of noise. The reason that the frequency characteristics show a lot of noise is due to the influence of interference from reflections from the hand. Therefore, when estimating changes in the position of a hand approaching the ear, estimation using cross-correlation rather than frequency characteristics provides higher accuracy.

Note that cross-correlation is an example of information obtained from the detection results of the vibration detector 2, and the process of obtaining the cross-correlation may be included in the analysis process by the analysis device 3. In such a case, the analysis process may execute a process of estimating the position of a detection target such as a finger or hand using the obtained cross-correlation after executing the process of obtaining the cross-correlation. The process of estimating the position of a detection target such as a finger or hand using the cross-correlation may be performed using, for example, a trained mathematical model.

In such a case, the trained mathematical model is a mathematical model that estimates the position of the detection target based on cross-correlation. Therefore, when training a mathematical model that estimates the position of the detection target based on cross-correlation, training is performed using cross-correlation and information indicating the position of the detection target.

In order to obtain the cross-correlation, information on the transmitted signal is required, and this information is stored in advance in a predetermined storage device such as the storage unit 33 described below, and is read out when obtaining the cross-correlation.

Figure 5 shows a third example of experimental results in an embodiment. Figure 5 shows two images showing deformation of the pinna (the images in the upper row of each of (a) and (b) in Figure 5) and the frequency characteristics of the response waveforms obtained in the scenes shown by the images in the upper row of Figure 5 (the images in the lower row of each of (a) and (b) in Figure 5). In the graphs showing the frequency characteristics of the response waveforms shown in (a) and (b) in Figure 5, the horizontal axis indicates frequency and the vertical axis indicates the amplitude of the response waveform. The results in Figure 5 show that the frequency characteristics of the response waveform change with deformation of the pinna.

In such a case, the analysis process may perform a process of estimating the change in the shape of the pinna using the result of the Fourier transform after performing the Fourier transform. The process of estimating the change in the shape of the pinna using the result of the Fourier transform (i.e., the frequency characteristics of the response waveform) may be performed using, for example, a trained mathematical model.

In such a case, the trained mathematical model is a mathematical model that estimates changes in the shape of the pinna based on the frequency characteristics of the response waveform. Therefore, in training the mathematical model that estimates changes in the shape of the pinna based on the frequency characteristics of the response waveform, training is performed using the frequency characteristics of the response waveform and information indicating changes in the shape of the pinna.

Figure 6 is a diagram showing a fourth example of experimental results in the embodiment. An example of a confusion matrix result for identifying pinna deformation is shown. More specifically, an example of a confusion matrix result showing the accuracy rate of the result of executing an analysis process for estimating pinna deformation based on the detection result of the vibration detector 2 is shown.

The "True label" in Figure 6 indicates the correct answer; that is, it indicates the actual deformation. The "Predicted label" in Figure 6 indicates the result of the estimation by the analysis process. Therefore, the diagonal elements of the confusion matrix indicate the accuracy rate of the estimation for each deformation of the pinna. In the experiment to obtain the results in Figure 6, four types of deformation of the pinna were performed: "squeeze", "pull-up", "pull-down", and "fold". The results in Figure 6 show that high accuracy estimation was performed for all four types.

The analysis process used in the experiment to obtain the results in Figure 6 specifically involved estimating the type of deformation of the pinna based on frequency response, and was performed using a support vector machine, which is a machine learning technique.

Figure 7 is a diagram showing a fifth example of experimental results in an embodiment. Figure 5 shows an example of the results of an analysis process in which a process for estimating the finger position based on the detection result of the vibration detector 2 is executed. In Figure 7, the "estimated" result shows the output of a support vector machine. In Figure 7, the "filtered" result shows the result of an estimation smoothed by a Kalman filter. In Figure 7, "ground truth" shows the true value (i.e., the finger position in the example of Figure 7).

The x-axis in Figure 7 is a coordinate axis whose positive direction indicates a specific direction in real space, and the y-axis is a coordinate axis whose positive direction indicates a direction perpendicular to the x-axis in real space. The results in Figure 7 show that the estimation results obtained by the analysis process are highly accurate.

The analysis process used in the experiment to obtain the results in Figure 7 specifically involved estimating coordinates based on frequency response, a process obtained using a support vector machine, which is a machine learning technique.

<Examples of learning situations>
Here, in the case where a trained machine learning model is used in the analysis process, an example of a technique for acquiring true values (correct data) used for learning will be described using an example where the detection target is a finger. The true values are obtained by photographing a finger with a marker such as a red sticker using a photographing device such as a camera. In order to prepare learning data, the detection results of the vibration detector 2 at that time are also acquired together with the acquisition of the true values. A pair of the obtained detection results of the vibration detector 2 and the photographing results by the photographing device is used as learning data for learning the mathematical model of the learning target.

Note that it is not necessary for the fingers to have markers. When there are no markers, the techniques described in the following references 1 to 3 may be used to estimate the finger position from the photographed results. Reference 2 is a technique that uses an inertial sensor, and reference 3 is a technique that uses myoelectric potential.

Reference 1: Zhang, Fan, et al. "Mediapipe hands: On-device real-time hand tracking." arXiv preprint arXiv:2006.10214(2020).
Reference 2: Ting Kwok Chan, et al. "Robust hand gesture input using computer vision, inertial measurement unit (IMU) and flex sensors." 2018 IEEE International Conference on Mechatronics, Robotics and Automation (ICMRA). IEEE, 2018.
Reference 3: Yilin Liu, Shijia Zhang, and Mahanth Gowda. "NeuroPose: 3D hand pose tracking using EMG wearables." Proceedings of the Web Conference 2021. 2021.

The true value may be obtained in any manner as long as information indicating the position of the detection target can be obtained, and may be obtained, for example, by a technique for detecting the position, such as three-dimensional measurement.

<Example of Hardware Configuration of Analysis Device 3>
8 is a diagram showing an example of a hardware configuration of the analysis device 3 in the embodiment. The analysis device 3 includes a control unit 31 including a processor 91 and a memory 92 connected by a bus, and executes a program. The analysis device 3 functions as a device including the control unit 31, an interface unit 32, and a storage unit 33 by executing the program.

More specifically, the processor 91 reads out a program stored in the storage unit 33 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the analysis device 3 functions as a device including a control unit 31, an interface unit 32, and a storage unit 33.

The control unit 31 controls the operation of each functional unit of the analysis device 3. The control unit 31 obtains the detection result of the vibration detector 2, for example, via the interface unit 32. The control unit 31 executes, for example, an analysis process. The control unit 31 obtains, for example, information stored in the memory unit 33. The process of obtaining the information stored in the memory unit 33 is, specifically, a read process.

When a trained mathematical model is used in the analysis process, the control unit 31 may execute a learning process to obtain the trained mathematical model. In such a case, the control unit 31 acquires the training data via, for example, the interface unit 32.

The interface unit 32 includes a communication interface for connecting the analysis device 3 to an external device. The interface unit 32 communicates with the external device via wired or wireless communication. The external device is, for example, the vibration detector 2. In such a case, the interface unit 32 obtains the detection result of the vibration detector 2 by communicating with the vibration detector 2.

The interface unit 32 may be configured to include input devices such as a mouse, keyboard, and touch panel. The interface unit 32 may be configured as an interface that connects these input devices to the analysis device 3. In this way, the input device of the interface unit 32 accepts input of various information to the analysis device 3 via a wired or wireless connection. Note that the various data acquired by the interface unit 32 described above does not necessarily have to be acquired by the communication interface of the interface unit 32, and may be input to the input device of the interface unit 32.

The interface unit 32 outputs, for example, various types of information. The interface unit 32 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 32 may be configured as an interface that connects these display devices to the analysis device 3. The interface unit 32 outputs, for example, information input to a communication interface or an input device of the interface unit 32.

The storage unit 33 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 33 stores various information related to the analysis device 3. The storage unit 33 stores, for example, various information generated by the operation of the control unit 31. The storage unit 33 stores, for example, information acquired by the interface unit 32.

Figure 9 is a flow chart showing an example of the flow of processing executed by the detection system 100 in an embodiment. The vibrator 1 excites vibrations (step S101). The excited vibrations of the vibrator 1 cause the ear cartilage to vibrate, and the vibrations vibrate the air. The air vibrations reach the vibration detector 2 while being influenced by elastic bodies that are located on the way there.

Next, the vibration detector 2 detects vibration (step S102). Next, the control unit 31 executes an analysis process on the result of the detection by the vibration detector 2 (step S103). Next, the control unit 31 outputs the result of the analysis process to a predetermined output destination (step S104).

The specified output destination may be, for example, the display device of the interface unit 32. In such a case, the control unit 31 controls the display device of the interface unit 32 to display the results of the analysis process. The specified output destination may be, for example, the storage unit 33. In such a case, the control unit 31 records the results of the analysis process in the storage unit 33.

The detection system 100 of the embodiment configured in this manner includes a vibrator 1 and a vibration detector 2, and is capable of detecting a response waveform according to the position of the detection target. Furthermore, since the vibrator 1 is in contact with the ear cartilage, the vibrator 1 does not block the ear canal. Therefore, detection using the detection system 100 does not impose a burden on the user that would be caused by blocking the ear canal, and the detection system 100 can reduce the burden on the user required to detect the movement of a moving object near the auricle.

The detection system 100 of the embodiment configured in this manner also includes a vibrator 1 that is in contact with the ear cartilage. Because the vibrator 1 is in contact with the ear cartilage, it can excite vibrations in the ear cartilage more efficiently than when it is not in contact. As described above, vibrations in the ear cartilage are propagated through an elastic body such as air. Therefore, it can be said that the ear cartilage acts as a megaphone. Therefore, when the vibrator 1 is in contact with the ear cartilage, the detection accuracy of the vibration detector 2 is higher than when it is not in contact.

When the vibrator 1 is located at a forward position in front of the auricle, and the vibration detector 2 is located at a rear position behind the auricle, the vibrator 1 can excite the vibration of the ear cartilage with higher efficiency and the vibration detector 2 can detect the vibration with higher sensitivity, compared to other arrangements such as an arrangement in which the vibrator 1 is located at a rear position and the vibration detector 2 is located at a forward position. This is because when the vibrator 1 is located at a forward position, the contact area with the ear cartilage is larger than in other cases. Also, since vibrations of the auricle tend to propagate strongly toward the front side of the auricle, it is preferable to have the vibration detector 2 located at a rear position compared to other arrangements.

The detection system 100 of the embodiment configured in this manner also includes a vibrator 1. The vibrator 1 may have an ultrasonic frequency, an audible frequency, or a low frequency of 0 Hz to 20 Hz, but in any case, excites vibrations. Such vibrations have a wavelength sufficiently longer than that of a hair. Therefore, compared to, for example, using a device that emits light instead of the vibrator 1, the detection system 100 suppresses degradation of sensing accuracy caused by hair.

The detection system 100 of the embodiment configured in this manner also makes it possible to estimate the movement of the user's fingers or the like near the auricle. Therefore, by using the detection system 100, the user can intuitively operate the device to be operated.

In addition, by using the detection system 100 of the embodiment configured in this manner, the user can operate the device to be operated by moving a finger or the like near the auricle, without touching a button or the like. Therefore, compared to technologies that require contact with small objects such as touching a button, the detection system 100 has the effect of expanding the input surface.

Furthermore, devices originally used for calls or music playback can be used as the vibrator 1. This makes it possible to miniaturize the device and reduce costs compared to using technologies that are rarely used for calls or music playback, such as lasers.

In addition, if a photographing device such as a camera is used to detect the detection target, the photographing device needs to be installed in a position where it can photograph the detection target. Therefore, it becomes necessary to use a device such as a selfie stick that places a camera away from the detection target. However, if a vibrator 1 and a vibration detector 2 are used instead of a photographing device, this is not necessary because the vibrations generated by vibrator 1 propagate to the back side of the obstruction due to diffraction.

(Modification)
The analysis device 3 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each process executed by the control unit 31 may be distributed and executed by the plurality of information processing devices.

When the vibrator 1 and the vibration detector 2 are provided in the glasses 10, the analysis device 3 may be provided in part or in whole in the glasses 10. The attachment point of the glasses 10 in which the analysis device 3 is provided in part or in whole may be, for example, the temple 103. In other words, the analysis device 3 may be provided in part or in whole in the temple 103, for example.

Note that the detection system 100 does not necessarily need to include an analysis device 3.

All or part of the functions of the control unit 31, the interface unit 32, and the storage unit 33 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the present invention.

100...detection system, 1...vibrator, 2...vibration detector, 3...analysis device, 10...glasses, 101...glasses lens, 102...eyeglasses, 103...temple, 31...control unit, 32...interface unit, 33...storage unit, 91...processor, 92...memory

Claims (6)

a vibrator that contacts the ear cartilage and vibrates the ear cartilage;
a vibration detector for detecting vibrations resulting from vibration of the ear cartilage;
A detection system comprising: The vibration detector is located at a front position, which is a position in front of the pinna, and the vibrator is located at a rear position, which is a position behind the pinna.
The detection system of claim 1 . The frequency of the vibration excited by the vibrator is an ultrasonic frequency.
The detection system of claim 1 . 1. A detection system comprising eyeglasses,
a vibrator that contacts the ear cartilage and vibrates the ear cartilage;
a vibration detector for detecting vibrations resulting from vibration of the ear cartilage;
Equipped with
the vibrator is located inside the end piece of the eyeglasses;
The vibration detector is located at the temple of the eyeglasses.
Detection system. A detection method performed by a detection system including a vibrator that contacts an ear cartilage and vibrates the ear cartilage, and a vibration detector that detects vibrations resulting from the vibration of the ear cartilage, the detection method comprising:
an excitation step in which the vibrator excites vibration;
a detection step in which the vibration detector detects vibration resulting from the vibration;
The detection method comprising: A program for causing a computer to function as the detection system according to any one of claims 1 to 4.

Images