JP2018093503A - Sound content reproduction earphone, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an earphone unit for providing appropriate information and contents by obtaining a present state of a user.SOLUTION: An earphone 101 includes sensors such as a blood flow sensor, an acceleration sensor, a gyro sensor, and a geomagnetic sensor, so as to obtain a present state of a user. The system allows the blood flow sensor to estimate a psychological state of a user during reproduction of contents, so as to analyze correlation between contents and a psychological state and to provide a personalization function by preservation, and adjusts contents to be reproduced in response to a direction of a head of a user obtained by the acceleration sensor and the gyro sensor.SELECTED DRAWING: Figure 2

Description

The present invention relates to an earphone that can provide optimal information according to a user's situation, and more particularly, to an autonomous assistant system provided by an earphone that utilizes various sensor information and enables optimization of playback content.

Among the wearable devices, a field called “Hearable” is attracting attention, especially based on a user interface centered on sound from earphones and headphones. “Hearables” have various advantages in today's technical and social environments.

Smart glasses, which are spectacle-type wearable devices, are not yet accepted socially at this time, and have great psychological resistance to daily use. In addition, there is a problem that the amount of information that can be provided to the user is limited because the viewing angle is narrow. In addition, smart watches, which are wristwatch-type wearable devices, also have a problem that the amount of information that can be provided to the user is limited due to the small screen.

On the other hand, “Hearable” using earphones or headphones has an advantage in that a large amount of information by voice can be continuously provided to the user. In addition, since the earphone is a device that is worn all the time, it is possible to provide users with the necessary information instantly without taking out the smartphone or watching the smartwatch. is there. In addition, wearing earphones while going out for music appreciation is socially acceptable in many cases, and there is little psychological resistance for use by both the person himself and those around him. Furthermore, the earphone is not affected by the screen size, and the main body can be made extremely small, and can be used more naturally.

However, until now, the devices that can be used for hearing have mainly been provided with voice information, and it has not been sufficient to obtain feedback on the user's condition. It is technically possible to measure blood flow and pulse wave with a small sensor (for example, Non-Patent Document 1) and estimate a psychological state from a pulse wave waveform (for example, Non-Patent Document 2). However, there are no hearable devices that can make full use of these and provide an assistant function according to the user's situation.

“Development of small laser blood flow sensor” (PIONEER R & D (Vol.21, No.1 / 2012)) (http://pioneer.jp/corp/crdl_design/crdl/rd/pdf/21-1-5.pdf ) "Estimation and visualization of physical and psychological states based on pulse wave signals" (Keiichi Horio, Kageri Lee) https://www.jstage.jst.go.jp/article/fss/25/0/25_0_116/ _article / -char / ja /

Provide an earphone device that can customize the provided content according to the user's situation.

The present invention relates to a means for measuring a user's pulse wave, a means for grasping an identifier of the content being reproduced, information relating to the user's pulse wave measured during the content reproduction, and an identifier of the content being reproduced. The above-described problems are solved by providing an earphone that is stored in association with each other or stored in an external device.

The present invention also includes a step of measuring a user's pulse wave using a sensor provided in the earphone, a step of grasping an identifier of content being played back by the earphone, and information on the measured pulse wave The above-mentioned problem is solved by providing a method of reproducing audio content executed by a computer comprising the step of associating and storing an identifier of the content.

Further, the present invention provides a command for measuring a user's pulse wave using a sensor provided in the earphone, a command for grasping an identifier of a content being played back by the earphone, and information on the measured pulse wave And an audio content reproduction program that causes a computer to execute an instruction to store the content identifier in association with each other.

There is provided an earphone device that can customize provided content and information according to a user's situation.

1 is an overall view of an example of an information providing system including an embodiment of an earphone according to the present invention. It is an example of a functional structure of the Example of the earphone which concerns on this invention. It is an example of a functional structure of the portable information device used with the Example of the earphone which concerns on this invention. It is an example of the functional structure of the server used with the Example of the earphone which concerns on this invention. It is a figure showing the 1st Example using the earphone which concerns on this invention. It is a figure showing the content of the data preserve | saved in the 1st Example using the earphone which concerns on this invention. It is a figure showing the 2nd Example using the earphone which concerns on this invention. It is a figure showing the 3rd Example using the earphone which concerns on this invention. It is a figure showing the 4th Example using the earphone which concerns on this invention. It is a flowchart which concerns on the 5th Example using the earphone which concerns on this invention.

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows an overall view of an example of a common information providing system of an embodiment of an earphone according to the present invention. The earphone (101) typically communicates with a portable information device (102) such as a smartphone via a short-range wireless communication technology such as Bluetooth (registered trademark) or a normal cable. In general, a shape in which the left and right are separated is often referred to as an earphone, and a shape in which the left and right are coupled is often referred to as a headphone. However, in the present specification, the two are collectively referred to as an earphone. The earphone (101) communicates with a server (103) typically placed on a cloud via a network such as the Internet. At the technical level at the time of filing of the present application, it is preferable that each function is distributed between the earphone (101), the portable information device (102), and the server (103). With the miniaturization and high performance, functions may be concentrated on the earphone (101) side. Further, the earphone (101) is connected directly to the information providing device (104) installed in advance (typically wirelessly) without using the portable information device (102), and receives information by voice. May be used.

FIG. 2 shows a functional configuration of an embodiment of the earphone (101) according to the present invention. An earphone (101) according to the present invention includes a microphone (202) for a telephone call or a command input by a user in addition to a speaker (201) for generating sound that is an original function of the earphone. It is desirable. Further, it is desirable to provide a control button (203) for the user to give a command. The microphone and the control button may be integrated with the earphone main body, or may be a case separate from the earphone (101) main body and designed to communicate with the earphone (101) main body in a wired or wireless manner. The earphone (101) preferably has a wireless connection function (204) for connection with the portable information device (102) or the information providing device (104). The wireless connection function (204) may use Bluetooth (registered trademark) or BLE (Bluetooth Low-Energy) technology. As an alternative configuration, the earphone (101) may be connected to the portable information device (102) by wire. The earphone (101) preferably includes a blood flow sensor (205) for measuring a pulse wave. It is desirable that the blood flow sensor is provided at a portion to be crimped to the ear hole or the earlobe of the earphone (101), or can be crimped and fixed to the user's earlobe with a clip or the like. By doing so, it is possible to accurately measure pulse waves that are not easily disturbed by noise. In addition, the earphone includes a gyro sensor (206) for measuring the orientation of the user's head, a geomagnetic sensor (compass) (207), and an acceleration sensor (208) for measuring the user's movement. It is desirable to have it. These sensors are preferably fixed in the body of the earphone (101). By doing so, it is possible to accurately measure the position and direction of the head of the user of the earphone. As an alternative configuration, the earphone (101) is provided by connecting the above sensor to an object that has a fixed positional relationship with the direction of the user's head, such as glasses or a hat, or by fixing it with a clip or the like and by wired or wireless connection The sensor information may be provided to the portable information device (102) or the information providing device (104). The earphone (101) may include a proximity sensor (209) for detecting whether the earphone (101) is worn by the user. Furthermore, you may provide other types of sensors, such as an electroencephalogram sensor and GPS.

FIG. 3 shows a functional configuration of an example of the portable information device (102) used together with the embodiment of the earphone (101) according to the present invention. The portable information device operates an application program (301) for providing content such as music or a telephone call to the earphone or providing a navigation service using map information. The application program (301) can switch the content to be played back by voice input by the user's microphone (202), operation of the control button (203), a command from the server (103), or logic inside the program. . In addition, the application program (301) adjusts the content playback method and adjusts the position in the navigation service based on the earphone user's head position and voice commands ascertained from the sensor information of the earphone (101). Provides functions such as The portable information device (301) also includes a sensor information acquisition function (302) for obtaining sensor information of the earphone (101) wirelessly or via a wire. Furthermore, an earphone communication function (303) for transmitting sound to the earphone (101) and a server communication function (typically via the Internet) for exchanging information with the server (103) ( 304).

FIG. 4 shows an example of the functional configuration of the server (103) used together with the embodiment of the earphone according to the present invention. The server (103) is typically managed on the cloud. The server analyzes the sensor information of the earphone (101) and provides a function that provides the content that best matches the current situation of the earphone user (or instructs the portable information device to provide such content). The server program (401) to be provided is run. In addition, it is desirable that the past sensor information of the earphone user and the reproduced content are stored in the database (402) in association with each other and used for analysis. In addition, a history of sensor information of other earphone users may be maintained and used for analysis. Further, the server (103) includes a mobile information device communication function (403) for receiving sensor information and the like from the mobile information device (102) and transmitting a playback instruction for the audio content itself or the specific content.

The function distribution form between the earphone (101), the portable information terminal (102), and the server (103) shown above has been optimized according to today's technical level, and more in the future. May be integrated on the earphone (101) side. For example, all the functions illustrated as being realized on the portable information terminal (102) may be realized on the earphone (101), and the earphone (101) and the server (103) may directly communicate with each other. Similarly, more functions except for the sensor equipped with the earphone may be concentrated on the server (103) side. Further, in the above, all or a part of the sensors provided on the earphone (101) may be provided in another wearable terminal worn by the user, and the sensor information may be integrated in the portable information terminal (102). .

Hereinafter, a first embodiment of the present invention will be described. The present embodiment is characterized in that a blood flow sensor (205) built in the earphone (101) is utilized. The blood flow sensor used today can measure not only the heart rate but also the waveform of the pulse wave in more detail, and it is technical to estimate the user's psychological state based on the waveform of the pulse wave Is possible. Some smartwatches and watch-shaped fitness wearable devices are equipped with a blood flow sensor, but in this example, the earphone (101) is equipped with a blood flow sensor (205), making it resistant to noise. Is advantageous (because earphones are essentially used in close contact with the ear canal or earlobe). In addition, since the earphone is necessary anyway when reproducing the audio content, this embodiment has an advantage in that it is not necessary to use another device in adjusting the audio content using the blood flow sensor data. .

FIG. 5 shows an outline of a method for analyzing the correlation between audio content (typically music) and a user's pulse wave. The content to be played back and the user's pulse wave information at that time are stored in association with each other at regular intervals. It is desirable to store the pulse wave information after extracting the feature points. The identifier of the music of the content to be played back may be the target of storage / analysis, or the feature points of the sound waveform may be extracted and stored / analyzed. By doing so, for example, it is possible to analyze a correlation such that the user enters a specific psychological state when a specific music (or a music having a specific characteristic) is played. This data storage and correlation analysis processing may be performed on the portable information device (102), but it can analyze a large amount of data, divert the data when the user uses another portable information device (102), and the user It is desirable to collect the data on the server (103) from the viewpoint of realizing a cross-sectional analysis using data of other persons.

FIG. 6 shows a storage format of correlation data between the user's psychological state and the music identifier (music ID) stored on the database (402) on the server (103). For example, when the user calms down when listening to a specific piece of music, the user wants to calm down by using voice input via the microphone (202) of the earphone (101) or operating the portable information device. The music (or similar music) that has been found by analysis to calm the person's psychology can be reproduced.

The user's psychological state during content reproduction may be grasped by the blood flow sensor (205), and the content to be reproduced may be selected and the reproduction volume / sound quality may be adjusted. For example, if it is clear from this history information that the user's spirit is concentrated when playing a specific song, the song is automatically played when the user requests the music that he wants to calm down, or It is also possible to recommend playback.

In addition, the earphone (101) is equipped with an electroencephalogram sensor that analyzes and stores the correlation between the feature quantity of the electroencephalogram and the psychological state, and automatically plays back or recommends the optimal music according to the electroencephalogram waveform. You may do it.

Also, using other sensor information, for example, when the user's heart rate is high and the person is moving at a speed that runs, if there is a high possibility that a specific piece of music is being played, It can be estimated that this is a favorite song when the user is running. Once such learning is performed, it is assumed that the user is running at the same heart rate and speed from the next time onward, and the music is automatically played, or to the user. Can be recommended. Or, when the user instructs to run by operating the mobile information terminal (102), etc., the music that the user often plays automatically by running is automatically played or recommended to play. It may be.

As another example, when the heart rate is constant and the user is moving at a speed of 10 kilometers or more per hour, it may be determined that the user is moving by train and news information by voice instead of music may be provided. It is desirable that what kind of content is provided in what situation can be customized by the user. In addition to the user's pulse wave and moving speed, time, day of the week, weather, temperature, vibration obtained by an acceleration sensor, and the like may be input to the content selection algorithm. In the analysis, in addition to correlation analysis using a conventional statistical method, machine learning using a neural network may be used.

In addition to content selection based on the history of a specific user, statistical analysis such as affinity analysis may be performed by integrating the information including information on other users on the server. For example, content that many people often listen to while running may be recommended while another user is running. In addition, content that is often listened to while a user is a friend on the user SNS may be recommended during the run. User preference information may be stored in a database and managed on a server, so that content recommended to users who have similar preferences to themselves may be recommended to themselves.

In addition to content selection, the playback volume and sound quality of the content may be automatically adjusted. For example, if it is estimated that the earphone user is running based on the pulse and moving speed, the playback volume is increased. If it is estimated that the user is on the train, the treble is cut to prevent sound leakage. You may do it.

In the first embodiment, the blood flow sensor (205) may be provided in another device such as a smart watch or a smart glass.

FIG. 7 shows a second embodiment of the earphone according to the present invention. Since the earphone (101) according to the embodiment of the present invention includes the gyro sensor (206) and the geomagnetic sensor (207), it is possible to grasp the orientation of the user's head, and according to the orientation of the user's head. Content can be reproduced, or the content reproduction method can be adjusted. For example, in an audio guide at a museum, it is possible to play audio content related to paintings in which the user is facing at a high volume, and not to play overseas audio content that is not facing, or at a low volume It is. Playing at a low volume has the effect of attracting the attention of a user who has not seen it directly. For example, the audio information related to the picture on the right side of the user can be directed to the work intended for the user by reproducing the right ear side greatly. In this embodiment, the audio content may be provided directly from the information providing apparatus (104) installed in the facility to the earphone (101). This is just an example. When providing audio content related to an object, the earphone according to the present invention grasps the position of the user and the direction of the head so that sound is generated from the location of the object. The audio content can be provided with an image as if it were, and a natural feeling for the user can be provided. Many of today's smartphones are equipped with a gyro sensor and a geomagnetic sensor, but since the orientation of the smartphone does not always match the orientation of the user's head, the earphone according to the present invention does not care about the orientation of the smartphone. It is advantageous in that the audio content can be adjusted.

FIG. 8 shows a third embodiment of the earphone (101) according to the present invention. Similarly, as an example in an art museum, an example in which the audio content played back on the information providing device (104) can be switched according to the distance from the picture by grasping the distance of the user from the information providing device (104). It is. For example, it is possible to provide more natural communication by switching the content to “a painting over there” when the distance between the user and the painting is long, and “this painting” when the distance between the user and the painting is short. For example, if a BLE-based communication technology is used between the earphone (101) and the information providing device (104), it is technically possible to roughly know the distance between the two. This example is used in a case where an earphone (101) that has been set in advance to connect to an information providing device (104) is lent as a voice guide at an art museum (a case where a user-owned smartphone is not used). It is particularly effective.

FIG. 9 shows a fourth embodiment of the earphone (101) according to the present invention. In this embodiment, the application program (301) of the portable information device (102) provides navigation information. Navigation is preferably performed by voice. It is a well-known technology to provide GPS to smart phones, etc., grasp the current position, display navigation information on the screen, and perform voice route guidance, but it is sufficient in places where GPS electric fields are weak, such as indoors. There is a problem that accuracy cannot be achieved. Even when the current position information is corrected based on the map information stored in advance, if the road or passage is complicated, it may be corrected to another road. In such an earphone according to the present embodiment, the user can appropriately correct the GPS position information and grasp the accurate position by conveying the state of the outside world by voice. Compared to taking out a smartphone and performing correction with a finger, the extra steps are unnecessary and the direction of the user's head is known by the sensor, so accurate correction is possible. This is the advantage of the present invention.

In the example of FIG. 9, although the navigation application user is actually on the narrow street B, the navigation application misrecognizes that the user is on the narrow street C due to a GPS error. Here, when the user communicates through the microphone (202) with the voice “Gyudon store to the right front”, the navigation application grasps the content by voice recognition and compares it with the map information. The position can be corrected. In this case, since the earphone (101) is fixed to the user's head, the geomagnetic sensor (207) accurately knows the direction of the user's head, so the user is on the narrow street B, In addition, the navigation application can accurately detect that it is near the intersection with narrow street D.

In addition, navigation can be performed relative to the facility in front of you. For example, in response to the user's question, “There is a convenience store in front of you but where is the movie theater?”, The navigation application can respond relatively from the user's position, “500 meters ahead right”. It becomes. Even in this case, since the direction of the user's head is accurately known by the geomagnetic sensor (207), the user can be provided with an accurate direction. Furthermore, when the direction of the right front is pointed out, the direction being explained can be expressed more clearly by sounding a guide sound from the right side using stereophonic sound as in the second embodiment. This embodiment is particularly effective when a smart phone is put in a bag or the like and the route guidance is performed only by voice.

FIG. 10 shows a flowchart relating to the notification function of the application program (301) in the fifth embodiment of the earphone (101) according to the present invention. In the "Hearable" environment, various notifications issued by the portable information device (102) (for example, real-time information from various application programs (301) such as email reception) are not only displayed on the screen but also transmitted by voice. Is desirable. In this case, it is desirable to delete the notification after the application program (301) confirms that the user has heard the notification by voice. Otherwise, important notices may remain undelivered. When it is not possible to confirm that the voice notification has been heard by the user, it is desirable to notify the voice again or other means such as a screen display after a certain period. Whether the user has been notified by voice is whether the earphone (101) was worn by the user by the proximity sensor (209) and the volume adjustment of the portable information device (102) is zero. It can be judged on the condition that there is not. This embodiment, which can confirm that the notification by voice has been performed reliably, provides a remarkable effect that the frequency with which the user looks at the screen is reduced and the user can enjoy the comfort that is possible only by the hilarious.

Claims (3)

A means for measuring a user's pulse wave;
Means to know the identifier of the content being played,
An earphone comprising means for storing the pulse wave of the user who is playing the content and the identifier of the content being played back in association with each other, or storing them in an external device. Measuring a user's pulse wave using a sensor provided in the earphone;
Grasping the identifier of the content being played by the earphone;
An audio content reproduction method executed by a computer comprising the step of storing information relating to the measured pulse wave and the content identifier in association with each other. A command to measure a user's pulse wave using a sensor provided in the earphone;
A command for grasping the identifier of the content being played by the earphone;
An audio content reproduction program for causing a computer to execute information relating to the measured pulse wave and a command for storing the content identifier in association with each other.

Images