JP2023539874A - Computerized decision support tools and medical devices for respiratory disease monitoring and care - Google Patents

Abstract

Techniques are disclosed for monitoring a user's respiratory disease and providing decision support by analyzing the user's audio data. Speech phonemes within the audio data may be detected and acoustic characteristics extracted for these phonemes. The computation of distance metrics may compare users' phoneme feature sets. Based on this comparison, a determination may be made regarding the user's respiratory illness, such as whether the user has a respiratory illness (e.g., an infection) and/or whether the illness is changing. . Some aspects include utilizing a set of phoneme features to predict a user's future respiratory illness. Utilization of detected or predictive respiratory disease information by decision support tools in the form of computer applications or services may initiate actions to treat current disease or mitigate future risks.

Description

Viral and bacterial respiratory infections, such as influenza, affect many people each year, with symptoms ranging from mild to severe. Viral or bacterial levels typically peak in an infected person's body before self-reported symptoms, and patients often do not realize they are infected. Additionally, most patients typically have difficulty detecting new or mild respiratory symptoms or quantifying changes in symptoms (worsening or improvement of symptoms). However, early detection of respiratory infections may lead to more effective interventions that reduce the duration and/or severity of infections. Early detection is also beneficial in clinical trials. This is because it may be impossible to ascertain symptoms in a potential trial participant that correlate with the infection of interest until such a time that the infectious agent burden in the potential trial participant becomes too low. Therefore, there is a need for tools that utilize objective measures to detect and monitor symptoms of respiratory infections before symptoms worsen to the point that a visit to a health care provider is usually required.

This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter or to be used alone as an aid in determining the scope of the claimed subject matter.

Embodiments of the technology described in this disclosure provide improved computerized decision support tools for determining and quantifying changes in a patient's respiratory disease, including the possibility of a respiratory infection. It is possible to monitor a patient's respiratory disease by determining the possibility of developing the disease or predicting the patient's future respiratory disease.

At a high level, these embodiments include automatically detecting data indicative of respiratory illness in a patient by utilizing audio data acquired by a sensor device, such as a microphone, that may be incorporated into a user computing device, such as a smartphone. It may include doing. For example, audio data may be provided as audio samples by users of one embodiment of these technologies, which may be provided during daily interactions with computing equipment (e.g., smart speakers). It may be in the form of a sustained utterance (e.g., "aaaaaaaa"), a planned utterance, or an unplanned utterance. In some embodiments, instructions may also be provided to guide the user through steps that provide audio data that can be used to monitor the user's respiratory illness. In this way, data for monitoring respiratory diseases can now be obtained unobtrusively and reliably in a non-laboratory setting while users perform their daily activities, including home activities. You can leave it there. Thus, embodiments described herein increase the likelihood of user compliance while providing reliable data to accurately and effectively monitor a user's respiratory disease.

According to one embodiment, phonemes may be detected from the user's recorded audio data, and acoustic characteristics for the detected phonemes may be extracted or determined. These characteristics may include a set of phoneme features or feature vectors that characterize a user's respiratory illness at a particular time interval (eg, date and time), and thus are considered associated with the particular time interval. The user may provide multiple audio voice samples at multiple time intervals (e.g., every morning and night of each day or multiple days) so that the audio sample data is determined for the particular time interval provided by the user. Each phoneme characteristic set may be associated with each other. For example, in one aspect, the detected phonemes include:

or any combination thereof. In another aspect, the detected phonemes include:

may include one or more of the basic vowel phonemes such as

may further include. Detected phonemes may be utilized by one embodiment of the techniques described herein to determine biomarkers of respiratory disease. In another aspect, a combination of one or more of the phonemes or their respective characteristics may be utilized to determine the biomarker. In yet other embodiments, other phonemes or phoneme characteristics and/or respiratory or speech-related data may be utilized to determine the biomarkers.

In order to determine the differences between the values of the phoneme characteristics, sets of phoneme characteristics of different time periods may be compared. For example, Euclidean distance measurements between phoneme feature sets may be determined. Similarly, in some embodiments, Levenshtein distances may be determined, such as embodiments that compare user readings of sentences aloud. A determination regarding a user's respiratory disease may be made based on differences between sets of phoneme characteristics in different time periods. For example, one embodiment of the present disclosure may provide information that the user has a respiratory illness in general, that the user has a particular type of respiratory illness (e.g., influenza), and/or that the user's respiratory illness has worsened over a period of time. It may be determined whether the change is occurring, improving, and/or not changing. As such, the techniques disclosed herein are based on objective data of a user's respiratory illness, such as quantifiable detected changes in phonemic characteristics, and the ability to detect a user's respiratory illness, such as the likelihood of a respiratory infection. It may also be used for automatic determination regarding organ diseases. In some embodiments, the differences between these determined phoneme characteristics may be utilized to predict a user's future (ie, future point in time) respiratory illness. In some embodiments, the user's physiological data, self-reported symptoms, sleep data, location, and/or weather-related information may also be used in conjunction with the phonemic feature data to determine or predict respiratory disease in the user. Context information such as the following may also be used.

The computing device may be activated based on the determination of the user's respiratory illness. As one non-limiting example, this action may include electronic communication of the alert or notification to the user, clinician, or caregiver of the user. The notification may include information regarding the user's respiratory illness and, in some cases, may include a detected change in the user's respiratory illness and/or a prediction of the user's future respiratory illness. Another example of an operation may include communicating a recommendation for treatment or support based on the user's determination or anticipated respiratory illness. For example, this recommendation may include consulting a health care provider, continuing an existing prescription or over-the-counter medication (e.g., refilling a prescription), modifying the dosage or medication of a current treatment procedure, and/or monitoring for respiratory disease. may include the continuation of In some embodiments, the operations include initiating one or more of the above or other recommendations, such as automatically reserving the user's healthcare provider and/or communicating notifications to the pharmacy regarding prescription refills. May contain.

In some cases, acoustic signature information from a user's voice sample can be used to determine a respiratory illness (for example, the user may have an infection) even if the user is not experiencing symptoms. It may be. This functionality provided by some embodiments of the techniques disclosed herein is an advantage over prior art techniques that may rely solely on subjective or objective data obtained from a clinician's visit after the onset of symptoms. and improvements. Early detection and warning of this respiratory disease may allow for more effective treatment to reduce the duration and/or severity of the disease. In addition, these embodiments that enable early detection can reduce severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) or coronavirus infection ( It is believed to be particularly useful in combating respiratory pandemics such as COVID-19). If the disease is caused by a virus or bacteria, early warning allows users to take immediate precautions against transmission (e.g., wearing a mask, self-quarantine, or practicing social distancing), thereby reducing the risk to others. Potential for transmission may be reduced. Early detection provided by embodiments of the present disclosure may also be useful in clinical trials investigating vaccines and/or treatments for respiratory infections. According to some embodiments, participants are able to confirm that any symptoms are correlated with the infection of interest before the pathogen load becomes too small, a problem that frequently occurs with conventional techniques used in clinical trials. Confirmation regarding the following may be carried out.

Furthermore, monitoring respiratory diseases using acoustic signatures from voice recordings can improve the accuracy of treatment of patients with respiratory diseases. For example, in order to more accurately determine when treatment, such as antibiotics, is needed, without treating the patient prematurely and/or for too long, a patient's underlying respiratory illness may be detected in audio recordings in accordance with the present disclosure. You may also be able to be tracked at home using Additionally, tracking the evolution of a patient's disease being treated according to embodiments of the present disclosure can help determine whether changes in treatment are recommended, such as changes in drugs and/or dosages. As such, the techniques disclosed herein may facilitate more accurate utilization of antibiotics/antimicrobials. This is because such drugs require prescription or continuation based on objective and quantifiable changes detected in a patient's respiratory disease.

Hereinafter, aspects of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an exemplary operating environment suitable for implementing aspects of the present disclosure. FIG. 1 illustrates an example computer architecture suitable for implementing aspects of the present disclosure. FIG. 1 is an exemplary diagram illustrating a diagrammatic representation of an exemplary process for monitoring respiratory disease, according to an embodiment of the present disclosure; FIG. 1 is an exemplary diagram illustrating a diagrammatic representation of an exemplary process for collecting data for monitoring respiratory disease, according to an embodiment of the present disclosure; FIG. 1 is an illustration of an example scenario utilizing an embodiment of the present disclosure; FIG. 1 is an illustration of an example scenario utilizing an embodiment of the present disclosure; FIG. 1 is an illustration of an example scenario utilizing an embodiment of the present disclosure; FIG. 1 is an illustration of an example scenario utilizing an embodiment of the present disclosure; FIG. 1 is an illustration of an example scenario utilizing an embodiment of the present disclosure; FIG. 1 is an illustration of an example scenario utilizing an embodiment of the present disclosure; FIG. 1 is an illustrative illustration of an example screenshot from a computing device illustrating an aspect of an example graphical user interface (GUI), in accordance with an embodiment of the present disclosure; FIG. 1 is an illustrative illustration of an example screenshot from a computing device illustrating an aspect of an example graphical user interface (GUI), in accordance with an embodiment of the present disclosure; FIG. 1 is an illustrative illustration of an example screenshot from a computing device illustrating an aspect of an example graphical user interface (GUI), in accordance with an embodiment of the present disclosure; FIG. 1 is an illustrative illustration of an example screenshot from a computing device illustrating an aspect of an example graphical user interface (GUI), in accordance with an embodiment of the present disclosure; FIG. 1 is an illustrative illustration of an example screenshot from a computing device illustrating an aspect of an example graphical user interface (GUI), in accordance with an embodiment of the present disclosure; FIG. 1 is a flow diagram illustrating an example method of monitoring respiratory disease, according to an embodiment of the present disclosure; FIG. FIG. 2 is a flow diagram illustrating an example method of monitoring respiratory disease, according to another embodiment of the present disclosure. FIG. 3 is an exemplary diagram illustrating a representation of an exemplary change in acoustic characteristics over time, according to an embodiment of the present disclosure. 1 is an exemplary diagram illustrating a graphical representation of a decay constant of symptoms of a respiratory infection, according to an embodiment of the present disclosure; FIG. 1 is an exemplary diagram illustrating a graphical representation of a correlation between acoustic characteristics and symptoms of a respiratory infection, according to an embodiment of the present disclosure; FIG. FIG. 2 is an illustrative graphical representation of an exemplary patient's self-reported symptom score change over time, according to an embodiment of the present disclosure. FIG. 3 is an exemplary graphical representation of a rank correlation between distance metrics computed for different acoustic characteristics and self-reported symptom scores, according to an embodiment of the present disclosure. FIG. 3 is an exemplary graphical representation of a rank correlation between distance metrics computed for different acoustic characteristics and self-reported symptom scores, according to an embodiment of the present disclosure. FIG. 3 is an exemplary graphical representation of rank correlation between distance metrics and self-reported symptom scores for different patients, according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating statistically significant correlations between acoustic characteristic types and phonemes according to an embodiment of the present disclosure. FIG. 3 is an illustrative graphical representation of relative changes in acoustic characteristics and self-reported symptoms over time for three exemplary patients, according to an embodiment of the present disclosure. FIG. 2 is an illustrative diagram of an exemplary representation of the performance of a respiratory infection detector, according to an embodiment of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 is an illustration of an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 is an illustration of an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 is an illustration of an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. FIG. 3 illustrates an example computer program routine for extracting acoustic characteristics for monitoring respiratory disease, in accordance with various embodiments of the present disclosure. 1 is a block diagram of an exemplary computer environment suitable for use in implementing an embodiment of the present disclosure. FIG.

The subject matter of the present disclosure is particularly described herein using various aspects to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this patent. The claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to those described in this disclosure, in conjunction with other current or future technologies. . Additionally, although the terms "step" and/or "block" may be used herein to connote various elements of the method employed, these terms may refer to individual Unless the order of steps is explicitly stated, it should not be construed as implying any particular order between the various steps disclosed herein. Each method described herein may include a computer process that may be implemented using any combination of hardware, firmware, and/or software. For example, various functions may be performed by a processor that executes instructions stored in computer memory. Additionally, these methods may be embodied as computer-usable instructions stored on a computer storage medium. These methods may be provided by a standalone application, service or hosting service (standalone or in combination with another hosting service), or a plug-in to another product, to name a few.

Aspects of the present disclosure relate to computerized decision support tools for respiratory disease monitoring and care. Respiratory diseases affect many people every year, with symptoms ranging from mild to severe. Such respiratory diseases may include respiratory infections or non-infectious respiratory conditions caused by bacterial or viral agents such as influenza. Although some aspects of this disclosure describe respiratory infections, it is believed that such aspects may apply to respiratory diseases in general.

Patients typically have difficulty detecting new or mild respiratory symptoms as well as quantifying changes in symptoms (ie, worsening or improvement of symptoms). Traditionally, objective measures of respiratory disease are determined only when a patient presents to a health care professional and a sample analysis is performed. However, levels of viruses or bacteria that can cause respiratory infections usually peak in an infected patient's body before self-reported symptoms, and patients often do not realize they have an infection until they are diagnosed. For example, people with influenza or coronavirus disease 2019 (COVID-19) can infect others before they feel symptoms. If mild symptoms of respiratory diseases such as infections cannot be measured objectively at an early stage, there is a risk of transmitting the infection to others, prolonging the respiratory disease, and making the respiratory disease more severe. It gets expensive.

To improve respiratory disease monitoring and care, embodiments of the present disclosure determine a user's respiratory disease and/or predict a user's future respiratory disease based on acoustic data from the user's voice recordings. One or more decision support tools can be provided to assist in making decisions. For example, if a user provides audio data through a voice recording, the acoustic properties of phonemes in the audio data (sometimes referred to herein as phoneme properties) may be determined. good. In one embodiment, multiple audio recordings may be received, each corresponding to a different time segment (e.g., audio recordings may be obtained daily over several days). . In order to determine information regarding the user's respiratory disease, such as whether the user's respiratory disease has changed over time, phoneme characteristic values in different time periods may be compared. Based on the determination of the user's respiratory illness, actions such as alerts or decision support recommendations may be automatically provided to the user and/or the user's clinician.

In one embodiment, acoustic information is received from a monitoring patient (sometimes referred to herein as a user) using a sensor, such as a microphone, as described elsewhere herein. It may be. The acoustic information may include one or more recordings of the user's sounds (eg, vocalizations or other breathing sounds). The audio recording may include, for example, audio samples of sustained utterances (eg, "aah"), planned utterances, or unplanned utterances. A microphone may be integrated into or coupled to user computing equipment such as a smartphone, smart watch, or smart speaker. In some cases, voice audio samples may be recorded at the user's home or in the user's daily activities, and may also be recorded during the user's daily interactions with the user's computer equipment, such as smart speakers. It may also contain data that was

Additionally, some embodiments may generate and/or provide instructions that guide the user through providing audio data that can be used to monitor the user's respiratory illness. For example, FIGS. 4A, 4B, and 4C each show that user computing equipment (or user equipment) outputs instructions (e.g., in the form of text and/or audible instructions) to the user as part of an evaluation exercise. It shows a scenario where These instructions prompt the user to utter a particular sound and, in some embodiments, indicate the duration of the utterance (eg, "Keep saying the sound 'aah' for 5 seconds"). In some embodiments, the instructions include:

The user may be asked to hold or maintain a utterance for as long as possible, such as the utterance of one of the basic vowels such as . In some embodiments, the instructions also include asking the user to read the written text aloud. Some embodiments include audio instructions, such as instructing the user when to start/stop, to speak longer, to hold for longer periods of time, to reduce background noise, and/or other feedback for quality control. The method may further include providing feedback to the user to enable the sample.

In some embodiments, acoustic and audio information, such as phonemes, may be detected from audio data received from a user. In one embodiment, the detected phoneme includes the phoneme

may include. In another embodiment, the detected phoneme includes the phoneme

including. In some embodiments of the techniques described herein, the detected phonemes may be adapted for use in determining biomarkers for detection and monitoring of respiratory diseases. Once a phoneme is detected, acoustic characteristics of the detected phoneme may be extracted or determined from the audio data. Examples of acoustic properties include, but are not limited to, data characterizing power and power variability, pitch and pitch variability, spectral structure, and/or formant measures. In some embodiments, different sets of characteristics (ie, different combinations of acoustic characteristics) may be determined for different phonemes detected in the audio data. In one exemplary embodiment, the phoneme

Twelve characteristics were determined for the phoneme

Twelve characteristics were determined for the phoneme

Eight characteristics are determined for . In some embodiments, preprocessing or signal conditioning operations may be performed to facilitate phoneme detection and/or determine phoneme characteristics. These operations may include, for example, trimming the audio sample data, frequency filtering, normalization, removing background noise, intermittent spikes, other acoustic artifacts, or other operations as described herein. You can stay there.

As audio data is acquired from a user over time, multiple phoneme feature sets, which may include phoneme feature vectors, are generated and may be associated with different time segments. In some embodiments, a time series may be assembled with successive feature sets in chronological or reverse chronological order according to temporal information associated with a user's phoneme feature set. Differences or changes in the values of characteristics within the associated characteristic set at different points in time or time intervals may be determined. For example, differences in the user's phoneme feature vectors may be determined by comparing one or more phoneme feature vectors associated with different points in time or time periods. In one embodiment, this difference may be determined by computing a distance metric such as the Euclidean distance between the characteristic vectors. In some cases, one of the phoneme feature sets utilized for comparison represents a health criterion for the user. The set of health criteria characteristics may be determined based on audio data obtained when it is known or estimated that the user does not have a respiratory disease. Similarly, a set of disease criteria characteristics determined based on audio data obtained when a user is known to have a respiratory disease or is presumed to have a respiratory disease may be used. .

Based on the differences between sets of phoneme features at different times, information regarding a user's determination of respiratory disease may be provided. In some embodiments, this determination may be provided as a respiratory disease score, as described elsewhere herein. A respiratory illness score measures the likelihood or probability that a user has (or does not have) a respiratory illness, such as an infection (e.g., with respect to any respiratory illness in general or a particular respiratory illness). It may be compatible. Alternatively or additionally, the respiratory illness score may indicate whether the user's respiratory illness is improving, worsening, or unchanged. For example, the example scenario of FIG. 4F illustrates one embodiment in which a user is determined to have not recovered from a respiratory illness based on an analysis of the user's audio information, as described herein. There is. In other embodiments, the respiratory illness score may indicate the likelihood that the user will develop, remain afflicted with, or recover from a respiratory illness within a future time interval. . The example scenario of FIG. 4E depicts one embodiment in which a user with a cold is expected to get better within the next three days.

In some embodiments, context information in addition to the user's voice information may be utilized to determine or predict a user's respiratory illness. As described elsewhere herein, contextual information may include, but is not limited to, user physiological data such as body temperature, sleep data, travel information, self-reported symptoms, location, or weather-related information. Self-reported symptom data may include, for example, whether the user is experiencing a particular symptom, such as nasal congestion, and may further include an assessment of the perceived degree or severity of the symptom. . In some cases, symptom self-reporting tools may be utilized to obtain user symptom information. In some embodiments, automated instructions to provide self-report information (or notifications requesting the user to report symptom data) based on an analysis of the user's voice-related data or a determination of the user's respiratory illness. ) may occur. The example scenario of FIG. 4D illustrates one embodiment in which a user is determined to be potentially ill based on an analysis of the user's voice. In this embodiment, the monitoring software application may ask the user, for example, whether the user is experiencing certain respiratory-related symptoms (eg, stuffy nose, fatigue, etc.). The example of FIG. 4D further shows that a user who recognizes nasal congestion is prompted to rate the severity of the nasal congestion. The user's self-reported symptoms may be used to make additional determinations or predictions regarding the user's respiratory illness. In some embodiments, the user's physiological data (such as heart rate, body temperature, sleep, or other data), weather-related information (such as humidity, temperature, pollution, or similar data), location, or Other context information may also be used, such as information about the prevalence of respiratory infections in the user's area, such as information about the prevalence of respiratory infections in the user's area.

The computing device may initiate an operation based on a determination of a user's respiratory illness, which may include a change (or lack of change) in the disease. This action may include, for example, electronic communication of the alert or notification to the user, clinician, or caregiver of the user. In some embodiments, the notification or alert includes a respiratory disease score, information quantifying or characterizing changes in the user's respiratory disease, the current state of the respiratory disease, and/or the user's future respiratory disease. It may also include information regarding the user's respiratory disease, such as a prediction of. In some embodiments, the operations may include processing the respiratory disease information to further make decisions, including recommending treatment and support based on the user's respiratory disease. Good too. For example, this recommendation may include consulting a health care provider, continuing an existing prescription or over-the-counter medication (e.g., refilling a prescription), modifying the dosage or medication of a current treatment procedure, and/or monitoring for respiratory disease. It may include modification or no modification (i.e., continuation) of . In some embodiments, the operations include initiating one or more of the above or other recommendations, such as automatically scheduling a user's healthcare provider and/or communicating a prescription refill notification to a pharmacy. May contain. The example scenario of FIG. 4F is that based on a determination that the user's respiratory illness is not improving, the user's physician is notified and the antibiotic prescription is refilled and delivered to the user. 3 illustrates one embodiment of a scheduled embodiment.

Still other types of operations may include automatically initiating or performing operations associated with monitoring or treating a user's respiratory illness. By way of non-limiting example, this action may include automatically scheduling a user's healthcare provider, sending notifications to a pharmacy regarding prescription refills, or procedures associated with monitoring a user's respiratory illness or breathing. may include modifications to computer operations used to monitor organ disease. In one embodiment of an exemplary operation, a voice analysis procedure, such as a computer programming operation, utilized to obtain or analyze user voice-related data is modified. In one such embodiment, the user may be prompted to provide audio samples more frequently, such as twice a day, or may be prompted to provide audio samples more frequently, such as twice a day, or from daily interactions with computer equipment. In embodiments where audio information is collected, the audio information may be collected more frequently. In another such embodiment, specific phoneme or characteristic information collected or analyzed by the respiratory disease monitoring application may be modified. In one embodiment, the computer programming operations may be modified such that the user may be instructed to produce a different set of sounds than previously provided sounds. Similarly, in another type of operation, computer programming operations may be modified to prompt a user to provide symptom data as described above.

Among other benefits, embodiments of the technology disclosed herein may provide early detection of respiratory diseases such as infections. These embodiments may detect much milder respiratory symptoms or signs of respiratory illness before the patient suspects illness (e.g., before the user feels symptoms) and may be used to detect symptoms by the patient or healthcare provider. The acoustic characteristics of the user's vocalizations, including breathing sounds, may be used to alert the user. Early detection of respiratory disease may lead to more effective interventions to reduce the duration and/or severity of infection. Also, early detection of respiratory infections can reduce the risk of transmission to other patients. This is because infected patients can take precautions against transmission, such as wearing masks or self-quarantining, before other patients do. As such, these embodiments provide an improvement over traditional respiratory disease (including respiratory infections) detection techniques that rely on user reporting of symptoms, resulting in slow (or no) disease detection. provide. Additionally, these traditional techniques have low accuracy or inaccuracy due to the subjectivity of users' self-reported data.

Early detection of respiratory infections can also be beneficial in clinical trials. For example, in vaccine clinical trials, it is necessary to confirm the correlation between patient symptoms and the infectious disease of interest. If a patient is not diagnosed early enough, the patient's pathogen burden may become too small to correlate the patient's symptoms with the infection of interest. Without confirmation, it is also not possible for the patient to participate in the trial. Therefore, the embodiments described herein can be utilized not only for early detection for more effective treatment, but also, when utilized in clinical trials, to increase potential new potential by increasing trial participation. Treatments or vaccines may be developed.

Another advantage that may be provided by embodiments of the technology disclosed herein is that users are more likely to comply with respiratory disease monitoring. For example, as described elsewhere herein, a user's voice may be recorded remotely from the home or clinic and, in some embodiments, while performing daily routines (e.g., daily conversations) that are of little burden to the patient. Records may be obtained unobtrusively. A low-burden method of monitoring respiratory diseases, including the capture of user data, could increase user compliance, help ensure early detection, and provide another improvement to traditional methods of monitoring respiratory diseases. Available.

Yet another advantage that may be provided by embodiments of the technology disclosed herein is increased precision in treating patients with respiratory diseases. In particular, some of the embodiments of the present disclosure enable tracking of underlying respiratory illnesses, such as infections, to determine whether the illness is getting worse, improving, or unchanged. Yes, this may affect patient treatment. For example, patients with mild initial symptoms may not require immediate medication or treatment. Some embodiments of the present disclosure include monitoring disease evolution and alerting patients and/or health care providers when the disease worsens to the point where treatment (e.g., medication) may be required or recommended. It may also be used for arousal. Embodiments of the present disclosure also determine whether a patient is recovering from a respiratory illness, such as an infection, and thus whether a change in treatment, such as a change in drug and/or dosage, is recommended. You may also do so. In another example, embodiments of the present disclosure may determine a user's respiratory disease if the user is prescribed a drug with potential respiratory-related side effects, such as a certain anti-cancer drug, and It may be determined whether a change in treatment is recommended based on whether respiratory-related side effects are occurring and their severity. As such, some embodiments of the technology described herein provide improvements over the prior art by allowing for more accurate utilization of drugs, particularly certain drugs such as antibiotics/antibacterials. Available. This is because such drugs can be prescribed or continued based on quantifiable changes detected in a patient's respiratory disease.

Referring now to FIG. 1, a block diagram illustrating an example operating environment 100 in which some embodiments of the present disclosure may be employed is provided. It is to be understood that this configuration and other configurations described herein are shown by way of example only. Other configurations and elements may be used in addition to or in place of the configurations and elements (e.g., devices, interfaces, functions, orders, and groupings of functions) shown in FIG. 1 and other drawings, and some of the elements may be omitted entirely for clarity. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components in any suitable combination and location. Various functions or operations described herein are performed by one or more entities including hardware, firmware, software, and combinations thereof. For example, some functions may be performed by a processor executing instructions stored in memory.

As shown in FIG. 1, an exemplary operating environment 100 includes a number of user equipments, such as user computing equipment (synonymously referred to as "user equipment") 102a, 102b, 102c-102n and clinician user equipment 108. one or more decision support applications, such as decision support applications 105a and 105b, an electronic health record (EHR) 104, one or more data sources, such as a data store 150, a server 106, and a sensor. 103 and a network 110. It is understood that the operating environment 100 shown in FIG. 1 is one example of one suitable operating environment. Each of the components shown in FIG. 1 may be implemented by any type of computer equipment, such as computer equipment 1700 described in connection with FIG. 16, for example. These components may be adapted to communicate with each other via a network 110, which may include, without limitation, one or more local area networks (LANs) and/or wide area networks (WANs). In an exemplary implementation, network 110 may include the Internet and/or a cellular network, among any of a variety of possible public and/or private networks.

Operating environment 100 includes any number of user equipment (such as 102a-102n and 108), servers (such as 106), decision support applications (such as 105a, 105b), and data sources (such as data store 150) within the scope of this disclosure. ), and EHRs (such as 104) may be employed. Each element may include a single device or component, or multiple devices or components working together in a distributed environment. For example, server 106 may be provided by multiple devices located in a distributed environment that collectively provide the functionality described herein. Additionally, other components not illustrated herein may be included in the distributed environment.

User equipment 102a, 102b, 102c-102n and clinician user equipment 108 may be client user equipment on the client side of operating environment 100, while server 106 may be on the server side of operating environment 100. Server 106 includes server-side software designed to interface with client-side software on user equipment 102a, 102b, 102c-102n, and 108 to achieve any combination of features and functionality discussed in this disclosure. You may do so. This division of operating environment 100 is provided to illustrate one example of a suitable environment, and any combination of server 106 and user equipment 102a, 102b, 102c-102n, and 108 remains a separate entity. It's not something I'm asking for.

User equipment 102a, 102b, 102c-102n, and 108 may include any type of computing equipment that can be used by a user. For example, in one embodiment, user equipment 102a, 102b, 102c-102n, and 108 may be computer equipment of the type described in connection with FIG. 16 herein. By way of non-limiting example, user equipment may include a personal computer (PC), a laptop computer, a mobile or mobile device, a smartphone, a smart speaker, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA) device, a music player. or MP3 players, Global Positioning System (GPS), video players, handheld communication devices, gaming devices, entertainment systems, vehicle computer systems, embedded system controllers, cameras, remote controllers, appliances, consumer electronics, workstations, Alternatively, it may be embodied as any combination of these devices, or any other suitable computer device.

Some user equipment, such as user equipment 102a, 102b, 102c-102n, may be intended for use by a user being observed by one or more sensors, such as sensor 103. In some embodiments, the user equipment may include an embedded sensor (similar to sensor 103) or may be adapted to interface with an external sensor (similar to 103). In an exemplary embodiment, sensor 103 detects acoustic information. For example, sensor 103 may be implemented in, through, or communicatively coupled to a smart device, such as a smart speaker, smart mobile device, smart watch, or as a separate microphone device. It may include one or more microphones (or microphone array). Other types of sensors may also be integrated into or integrated with the user equipment, such as physiological sensors (e.g., sensors that detect heart rate, blood pressure, blood oxygen levels, body temperature, and related data). They may also collaborate. However, it is also contemplated that physiological information regarding a patient, according to embodiments of the present disclosure, may be received from patient historical data in the EHR 104 or human measurements or human observations. Other types of sensors that may be implemented in the operating environment 100 include sensors configured to detect the user's location (e.g., indoor positioning system (IPS) or global positioning system (GPS)), detect atmospheric information, etc. sensors configured to detect ambient light (e.g., light detectors); and sensors configured to detect motion. (e.g. gyroscope or accelerometer).

In some embodiments, the sensor 103 operates or operates on a smartphone carried by a user (such as user equipment 102c) or a smart speaker (such as user equipment 102b) located in one or more areas where a patient may be present. Operation through these may be possible. For example, sensor 103 may be a microphone that is integrated into a smart speaker located in the patient's home and can detect sound information (including the user's voice) occurring within a maximum distance from the smart speaker. Alternatively, it is contemplated that sensor 103 may be incorporated in other ways, such as incorporation into equipment placed on or near the wearer's body. In other embodiments, the sensor 103 is, for example, a skin patch sensor attached to the user's skin, an ingestible sensor, or a subcutaneous sensor, or attached to the user's living environment (including appliances such as a television, thermostat, doorbell, camera, etc.). It may also be an integrated sensor component.

Data may be acquired by sensor 103 continuously, periodically, from time to time, or as it becomes available. Additionally, the data acquired by sensor 103 may be associated with time and date information and may be represented as one or more time-series measured variables. In one embodiment, the sensor 103 collects raw sensor information to perform signal processing, forming variable decision statistics, cumulative sums, trend analysis, wavelet processing, thresholding, computing decision statistics, and determining decision statistics. logic processing, preprocessing, and/or signal conditioning may be performed. In some embodiments, sensor 103 may include an analog-to-digital converter (ADC) and/or processing functionality to perform digital audio sampling of analog audio information. In some embodiments, analog-to-digital converters and/or processing functionality that performs digital audio sampling to determine digital audio information may be implemented in either user equipment 102a-102n or server 106. . Alternatively, one or more of these signal processing functions may be performed by user equipment such as user equipment 102a-102n or clinician user equipment 108, server 106, and/or decision support application (app) 105a or 105b. It may be possible to do so.

Some user equipment, such as clinician user equipment 108, may be configured for use by a clinician treating or monitoring a user associated with sensor 103. Clinician user equipment 108 may be embodied as user equipment 102a-102n or one or more computer equipment, such as server 106, and is communicatively coupled to EHR 104 through network 110. Operating environment 100 illustrates indirect communication coupling between clinician user equipment 108 and EHR 104 through network 110 . However, it is also contemplated that one embodiment of clinician user equipment 108 may be communicatively coupled directly to EHR 104. One embodiment of clinician user equipment 108 may include a user interface (not shown in FIG. 1) that is operated by a software application or set of applications on clinician user equipment 108. In one embodiment, this application may be a web-based application or applet. An example of this application includes a clinician dashboard, such as the example dashboard 3108 described in connection with FIG. 3A. Embodiments described herein enable healthcare provider applications (e.g., clinical Clinician applications (such as dashboard applications) that may run on clinician user equipment 108 may be facilitated. Some embodiments of clinician user equipment 108 (or a clinician application running on the equipment) may record patient history, healthcare resource data, physiological variables or data (e.g., vital signs), measurement results, time further facilitate accessing and receiving information regarding a particular patient or set of patients, such as series, predictions described below (including plotting or displaying decision results and/or issuing alerts), or other healthcare related information; obtain. Clinician user equipment 108 may further facilitate displaying results, recommendations, or other instructions, for example. In one embodiment, clinician user equipment 108 may facilitate receiving patient instructions based on the results of respiratory disease monitoring and determination or prediction described herein. Clinician user equipment 108 may also be used in conjunction with various embodiments to provide diagnostic services or evaluations of the performance of the techniques described herein.

Embodiments of decision support applications 105a and 105b include a software application or set of applications (including programs, routines, functions, or computer-executed services) that reside on one or more servers distributed in a cloud computing environment. (e.g., decision support application 105b), or any of the user equipment 102a-102n, such as a personal computer, laptop, smartphone, tablet, mobile computing device, or front-end terminal in communication with a back-end computer system, or any of the user equipment 102a-102n. It may include a software application or set of applications (which may include programs, routines, functions, or computer-implemented services) (eg, decision support application 105a) residing on one or more client computing devices. In one embodiment, decision support applications 105a and 105b include client-based and/or web-based applications (or apps) that can be used to access user services provided by an embodiment of the present disclosure. It may include a set of applications (or apps). In one such embodiment, decision support applications 105a and 105b each include user equipment 102a-102n, clinician user equipment 108, sensor 103, and EHR 104, including predictions and assessments determined by embodiments of the present disclosure. , or may facilitate processing, interpretation, access, storage, retrieval, and communication of information obtained by data store 150.

Use of information and retrieval or related functionality through decision support applications 105a and 105b may require login with user credentials, such as a patient or clinician. Additionally, decision support applications 105a and 105b may be applicable to clinicians, patients, associated health care facilities or systems, and/or regarding the protection of health information, such as the provisions of the Health Insurance Portability and Accountability Act (HIPAA). Data may be stored and transmitted in accordance with privacy settings established by local and federal regulations.

In one embodiment, decision support applications 105a and 105b may communicate notifications (such as alerts or instructions) directly to clinician user equipment 108 or user equipment 102a-102n over network 110. If these applications are not running on these devices, notifications may be displayed on any other devices on which decision support applications 105a and 105b are running. Decision support applications 105a and 105b may also send or display maintenance instructions to clinician user equipment 108 or user equipment 102a-102n. Additionally, the use of interface components in decision support applications 105a and 105b may provide information about the functionality or functionality of sensor 103, such as operational settings or parameters, user identification, user data stored on sensor 103, and diagnostic services or firmware updates for sensor 103. Access to information by users (including clinicians/caregivers or patients) may be facilitated.

Further, embodiments of decision support applications 105a and 105b may collect sensor data directly or indirectly from sensors 103. As described with respect to FIG. 2, decision support applications 105a and 105b may utilize sensor data to extract or determine acoustic characteristics and to determine respiratory diseases and/or symptoms. In one aspect, decision support applications 105a and 105b may include user equipment such as user equipment 102a-102n and 108 (such as a variety of graphical user interfaces (GUIs), such as the exemplary graphical user interfaces (GUI) shown in FIGS. 5A-5E); The results of such processes may be displayed or provided to the user via an audio user interface or other user interface. As such, the functionality of one or more components discussed below with respect to FIG. or controlled by decision support applications 105a and 105b. Additionally or alternatively, decision support applications 105a and 105b may include a decision support tool, such as decision support tool 290 of FIG.

As previously mentioned, operating environment 100 includes one or more EHRs 104 that may be associated with monitoring patients. EHR 104 may be communicatively coupled directly or indirectly to user equipment 102a-102n and 108 via network 110. In some embodiments, the EHR 104 may represent health information from a variety of sources and may be embodied as different systems of record, such as separate EHR systems for different clinician user equipment (such as 108). You can. As a result, clinician user equipment (such as 108) may be for clinicians at different provider networks or care facilities.

Embodiments of EHR 104 may include one or more data stores of health records or health information that may be stored in data store 150 and one or more data stores that facilitate storage and retrieval of health records. It may further include a computer or server (such as server 106). In some embodiments, EHR 104 may be implemented as a cloud-based platform or distributed across multiple physical locations. EHR 104 may further include a record system that may store real-time or near real-time patient (or user) information, such as a wearable, bedside, or in-home patient monitor.

Data store 150 may include one or more data sources and/or configured to make data available to any of the various components of operating environment 100 or system 200 described in conjunction with FIG. May represent a computer data storage system. In one embodiment, data store 150 may provide sensor data that may be made available (or otherwise available for access) to data collection component 210 of system 200. Data store 150 may include a single data store or multiple data stores, and may be located locally and/or remotely. Some embodiments of data store 150 may include networked or distributed storage, including storage on a server (such as server 106) located in a cloud environment. Data store 150 may be discrete from user equipment 102a-102n and 108 and server 106, or may be combined and/or integrated with at least one of these equipment.

Operating environment 100 includes one or more components or operations of system 200 (shown in and described in conjunction with FIG. 2), including components or operations for collecting audio data or context information. implementing the operations performed by the components of the system, facilitating user interaction to collect such data, identifying possible or known respiratory diseases (e.g., respiratory infections or non-infectious diseases); and/or to implement decision support tools (such as decision support tool 290 of FIG. 2). Operating environment 100 may also be adapted to implement aspects of methods 6100 and 6200 as described in conjunction with FIGS. 6A and 6B, respectively.

Referring now to FIG. 2 and with continued reference to FIG. 1, a block diagram illustrating aspects of an exemplary computer system architecture, generally designated as system 200, suitable for implementing an embodiment of the present disclosure is provided. Ru. System 200 represents only one example of a suitable computer system architecture. Other structures and elements may be used in addition to or in place of those illustrated, and some elements may be omitted entirely for clarity. Further, similar to operating environment 100 of FIG. 1, many of the elements described herein may be implemented as discrete or distributed components or in conjunction with other components, in any suitable combination and location. It is a functional entity that obtains.

The example system 200 includes a data collection component 210, a presentation component 220, a user voice monitor 260, a user interaction manager 280, a respiratory disease tracker 270, a decision support tool 290, and a storage 250. It includes the network 110 described in connection with FIG. 1 communicatively coupling the elements. For example, as a composition of compiled computer instructions or functions, program modules, computer software services, or processes executed on one or more computer systems, such as computer equipment 1700 described in connection with FIG. One or more of the components may be embodied.

In one embodiment, the functions performed by the components of system 200 are associated with one or more decision support applications, services, or routines (decision support applications 105a and 105b of FIG. 1). In particular, such applications, services, or routines may be implemented on one or more user equipments (user equipment 102a and and/or clinician user equipment 108) or a server (such as server 106). Further, in some embodiments, these components of system 200 are integrated into one or more servers (such as server 106) and client devices (such as user equipment 102a-102n or clinician user equipment 108) in a cloud environment. may be distributed throughout the network connecting the computers or may reside on user equipment, such as any of the user computer equipment 102a-102n or the clinician user equipment 108. Furthermore, the functions or services performed by these components may be implemented at any suitable abstraction layer, such as an operating system layer, an application layer, a hardware layer, etc. of a computer system. Alternatively or additionally, these components and/or the functionality of the embodiments described herein may be at least partially performed by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), and systems-on-chip systems (SoCs). ), complex programmable logic device (CPLD), etc., but are not limited to these. Additionally, although functionality is described herein with respect to particular components shown in example system 200, in some embodiments the functionality of these components may be shared across other components. Alternatively, it is also possible that it can be dispersed.

With continued reference to FIG. 2, data collection component 210 generally includes one or more data sources, such as data from sensor 103 and/or data store 150 of FIG. 1, for utilization in embodiments of the present disclosure. may be responsible for accessing or receiving (and possibly identifying) data from. In some embodiments, the data collection component 210 may be configured to provide a specific (or, in some cases, multiple users, including crowdsourced data). This data is received (or accessed) by data collection component 210 , stored, reformatted, and/or combined, and stored in one or more data stores, such as storage 250 , to facilitate system 200 . May be available to other components. For example, user data may be stored in or associated with patient record 240, as described herein. Additionally or alternatively, in some embodiments, no personally identifiable data (i.e., user data that specifically identifies a particular user) is uploaded or provided from one or more data sources. is not stored permanently and/or made available to other components of system 200. In one embodiment, security measures are implemented such as encryption of user-related data so that user privacy is protected. In another embodiment, a user may be able to join or opt out of services provided by the technologies described herein and/or select the user data and/or user data sources utilized by these technologies. Become.

Data utilized in embodiments of the present disclosure may be received from a variety of sources and may be available in a variety of formats. For example, in some embodiments, user data received via data collection component 210 may be determined via one or more sensors (such as sensor 103 of FIG. 1). However, it may be stored on or associated with one or more user equipment (such as user equipment 102a), a server (such as server 106), and/or other computer equipment. As used herein, a sensor may include any functionality, routine, component, or combination thereof that senses, detects, or obtains information such as user data from data store 150, and may also include hardware , software, or both. As mentioned above, by way of non-limiting example, data sensed or determined by one or more sensors may include acoustic information (including user speech, utterances, breathing, coughing, or other vocal sounds), mobile device location information such as Indoor Positioning System (IPS) or Global Positioning System (GPS) data that can be determined by; atmospheric information such as temperature, humidity, and/or pollution; body temperature, heart rate, blood pressure, blood oxygen level, sleep; It may include physiological information such as related information, kinematic information such as accelerometer or gyroscope data, and/or ambient light information such as photodetector information.

In some aspects, the sensor information collected by data collection component 210 may include other characteristics or characteristics of the user equipment (device status, charging data, date/time, or mobile device or user activity information (e.g., app usage, online activities, online searches, audio data such as automatic speech recognition, or activity logs) (in some embodiments , user activity that occurs on two or more user devices), user history, session logs, application data, contacts, calendar and schedule data, notification data, social network data, news (e.g., search engines, social network popular or trending items, health department announcements that may provide information about the number or rate of respiratory infections in a geographic area), e-commerce activity (Amazon.com®, Google ) (including data from online accounts such as eBay®, PayPal®, etc.), user account data (user preferences associated with a personal assistant application or service) or settings), home sensor data, appliance data, vehicle signal data, traffic data, other wearable device data, other user equipment data (e.g., device settings, profiles, network-related information (e.g., network name or ID, domain information, workgroup information, connection data, wireless fidelity (Wi-Fi) network data, configuration data, such as model number if the user has a mobile phone paired with a Bluetooth headset, firmware, equipment, device pairing, or other network-related information), payment or credit card usage data (which may include, for example, information from a user's PayPal account), purchase history data (which may include information from a user's Amazon ( trademark or other sensor data that may be sensed or detected by the sensor (or other detector) component, including data originating from the sensor component associated with the user (such as information from a user's online drugstore account), , movement, orientation, location, user access, user activity, network access, user equipment charging, or other data that may be provided by one or more sensor components), other data (e.g., Wi-Fi, cellular network, or location data that may be derived from Internet Protocol (IP) address data), as well as nearly any other data source from which sensing or determinations may be made.

In some aspects, data collection component 210 may provide collected data in the form of a data stream or signal. A "signal" may be a feed or stream of data from a corresponding data source. For example, user signals may include smart speakers, smartphones, wearable devices (e.g., fitness trackers or smart watches), home sensor devices, GPS devices (e.g., for location coordinates), vehicle sensor devices, user equipment, calendar services, email User data obtained from accounts, credit card accounts, subscription services, news or notification feeds, websites, portals, or any other data source is also possible. In some embodiments, data collection component 210 receives or accesses data on a continuous, periodic, or occasional basis.

Further, user voice monitor 260 of operating environment 200 may generally be responsible for collecting or determining user voice-related data that can be used to detect or monitor respiratory disease. The term voice-related data (synonymously referred to herein as "voice data" or "voice information") is used herein in a broad sense and includes a non-limiting term. As one example, it may include data associated with the user's utterances, vocalizations or utterances that include vocal sounds, or other sounds produced by the user's mouth or nose, such as breathing, coughing, sneezing, or sniffing sounds. Embodiments of user voice monitor 260 may facilitate the acquisition of audio or acoustic information (eg, audio recordings of utterances or voice samples) and, in some aspects, contextual information that may be received by data collection component 210. Embodiments of user speech monitor 260 may determine relevant speech-related information, such as phoneme characteristics, from this audio data. User audio monitor 260 may receive data continuously, periodically, or from time to time, and similarly may receive audio information utilized for respiratory disease monitoring continuously, periodically, or from time to time. It may also be extracted or determined.

In an exemplary embodiment of system 200, user speech monitor 260 includes a recording optimizer 2602, a speech sample collector 2604, a signal preparation processor 2606, a sample recording examiner 2608, a phoneme segmenter 2610, an acoustic feature extractor 2614, and a context information determiner 2616. Alternate embodiments (not shown) of user voice monitor 260 may include only some of these subcomponents or may include additional subcomponents. . As described elsewhere herein, one or more components of user audio monitor 260, such as signal preparation processor 2606, may perform preprocessing operations on audio data, such as raw acoustic data. good. It is also contemplated that additional pre-processing may be performed in accordance with data collection component 210 in some embodiments.

Recording optimizer 2602 may generally be responsible for determining proper or optimized settings to obtain usable audio data. As mentioned above, embodiments of the technology described herein may be adapted for use in a home environment or by an end user in a setting other than a controlled environment, such as a laboratory or clinic. It may be possible to do so. Accordingly, some embodiments may include features that facilitate obtaining audio data of sufficient quality for use in monitoring a user's respiratory disease. In particular, in one embodiment, the recording optimizer 2602 may be utilized to provide such functionality with optimized settings for the acquisition of audio data voice-related information. In an exemplary embodiment, optimization is achieved by adjusting the sensor or modifying signal strength, directivity, sensitivity, frequency, and other acoustic parameters (e.g., microphone parameters) such as signal-to-noise ratio (SNR). settings may be provided. The recording optimizer 2602 may determine that the situation is within a predetermined range for proper settings, or may meet a predetermined threshold (e.g., when the user The sensitivity or level of the microphone is adjusted sufficiently so that voice data can be obtained from the audio data). In some embodiments, recording optimizer 2602 may determine whether recording has started. Also, in some embodiments, recording optimizer 2602 may determine whether the sampling rate meets a threshold sampling rate. In an exemplary embodiment, the recording optimizer 2602 is configured to determine that the audio signal is sampled at the Nyquist rate, which in some cases includes a minimum rate of 44.1 kilohertz (kHz). Good too. Further, the recording optimizer 2602 may determine that the bit depth satisfies a threshold value such as 16 bits. Additionally, in some embodiments, recording optimizer 2602 may determine whether the microphone has been adjusted.

In some embodiments, the recording optimizer 2602 may perform an initialization mode to optimize the microphone level for the particular environment in which the microphone is placed. The initialization mode may include prompting the user to play or generate sounds so that recording optimizer 2602 determines appropriate levels for a particular environment. Additionally, in the initialization mode, the recording optimizer 2602 may prompt the user to face or position the microphone where the user would normally face or position it when requesting user input. Based on user feedback (i.e., audio recording), in the initialization mode, the recording optimizer 2602 configures the audio collection and processing components to provide optimized settings for future recording sessions. Ranges, thresholds, and/or other parameters may be determined. In some embodiments, the recording optimizer 2602 additionally or alternatively includes signal processing features or settings (e.g., noise cancellation as described below) to facilitate the acquisition of usable audio data. It may be decided.

In some embodiments, recording optimizer 2602 makes optimized adjustments (e.g., level adjustments or amplification) to achieve preferred settings through preprocessing in conjunction with signal creation processor 2606. You may also do so. Alternatively, recording optimizer 2602 may configure sensors to achieve a level within a predetermined range or threshold for a particular parameter, such as signal strength.

As shown in FIG. 2, the recording optimizer 2602 may also include a background noise analyzer 2603, which may generally be responsible for identifying and, in some embodiments, removing or reducing, background noise. good. In some embodiments, background noise analyzer 2603 may ensure that the noise intensity level meets a maximum threshold. For example, background noise analyzer 2603 may determine that the ambient noise in the user's recording environment is less than 30 decibels (dB). Background noise analyzer 2603 may also check for speech (such as coming from television or radio). The background noise analyzer 2603 may also check for intermittent spikes or similar acoustic artifacts that may be the result of, for example, a child's scream, a loud clock ticking, or a mobile device notification.

In some embodiments, background noise analyzer 2603 may perform a background noise check after recording has begun. In one such embodiment, the background noise check is performed at a predetermined time interval prior to detection of the first phoneme in the recording (detectable as described in conjunction with phoneme splitter 2610). is performed on a portion of the audio data received within. For example, background noise analyzer 2603 may perform a background noise check for 5 seconds prior to the beginning of the first phoneme in the audio data.

If background noise is detected, background noise analyzer 2603 may process (or attempt to process) the audio data to reduce or remove the noise. Alternatively, an indication of the noise determined by the background noise analyzer 2603 may be provided to the signal preparation processor 2606 for noise reduction or removal by performing a filtering and/or subtraction process. In some embodiments, in addition to or as an alternative to automatically reducing or removing background noise, the background noise analyzer 2603 determines whether background noise is interfering with or has the potential of interfering with audio collection. An indicator may be sent that informs the user (or other components of system 200, such as user interaction manager 280), and requests action from the user to eliminate the background noise. . For example, a notification may be provided to the user (eg, via user interaction manager 280 or presentation component 220) to move to a quieter environment.

In some cases, after the audio data is acquired, the background noise analyzer 2603 may recheck the audio data for the presence or absence of background noise. For example, after the recording optimizer 2602 (or in some embodiments, the signal creation processor 2606) automatically adjusts settings to reduce or eliminate noise, another check is performed. You can. In some aspects, at the beginning of a recording session, after a predetermined period of time since the last check, and/or when an instruction is received from a user or the like to perform background noise reduction or removal. Subsequent checks may be performed as required.

Within user voice monitor 260, voice sample collector 2604 may be generally responsible for obtaining user voice-related data in the form of audio samples or recordings. Audio sample collector 2604 may work in conjunction with data collection component 210 and user interaction manager 280 to obtain samples of audio information, such as user utterances. The audio samples may be in the form of one or more audio files containing recordings or samples of sustained phonemes, planned utterances, and/or unplanned utterances. As used herein, the term audio recording generally refers to a digital recording (eg, of audio samples that may be determined by audio sampling using analog-to-digital conversion (ADC)).

In some embodiments, audio sample collector 2604 may include functionality such as ADC conversion functionality to capture and process digital audio from analog audio (which may be received from sensor 103 or analog recording). Thus, some embodiments of audio sample collector 2604 may provide or facilitate determination of digital audio samples. Additionally, in some embodiments, the audio sample collector 2604 may associate date and time information with audio samples that correspond to the time frame in which the audio data was acquired (e.g., date and/or time information). timestamp audio samples). In one embodiment, the audio sample may be stored in a patient record associated with the user, such as audio sample 242 in patient record 240.

With respect to user interaction manager 280, audio samples 242 may be obtained in response to the user's engagement in speech-related tasks, as shown in the examples of FIGS. 4A-4C and 5B. . For example, the user may continue to utter a particular sound (e.g., "mmmm") for a period of time or as long as possible, repeat a particular word or phrases, etc., to obtain the audio sample 242; You may be asked to read text, or be prompted to answer questions or participate in a conversation. Audio samples 242 representing different types of speech-related tasks may be obtained from the user in the same collection session. For example, a user may be asked to continue speaking one or more phonemes for a particular time interval, and to continue speaking one or more phonemes for as long as possible, with the latter phoneme may be the same as or different from the phoneme retained over a particular time interval. Also, in some embodiments, a user may be asked to read written text that may have a variety of phonemes.

As used herein, a voice sample represents voice-related information in an audio sample and may be determined from the audio sample as described herein. For example, the audio samples may include other acoustic information not associated with the user's voice, such as background noise. Thus, in some cases, the audio sample may represent a portion of the audio sample that includes audio-related information. In one embodiment, the audio samples are determined from audio collected during a user's daily interactions or daily interactions with user computing equipment (e.g., user equipment 102a of FIG. 1). You can. For example, voice samples may be collected when a user gives spontaneous commands to a smart speaker or has a phone conversation. In some embodiments, if voice sample information is obtained from the user's daily interactions with user equipment, there may be no need to prompt the user to engage in speech-related tasks. Similarly, in some embodiments, the user may be able to communicate through the user's utterances through speech that is not obtained from everyday interaction, such as when information about a particular phoneme is not obtained from everyday interaction utterances. You may also be prompted to complete speech-related tasks to obtain sample information.

As mentioned above, the techniques described herein enable the preservation and protection of user privacy. In embodiments that obtain audio samples from daily interactions with user equipment, it is also contemplated that the audio data may be erased once voice-related data for respiratory disease monitoring has been determined. It is likewise possible to encrypt audio data and/or "participate" in the collection of voice-related data (for respiratory disease monitoring) by the user from so-called daily interactions.

Signal creation processor 2606 may generally be responsible for creating audio samples for extracting speech-related information, such as phoneme characteristics, for further analysis. Accordingly, signal creation processor 2606 may perform signal processing, preprocessing, and/or conditioning on audio data obtained or determined by audio sample collector 2604. In one embodiment, signal generation processor 2606 may receive audio data from voice sample collector 2604 and access voice sample data from voice samples 242 in patient record 240 associated with the user. You can do it like this. Audio data created or processed by signal creation processor 2606 may be stored as audio samples 242 and/or provided to other subcomponents of user audio monitor 260 or other components of system 200. You can leave it there.

In some embodiments, there may be certain phoneme characteristics or audio information in some, but not all, frequency bands of the audio data that are utilized to monitor a user's respiratory disease. Accordingly, some embodiments of the signal creation processor 2606 are configured to perform frequency filtering, such as high-pass or band-pass filtering, to remove or attenuate frequencies of the audio signal that are not useful, such as low frequency background noise. You may also do so. Signal frequency filtering also improves computational efficiency by reducing audio sample size and improves sample processing time. In one embodiment, signal creation processor 2606 may apply a 1.5-6.4 kilohertz (kHz) bandpass filter. In one exemplary embodiment of the computer program routine provided in FIGS. 15A-15M, a Butterworth bandpass filter is utilized (as shown in FIG. 15A). In one example, signal creation processor 26066 may smooth out outliers and normalize characteristics by applying a rolling median filter. A rolling media filter can be applied by using a window of these samples. A z-score may be used to normalize the characteristic values.

Signal preparation processor 2606 may also perform audio normalization to target signal amplitude levels, apply bandpass filters and/or amplifiers to improve signal-to-noise ratio (SNR), or perform other signal conditioning or preprocessing. It may be realized. In some embodiments, signal creation processor 2606 may process the audio data to remove or attenuate background noise, such as the background noise determined by background noise analyzer 2603. For example, in some embodiments, the signal creation processor 2606 uses the background noise information determined by the background noise analyzer 2603 to perform noise canceling operations (or subtract or subtract background noise including noise artifacts). Attenuation) may also be performed.

In user audio monitor 260, sample recording checker 2608 may generally be responsible for determining whether sufficient audio samples (or voice samples) have been acquired. Accordingly, the sample recording examiner 2608 may determine that the sample recording has the shortest duration and/or contains particular audio-related information, such as utterances or vocal sounds. In some embodiments, sample recording examiner 2608 may apply criteria to validate audio samples based on particular phonemes or phoneme characteristics to be detected. Thus, some embodiments of the sample recording checker 2608 may perform phoneme detection on audio data and may be configured to work in conjunction with subcomponents of the user speech monitor 260, such as the phoneme splitter 2610. It may be . In some embodiments, sample recording examiner 2608 may determine whether an audio sample (or, in some cases, an audio sample within an audio recording) satisfies a threshold length of time. The threshold time length may vary depending on the particular type of speech-related task being recorded, and may vary depending on the particular phoneme or phoneme characteristics that are sought to be obtained from the speech sample and how these characteristics are determined in the current session or time frame. It may also be based on the extent to which it has already been completed. In one embodiment, if the user is prompted (e.g., by the user interaction manager 280) to record the reading of a sentence during a session in which a user audio sample is obtained, the sample recording examiner 2608 may record a subsequent audio sample. It may be determined whether the length of the recording is at least 15 seconds. Additionally, in one embodiment, the sample recording examiner 2608 is configured to determine whether a particular audio sample includes sustained utterances of sufficient duration, such as at least 4.5 seconds in length. good. Similarly, for embodiments that obtain audio data or voice samples (such as 242) from routine interactions with user computing equipment (such as user equipment 102a), sample recording examiner 2608 may include the determination of phonemes or phoneme characteristics. It may be determined that a specific audio sample used for separate analysis, such as, satisfies a threshold duration time and/or includes specific sound or phoneme information. Recordings or audio samples that do not meet inspection criteria (eg, minimum threshold duration) are considered incomplete and may be deleted or not processed further. In some embodiments, the sample record examiner 2608 provides an indication to the user (or user interaction manager 280, presentation component 220, or other system 200) that a particular sample is incomplete or insufficient. component) and may also indicate that the user needs to re-record a particular audio sample.

In some embodiments, sample recording examiner 2608 analyzes multiple audio samples (from audio samples 242), each of which may represent the same (or similar) audio-related information within a time frame (i.e., within a session). The audio samples may be selected from the available sources. In some cases, after this selection, other unselected samples may be deleted or discarded. For example, if multiple complete recordings of the desired phoneme exist for a given point in time or time segment (which may have been generated by the user repeating a particular speech-related task), sample recording inspection The device 2608 may select the most recently acquired recording (the last recorded one) for analysis, as this may be due to technical issues with the previous recording. It may also be executed on the premise that it has been re-recorded. Alternatively, sample recording examiner 2608 may select audio samples based on sound parameters such as minimum noise and/or maximum volume.

Determining a sufficient audio sample recording for separate processing may also include determining that no noise artifacts are present, that only minimal noise artifacts are present, and/or that the recording contains at least approximately correct sound or that the specified instructions are followed. may also include a determination of In some embodiments, the sample recording checker 2608 may determine whether the SNR of the audio sample meets a maximum allowed SNR, such as 20 decibels (dB). For example, the sample recording examiner 2608 may determine that the SNR of the recording is greater than a 20 dB threshold, and may also set an indicator to the user (or user interaction) that requests the acquisition of new audio samples from the user. (another component of system 200, such as action manager 280).

In some embodiments of the sample recording examiner 2608, certain sustained vocalizations (e.g.,

) etc., it may be determined whether a sample sound corresponding to the requested speech-related task exists. In particular, if the audio sample is obtained by the user performing a speech-related task (e.g., "saying 'Um' for 5 seconds"), the sample contains the sound (or phoneme) requested in the task. The verification or testing of the audio sample may be performed to determine whether the audio sample The determined phoneme in the sample may be compared to the requested sound or phoneme (ie, the "labeled" phoneme or sound). If a mismatch is determined or the labeled phoneme or sound is not detected in the sample, the sample recording examiner 2608 sends an indication to the user (or user interaction manager) so that the correct audio sample can be reacquired. 280). Additional details of ASR are described below in connection with phoneme splitter 2610.

Some embodiments of the sample recording examiner 2608 do not necessarily determine the presence or absence of a particular phoneme in an audio sample, but may determine that the sample captures a sustained phoneme or combination of phonemes. You may also do so. The sample recording checker 2608 may also determine whether a phoneme persists in the audio sample for a minimum duration. In one embodiment, the minimum duration may be 4.5 seconds.

The sample recording examiner 2608 may further perform trimming, cutting, or filtering to remove unnecessary and/or unusable portions of the audio sample recording. In some embodiments, sample recording examiner 2608 may perform such operations in cooperation with signal generation processor 2606. For example, sample record examiner 2608 may trim the start and end (eg, 0.25 seconds) from each record. The usable portion of the audio sample may include sufficient audio-related data to determine phoneme or characteristic information by further processing. In some embodiments, the sample recording examiner 2608 (or other subcomponents of the audio sample collector 2604 and/or user audio monitor 260) determines that the audio sample is usable by cutting or trimming the audio sample. It is also possible to retain only a portion. Similarly, sample recording examiner 2608 may facilitate determining a usable portion of audio samples among multiple samples (such as audio samples 242) that may be acquired within the same time frame (i.e., within a recording session). .

Sample recording examiner 2608 may be configured to receive audio sample data from audio sample 242 or another subcomponent of user audio monitor 260 and store processed or modified audio sample data in audio sample 242. Alternatively, processed or modified audio sample data may be provided to another subcomponent of user audio monitor 260. In some cases (such as when an unusable portion is recorded or the recording is incomplete after removal), the sample recording examiner 2608 determines whether a new recording or audio sample needs to be acquired and provides an indicator. may be provided to the user, as described below with respect to user interaction manager 280.

Phoneme segmenter 2610 may generally be responsible for detecting the presence or absence of individual phonemes in a speech sample and/or determining timing information for the presence of individual phonemes in a speech sample. For example, the timing information may include the onset time (i.e., start time), duration, and/or end time (i.e., stop time) for the occurrence of a phoneme in the audio sample, which may be used for feature analysis. It may be used to facilitate the identification and/or separation of phonemes for use. In some cases, start and stop time information may be referred to as phoneme boundaries. As previously discussed, speech samples may include recordings (eg, audio samples) of sustained individual phonemes or combinations of phonemes, such as planned and unplanned utterances, uttered by a user. For example, an audio sample may be generated when a user utters the word "spring," and this audio sample may be configured to generate individual phonemes (e.g.,

). In some cases, continuous segmentation of the audio samples of individual phonemes may be used to separate the phonemes from other samples.
In some aspects, phoneme segmenter 2610 detects phonemes (e.g., logically or from the audio sample using timing information available as pointers or references to phonemes in the audio sample). The phonemes may be further separated (physically, such as by copying or extracting the phoneme-related data of the above). Phoneme detection by phoneme segmenter 2610 may include determining that a speech sample (or portion of a speech sample) has a particular phoneme or a phoneme in a particular set of phonemes. Voice sample data may be received from voice samples 242 or another subcomponent of user voice monitor 260. The particular phonemes detected by phoneme segmenter 2610 may be based on the phonemes analyzed for the user's respiratory illness. For example, in some embodiments, phoneme segmenter 2610 is configured such that the sample(s) are

Alternatively, it may be detected whether or not a phoneme corresponding to is included. In another embodiment, phoneme segmenter 2610 is configured such that the sample(s) are

Alternatively, it may be determined whether or not a phoneme corresponding to is included. In other embodiments, phoneme segmenter 2610 may detect other phonemes or phoneme sets that may include phonemes of any spoken language.

Some embodiments of phoneme segmenter 2610 utilize automatic speech recognition (ASR) (referred to as "speech recognition") functionality to determine phonemes from a portion of a speech sample. ASR functionality may also utilize one or more acoustic models or speech corpora. In one embodiment, hidden Markov models (HMMs) are utilized in processing a speech signal corresponding to a user's speech samples to determine a set of one or more likely phonemes. You can leave it there. In another embodiment, an artificial neural network (ANN) (sometimes referred to herein as a "neural network"), other acoustic models for ASR, or a combination of these models is used. technology may be used. For example, neural networks may be utilized as a pre-processing step in ASR to perform dimensionality reduction or characteristic transformation prior to application of the HMM. Some embodiments of the operations performed by phoneme segmenter 2610 to detect or identify phonemes from speech samples include a speech recognition engine or ASR that may include software packages, modules, or libraries for processing speech data. ASR functionality or acoustic models provided via a software toolkit may be utilized. Examples of such speech recognition software tools include kaldi-asr. CMU Sphinx, developed at Carnegie Mellon University, and the Hidden Markov Model Toolkit (HTK), developed at Cambridge University.

In some embodiments of acquiring audio samples as described herein, the user may perform speech-related tasks, such as the repetitive sound exercise described in connection with FIG. 5B. may be part of the evaluation exercise. Some of these speech-related tasks may require the user to continue saying a particular sound or phoneme. Additionally or alternatively, speech-related tasks may require the user to continue saying a particular sound or phoneme for as long as possible. Further, various tasks may be used for different phonemes. For example, in one embodiment, the user may say "aah" (or

) over a predetermined period of time, such as 5 seconds, or another sound or phoneme (e.g.

) may be required to continue saying. In some embodiments, multiple types of speech-related tasks may be collected for the same phoneme.
The audio samples generated by performing this task may be labeled with or associated with the sounds or phonemes that the user is requested to utter. For example, if a user is prompted to say "Oooh" for 5 seconds, the recorded audio sample will contain the "Oooh" sound (or phoneme).

) or may be associated with a "woom" sound.
In some embodiments, the phoneme segmenter 2610 utilizes ASR functionality to generate audio that may be received from user utterances obtained by performing speech-related tasks or by daily interaction with user equipment. A particular sound or phoneme in the sample may be determined. In these embodiments, once the sounds or phonemes of the audio sample are determined, the audio sample (or portion of the sample) may be labeled or associated with a sound or phoneme. In an exemplary embodiment, the phoneme segmenter 2610 detects an "aah" sound if it determines that the audio sample obtained from the user has an "aah" sound occurring in a particular portion of the sample. , the portion of the audio sample may be labeled accordingly (eg, by association of the label with the audio sample or portion thereof in the database). In another embodiment, phoneme segmenter 2610 may determine timing or phoneme boundaries in an audio sample by separating phonemes.

In some embodiments, the phoneme segmenter 2610 may separate phonemes by identifying the start time, duration, and/or stop time of a phoneme boundary or a segment within the audio sample that captures the phoneme. good. In some embodiments, phoneme segmenter 2610 first detects the presence or absence of a particular phoneme and then separates the particular phoneme (e.g.,

etc). In an alternative embodiment, phoneme segmenter 2610 may detect the presence of a particular phoneme in a speech sample and separate all detected phonemes. Some embodiments of phoneme segmenter 2610 may utilize phoneme segmentation or phoneme alignment tools to facilitate determining the temporal location of phonemes or phoneme boundaries in an audio sample. Examples of such tools include the computer software package Praat for speech analysis and phonetics developed at the University of Amsterdam, and/or EasyAlign for performing phoneme alignment developed at the University of Geneva; Examples include functions provided by software modules that work with.

In an exemplary aspect, phoneme segmenter 2610 may perform automatic segmentation by applying a threshold to detected intensity levels in the audio sample. For example, sound intensity may be calculated across the recording and a threshold applied to separate background noise from high-energy events (representing speech events) in the sample. In one embodiment, the calculation of sound intensity may be performed using functionality provided by the computer software package Praat for speech analysis and phonetics. Figures 15A-15M illustrate one such example using Praat and are shown using the Parselmouth Python library. According to one embodiment, the phoneme segmentation threshold may be determined by Otsu's method. In some embodiments, this threshold may be determined for each audio sample, such that different thresholds may be determined and applied to different audio samples of the same user. Once the sound intensity level is computed and a threshold is determined, the phoneme divider 2610 applies this threshold to the computed intensity level to detect the presence or absence of a phoneme, and to detect the presence or absence of a phoneme, as well as to detect the beginning and end of the detected phoneme, respectively. Further identification of time and stop time may be provided. Some embodiments include validating the automatic segmentation performed by phoneme segmenter 2610 through the use of manual segmentation on at least a portion of the audio samples.

In some embodiments, a morphological "filling" operation may be used to fill gaps within segments detected as phonemes. The gap may be filled if its duration is less than a maximum threshold, such as 0.2 seconds. Embodiments of phoneme segmenter 2610 may also trim one or more portions of detected phonemes. For example, phoneme segmenter 2610 may avoid transient effects by trimming or ignoring the initial duration, such as the first 0.75 seconds, of each detected phoneme. Therefore, the start time of the detected phoneme may be changed so that the detected phoneme does not include the first 0.75 seconds. Additionally, in some embodiments, each detected phoneme may be trimmed such that the total duration of the phoneme is two seconds or some other set duration.

In some embodiments, data quality checks may be performed on the segmented phonemes. These data quality checks may be performed by phoneme segmenter 2610 or by other components of user speech monitor 260, such as signal preparation processor 2606 and/or sample recording examiner 2608. It is also possible to make it look like this. In one embodiment, the signal-to-noise ratio (SNR) is estimated for each phoneme segment as the ratio of the average intensity within the detection segment divided by the average intensity outside the detection segment. Furthermore, a predetermined segment duration threshold may be applied to determine whether the detected phoneme satisfies the minimum duration. Another quality check may include determining the correct number of phonemes by comparing the number of detected phonemes to the expected number of phonemes, which may be based on a prompt for a voice sample from the user. For example, in one embodiment, the correct number of phonemes may include three split phonemes for sustained nasal recordings and four split phonemes for sustained vowel recordings. In an exemplary aspect, if the correct number of phonemes is found (e.g., three for sustained nasal recordings and four for sustained vowel recordings), if the SNR is greater than 9 dB; and the duration of each phoneme is 2 seconds or more, the segmented audio sample may be determined to be of high quality. In some embodiments, additional quality checks may be performed on vowel speech samples, including determining whether the first formant frequency falls within an acceptable range. may include determining. If it falls within the acceptable range, the sample is determined to be of high quality. If not, an indication is provided (which may be provided to the user interaction manager 280) that the sample is insufficient, incomplete, or requires reacquisition. ).

Continuing with user speech monitor 260, acoustic feature extractor 2614 may generally be responsible for extracting (or determining) features of phonemes within a speech sample. The phoneme characteristics may be extracted from the audio samples at a predetermined frame rate. In one example, features are extracted at a rate of 10 milliseconds. The extracted characteristics may be utilized to track a user's respiratory illness as described further with respect to respiratory illness tracker 270. Examples of extracted acoustic characteristics include, by way of non-limiting example, data characterizing power and power variability, pitch and pitch variability, spectral structure, and/or formant measures.

Other examples of properties related to power and power variability (which may also be referred to as amplitude-related properties) include the root mean square (RMS) of acoustic power per segment phoneme, shimmer, and power variability in the 1/3 octave band. Can be mentioned. In some embodiments, the RMS of sound power is computed and utilized for data normalization prior to extraction of any other sound characteristics. The RMS may also be converted to decibels for consideration as a power-related characteristic itself. Shimmer captures the rapid variability in waveform amplitude measured at the glottal pulse interval. Further, the power fluctuation in the output of the 1/3 octave band filter may be calculated at various frequencies. In one exemplary embodiment, the extraction characteristics may exhibit variations in a 200 Hertz (Hz) 1/3 octave band, which may be determined by applying a passband frequency of 178-224 Hz.

Other examples of characteristics related to pitch and pitch variability include pitch coefficient of variation (COV) and jitter. To extract the pitch coefficient of variation, the average pitch (pitch _mn ) and pitch standard deviation (pitch _sd ) are determined over each segment, and the pitch coefficient of variation (pitch _cov ) is calculated as pitch _cov = pitch _sd /pitch _mn can be done. In some embodiments, particularly when the audio samples are noisy, a coefficient of variation threshold may be applied to ensure that the estimated pitch value is calculated for the appropriate frequency of the user's audio data. For example, it may be determined whether the coefficient of variation is less than a threshold of 10% of the coefficient of variation value (determined empirically), and segments with values greater than the threshold may be treated as missing data. Jitter can be thought of as capturing pitch variability on a shorter time scale. The jitter may be extracted in the form of local jitter or local absolute jitter. In some aspects, pitch-related characteristics are extracted from each segment by using an autocorrelation method. An example of autocorrelation for determining pitch-related properties is given by the computer software package Praat for speech analysis and phonetics developed at the University of Amsterdam. 15E and 15F illustrate aspects of an exemplary computer program routine of one embodiment that utilizes Praat functionality in this manner.

Some embodiments of acoustic feature extractor 2614 (or user voice monitor 260) may perform processing operations to adjust the pitch floor prior to extraction of pitch-related features by acoustic feature extractor 2614. . For example, the pitch floor may be as high as 80 Hz for male users and 100 Hz for female users to prevent false pitch detections. According to one embodiment, an increase in pitch floor may be guaranteed in the presence of low frequency periodic background noise. The determination of whether to adjust the pitch floor may depend on the system collecting the audio data, the environment in which the audio data is collected, and/or application settings (eg, settings 249).

Properties related to spectral structure include harmonic-to-noise ratio (HNR (sometimes referred to as "harmonicity"), spectral entropy, spectral contrast, spectral flatness, voice low-to-height ratio (VLHR), and Mel frequency cepstral coefficients. (MFCC), cepstral peak prominence (CPP), percentage or proportion of voiced (or unvoiced) frames, and linear prediction coefficient (LPC). It is the ratio of the power of the harmonic components to the power and represents the degree of acoustic periodicity. An example of determining HNR is shown in the computer programming routine of FIG. 15E, which uses a computer software package to determine harmonicity. Utilizes the functionality provided by Praat. Spectral entropy indicates the entropy of the spectrum in a particular frequency band. Spectral contrast sorts the power spectral values by the intensity of a particular frequency band, and the lowest quartile of the frequency band. Spectral flatness can be determined by calculating the ratio of the highest quartile (peak) value to the (trough) value. Spectral flatness can be determined by calculating the ratio of the geometric mean to the arithmetic mean of the spectral values in a given frequency band. Spectral entropy, spectral contrast, and spectral flatness may each be calculated for a particular frequency band. In one embodiment, the spectral entropy is between 1.5 and 1.5. 2.5 kilohertz (kHz) and 1.6 to 3.2 kHz, spectral flatness determined from 1.5 to 2.5 kHz, and spectral contrast from 1.6 to 3.2 kHz and 3.2 to 6 .4kHz.

VLHR can be determined by calculating the ratio of integrated low and high frequency energy. In one embodiment, the separation between low and high frequencies is fixed at 600Hz. Therefore, this characteristic may be designated as VLHR600.

The Mel frequency cepstral coefficients (MFCC) represent the discrete cosine transform of the scaled power spectrum, and the MFCC collectively constitutes the Mel frequency cepstral coefficient (MFC). MFCCs are typically sensitive to spectral changes and robust to environmental noise. In an exemplary embodiment, a mean MFCC value and a standard deviation MFCC value are determined. In one embodiment, mean values are determined for Mel frequency cepstral coefficients MFCC6 and MFCC8, and standard deviation values are determined for Mel frequency cepstral coefficients MFCC1, MFCC2, MFCC3, MFCC8, MFCC9, MFCC10, MFCC11, and MFCC12.

Voicedness refers to periodicity in recorded utterances, and some aspects of this disclosure include determining the rate, ratio, or ratio of frames of a voiced utterance recording. Alternatively, this characteristic may be determined using unvoiced frames. In some cases for the determination of voiced (or unvoiced) frames, a predetermined pitch threshold may be applied such that a proportion of voiced or unvoiced frames is calculated for frames suspected of being spoken. . In some embodiments, the proportion or ratio of voiced (or unvoiced) frames may be determined using the computer software package toolkit Praat for audio processing.

Other characteristics extracted or determined by acoustic feature extractor 2614 may be related to one or more acoustic formants representative of vocal tract resonance. In particular, in the case of phonemes in a speech sample, the average formant frequency and the standard deviation of the formant bandwidth may be calculated for one or more formants. In an exemplary embodiment, the average formant frequency and standard deviation of formant bandwidth are computed for formant 1 (denoted as F1). However, it is contemplated that additions or substitutions such as formants 2 and 3 (denoted as F2 and F3) may be utilized. In some aspects, formant characteristics may act as data quality control by facilitating automated checks that may be performed by sample recording checker 2608 to ensure that a user pronounces sounds correctly.

It is contemplated that in some embodiments, each of the above acoustic characteristics may be extracted or determined for a different phoneme. For example, in one embodiment, seven phonemes (

), 23 of the above characteristics (not including the RMS of amplitude) are determined, resulting in 161 unique phoneme characteristics. Some embodiments of the present disclosure may include identifying or selecting a set of characteristics for separate analysis. For example, one embodiment includes determining a total of 161 characteristics from one or more audio samples or reference audio data and selecting particular characteristics that are believed to be relevant to monitoring a user's respiratory infectious disease. Alternatively, it may include determining.

Also, one or more of these acoustic characteristics may be extracted from audio samples from a particular type of speech-related task. For example, the characteristics may be determined for phonemes extracted from utterances of a predetermined duration. Furthermore, one or more of the above-mentioned characteristics may be determined for the utterance extracted by reading out the text by the user. In some embodiments, other characteristics may be extracted from particular types of speech-related tasks. For example, in an exemplary embodiment, the maximum vocalization time that can be used as a measure of respiratory capacity may be determined from an audio sample of sustained vocalizations in which the user holds the sound for as long as possible. . As used herein, maximum utterance time represents the duration that a user sustains a particular utterance.

Furthermore, in some embodiments, amplitude changes in sustained utterances may be determined for audio samples of the type described above. In some exemplary embodiments, other acoustic characteristics are determined from the text audio sample. For example, speech rate, average pause length, number of pauses, and/or global SNR may be determined from recording or monitoring a user's reading of sentences. Speech rate may be determined as the number of syllables or words per second. The pause length may represent a pause in the user's speech that is at least a predetermined minimum duration, such as 200 milliseconds. In some aspects, the text is generated from a user's audio sample, determining timestamps for when the user starts a word and when the user ends a word, and using these timestamps to In order to determine the duration of the pauses used to determine the average pause length and/or number of pauses, an automatic speech-to-text algorithm may be used to determine the pauses. Global SNR can be thought of as the signal-to-noise ratio over the entire recording, including periods without speech.

Additionally, certain properties or combinations of properties may be more suitable for monitoring certain types of respiratory infections. Embodiments of trait selection may include identifying possible trait combinations, calculating a distance metric between trait sets or vectors of different dates, and correlating the distance metric to self-reported ratings of respiratory symptoms. . In one example, the first six principal components of possible phoneme combinations (e.g., shown in FIGS. 11A and 1IB as exemplary phoneme combinations) are computed, and across each pair of dates on which audio data was collected. Principal component analysis (PCA) is utilized to calculate distance metrics, such as Euclidean distances, between vectors representing acoustic properties for combinations of phonemes. Additionally, a Spareman's rank correlation may be calculated between the distance metric and self-reported symptom ratings for each day relative to the last day representing a good condition.

Additionally, in some embodiments, unsupervised feature selection is performed by applying sparse PCA to further reduce the dimensionality of the dataset. Alternatively, in some embodiments, linear discriminant analysis (LCA) may be utilized for dimensionality reduction. In some embodiments, features (specifically phoneme and feature combinations) in the top principal components (as determined empirically) with non-zero weights are selected for further analysis. You can leave it there. Aspects of characteristic selection are discussed in detail below in conjunction with FIGS. 7-14.

In an exemplary embodiment, the representative phoneme feature set determined by the feature selection described in connection with FIGS.

12 characteristics, phonemes

12 characteristics of and phonemes

Contains 32 phoneme characteristics, including 8 characteristics. Thirty-two of these exemplary properties are listed in the table below.

As shown in the table above, acoustic feature extractor 2614 may transform the values of one or more features for normality. For example, a log transformation (denoted as LG) may be applied to a subset of characteristics. Other properties may not include transformations. Additionally, although not included in the above table, it is also contemplated that other transformations may be applied, such as a square root transformation (SRT). In one embodiment, characteristic selection includes selecting transformations for one or more various characteristics. In one example, different types of transformations or no transformations, such as SRT, LG, etc., are tested for one or more characteristics, and the Shapiro - The Wilk test may be used.

In some embodiments, acoustic feature extractor 2614, phoneme segmenter 2610, or other subcomponents of user speech monitor 260 implement speech-to-phoneme extraction logic 233 (as shown in storage 250 in FIG. 2). This may be used to determine phonemes or extract characteristics of phonemes. Speech-to-phoneme extraction logic 233 may include instructions, rules, conditions, associations, machine learning models, or other criteria for identifying and extracting acoustic feature values from acoustic data corresponding to segmented phonemes. In some embodiments, speech-to-phoneme extraction logic 233 utilizes ASR functionality, acoustic models, or related functionality described in connection with phoneme segmenter 2610. For example, various classification models or software tools (e.g., HMMs, neural network models, and other software tools mentioned above) may be utilized to identify specific phonemes in audio samples and determine the corresponding acoustic characteristics. It may be . An exemplary embodiment of the acoustic feature extractor 2614 or speech-phoneme extraction logic 233 may include or utilize functionality provided in the speech analysis and phonetics computer software package Praat. Aspects of one such embodiment (including computer program routines) are illustratively provided in FIGS. 15A-15M, which are illustrated using the Parselmouth Python library to access the software package Praat.

After determining the phoneme properties, the acoustic feature extractor 2614 generates a phoneme feature set that may include a phoneme feature vector (or set of phoneme feature vectors) for the phonemes determined from the user audio samples corresponding to the recording session or time frame. It may be decided. For example, a user may provide audio samples twice a day (e.g., a morning session and an evening session), with each session extracting phonemes from the audio samples captured during that session. or may correspond to a phoneme feature vector or set of vectors representing the determined properties. The phoneme feature set may be stored in the patient record 240 associated with the user, such as a phoneme feature vector 244, and may also be associated with the date or time at which the speech sample used to determine the phoneme feature was obtained. It may also be stored or associated with corresponding date and time information.

In some cases, the terms "feature set" and "feature vector" may be used interchangeably herein. For example, to facilitate performing comparisons between two sets of properties, distance measurements between corresponding properties of each vector can be determined (i.e., property vector comparison), or to properties of other operations. To facilitate the application of , the member characteristics of the set are considered as characteristic vectors. In some embodiments, phoneme feature vector 244 may be normalized. In some embodiments, the characteristic vector may be a multidimensional vector, with each phoneme having dimensions that represent the characteristic. In some embodiments, multidimensional vectors may be flattened, such as before determining a comparison between two characteristic vectors, as described in connection with respiratory disease tracker 270. good.

In addition to determining acoustic characteristics, some embodiments of user audio monitor 260 may include a context information determiner 2616 that determines context information associated with the characterized audio samples. The context information may indicate, for example, the conditions under which the audio sample was recorded. In an exemplary embodiment, the context information determiner 2616 includes a date and/or time (i.e., a timestamp) of the recording that may be stored or associated with the phoneme feature vector generated by the acoustic feature extractor 2614 or a The duration may also be determined. The information determined by the context information determiner 2616 may be related to the extracted acoustic characteristics as well as tracking of the user's respiratory illness. For example, the context information determiner 2616 may determine the specific time of day (e.g., morning, noon, or night) during which the audio samples are obtained and/or environmental or atmospheric related information (e.g., weather, humidity, and/or pollution levels). A user location that may be determined may also be determined. Additionally, in one embodiment, the duration of the audio sample may be used to track a user's respiratory illness. For example, the user may want to play the "aah" sound (i.e., the phoneme for as long as possible).

), and determining a user's respiratory illness may use a duration metric that measures how long the user can continue to say this sound.

In some embodiments, the context information determiner 2616 may be configured to determine or receive physiological information about the user that may be associated with the time frame in which the audio samples were acquired. For example, the user may provide information regarding the symptoms he or she is experiencing, as illustrated and described in the embodiments shown in FIGS. 4D, 5D, and 5E. In some cases, context information determiner 2616 may obtain symptom data in conjunction with user interaction manager 280, as described below. In some embodiments, the context information determiner 2616 determines temperature or blood pressure on a wearable user device (e.g., a fitness tracker) from a user's profile/health data (EHR) 241 or a sensor (such as 103 in FIG. 1). Physiological data such as oxygen concentration may also be received.

In some embodiments, the context information determiner 2616 may determine whether the user is on medication and/or whether the user has taken medication. This determination is based on the user providing an explicit signal, such as selecting an indicator on a digital application that indicates the user has taken the medication or responding to a prompt from a smart device asking if the user has taken the medication. Alternatively, the information may be provided by another sensor, such as a smart pill box or medicine chest, or by another user, such as the user's caregiver. In some embodiments, the context information determiner 2616 provides information about the user, physician, or health care provider by accessing the user's electronic health record (EHR) 241, email, or message indicating prescription or purchase and/or purchase information. It may be determined whether the user is on medication based on information provided by a care provider or caregiver. For example, a user or healthcare provider may prompt the user to specify a particular medication or course of treatment via a digital application, such as the exemplary respiratory infection monitoring app 5101 described in conjunction with FIG. 5D. You may also do so.

The context information determiner 2616 may further determine the user's geographic region (eg, via a location sensor on the user equipment or input of location information by the user, such as a zip code). In some embodiments, the context information determiner 2616 determines which viruses or bacteria (such as influenza or COVID-19) are present in the user's geographic region and are known to cause respiratory infections. The degree may be further determined. Such information may be available from government or health care agency websites or web portals, such as those operated by the Centers for Disease Control and Prevention (CDC), the World Health Organization (WHO), state health departments, or national health agencies. Conceivable.

The information determined by the context information determiner 2616 may be stored in the patient record 240, and in some embodiments, the information determined by the context information determiner 2616 may be stored in a relational database and may also be stored in the patient record 240. The context information may be associated with a speech sample or a specific phoneme feature vector determined from the speech sample.

As mentioned above, user speech monitor 260 may generally be responsible for obtaining relevant acoustic information from audio samples of the user's speech. This data collection may include instructions for interaction with the user. Accordingly, embodiments of system 200 may further include a user interaction manager 280 that facilitates the collection of user data, including obtaining voice samples and/or user symptom information. As such, embodiments of user interaction manager 280 may include a user instruction generator 282, a self-reporting tool 284, and a user input response generator 286. As described later herein, user interaction manager 280 includes user audio monitor 260 (or one or more of its subcomponents), presentation component 220, and, in some embodiments, It may be adapted to cooperate with report data evaluator 276.

User instruction generator 282 may generally be responsible for guiding the user toward providing audio samples. User instruction generator 282 may provide instructions to the user for capturing audio data (e.g., via a graphical user interface as shown in the example of FIG. 5A or the example interaction of FIG. 4C). utterances may be facilitated via an audio or voice user interface such as that shown in Figure 1). Among other things, user instruction generator 282 may read and/or utter instructions 231 to the user (eg, "Say 'Ah' for 5 seconds"). These instructions 231 may be pre-programmed and specific to the phonemes, speech-related data, or other user information desired by the user. In some cases, instructions 231 may be determined by the user's clinician or caregiver. Thus, according to some embodiments, instructions 231 may be specific to the user and/or respiratory infection or medication (eg, as part of treatment as a patient). Alternatively or additionally, instructions 231 may be automatically generated (eg, synthesized or constructed). For example, the instruction 231 requesting a specific phoneme may be generated based on a determination that characteristic information regarding a specific phoneme is necessary or useful for determining a user's respiratory disease. . Similarly, a set of predetermined instructions 231 or actions (e.g., from a clinician or caregiver) may be provided (or such as 105a or 105b) for constructing specific or adapted instructions for the user. may be programmed into a decision support application) and adapted for use.

The preprogrammed or generated instructions 231 may be specific, such as uttering a particular phoneme for a set duration, continuing to utter a particular phoneme for as long as possible, uttering a particular word or combination of words, or reading a sentence aloud. may be related to performing speech-related tasks. In some embodiments where a user is required to read a text aloud, the text of the text may be provided to the user so that the user can read the provided text aloud. Additionally or alternatively, portions of the text may be output audibly to the user so that the user can repeat the audible text without reading the text. In one embodiment, the user speaks aloud (by reading written text or repeating spoken instructions) a predetermined phonologically balanced sentence (such as a rainbow passage) to create a rainbow passage. You may be asked to read a part of the text, such as the five lines of a passage. In some cases, the user may be given a predetermined amount of time, such as two minutes, to complete reading the text.

In some embodiments, the instructions 231 may provide sample sounds of the phonemes that the user is instructed to provide. In some embodiments, the user instruction generator 282 may provide instructions 231 regarding only the phonemes or sounds required for respiratory disease analysis, including providing only a portion of the instructions 231. may be included. For example, if the user speech monitor 260 has not yet acquired a speech sample containing a particular phoneme over a given time frame, the user instruction generator 282 may generate instructions 231 to facilitate the acquisition of a speech sample containing the phoneme information. You may also give Another example of instructions 231 that may be provided by user instruction generator 282 (or user interaction manager 280) is illustrated and detailed in connection with FIGS. 4A, 4B, and 5B.

Some embodiments of user instruction generator 282 may provide instructions 231 that are tailored to a particular user. To this end, the user instruction generator 282 may include information such as a particular user's health status, a clinician's instructions, a prescription or recommendation for the user, user demographic information or EHR information (e.g., if the user is determined to be a smoker, an instruction The instructions 231 may be generated based on previously captured audio/phoneme information from the user. For example, an analysis of past phonemes provided by a user may indicate certain phonemes that exhibit greater changes during all or part of a respiratory infection (eg, during recovery). Additionally or alternatively, some phoneme characteristics may indicate that a user has a respiratory disease that is more easily detected or tracked than other characteristics. In these examples, one embodiment of user instruction generator 282 may instruct the user to capture additional samples of the phoneme of interest, or generate or modify instructions 231 to Instructions to obtain audio samples containing phonemes that are not useful to the user may be removed (or not provided). In some embodiments of the user instruction generator 282, the instructions 231 are based on past determinations regarding the user's respiratory illness (e.g., whether the user is sick or recovering). It may be modified.

Self-reporting tools 284 may generally be responsible for guiding the user toward providing data and other contextual information that may be associated with a respiratory disease. Self-reporting tool 284 may be adapted to work with self-reporting data evaluator 276 and data collection component 210. Some embodiments of self-reporting tool 284 may work in conjunction with user instruction generator 282 to provide instructions 231 that guide the user to provide user-related data. For example, self-reporting tool 284 may utilize instructions 231 to prompt a user to provide information regarding symptoms the user is experiencing regarding a respiratory illness. In one embodiment, the self-report tool 284 allows the user to rate the severity of each symptom of the set of symptoms, which may or may not be associated with nasal congestion. You may also be prompted to do so. Additionally or alternatively, the self-reporting tool 284 may utilize instructions 231 or ask the user to provide information regarding the user's health or how the user is feeling generally. In one embodiment, the self-report tool 284 may prompt the user to indicate the severity of post-nasal drip, nasal congestion, runny nose, mucus-filled nasal discharge, cough, sore throat, and need to blow the nose. . In some embodiments, self-reporting tool 284 may include user interface elements that facilitate prompting or accepting data from a user. For example, FIGS. 5D and 5E depict aspects of a GUI that provides a self-reporting tool 284. Also shown in FIGS. 4D, 4E, and 4F are example user interactions that illustrate aspects of a voice user interface (VUI) that provides self-reporting tool 284.

In some embodiments, self-reporting tool 284 may utilize instructions 231 to prompt the user to enter symptoms or general condition multiple times per day, and the requested inputs may include: May vary depending on time of day. In some embodiments, the input time may correspond to a time frame or session in which the user audio samples are obtained. In one example, self-report tool 284 may prompt the user to rate the perceived severity of 19 symptoms in the morning and 16 symptoms in the evening. Additionally or alternatively, the self-report tool 284 may prompt the user to answer four sleep-related questions in the morning and one end-of-day fatigue question at night. The following table is an exemplary list of user input prompts that may be determined by self-reporting tool 284 by utilizing instructions 231 and output by self-reporting tool 284 or other subcomponents of user interaction manager 280. It shows.

In some embodiments, the self-report tool 284 is based on the user's detected phoneme characteristics (i.e., based on suspected respiratory illness), previously captured phoneme data, and/or other self-report input. may be used to ask additional questions or provide additional prompts. In an exemplary embodiment, if the analysis of phonemic characteristics indicates that the user may have a respiratory infection or is recovering from a respiratory infection, the self-reporting tool 284 may indicate that the user may be suffering from a respiratory infection. It may also be possible to facilitate prompting of users regarding reporting. For example, a self-reporting tool 284 that can utilize instructions 231 and/or interface with a user interaction manager 280 may be configured to ask questions regarding (or display requests for) a user's symptoms. good. In this embodiment, the user may be asked questions regarding how the user feels, such as "Do you feel stuffy?" In a similar example, if a user reports having a stuffy nose or certain symptoms, the self-report tool 284 may ask, "On a scale of 1-10, how stuffy is your nose?" or this additional detail. It is also possible to separately remind the user that the service will be provided.

In some embodiments, self-reporting tools 284 may include functionality that allows a user to facilitate automatic collection of a user's physiological data through communicative coupling of a wearable device, health monitor, or physiological sensor. In one such embodiment, data may be received by context information determiner 2616 or other components of system 200 and stored in patient record 240. In some embodiments, as described above, the information received from this self-reporting tool 284 is stored in a relational database to identify specific speech samples obtained from a session or specific phoneme characteristics determined from the speech samples. It may also be associated with a vector. In some embodiments, based on the physiological data received, self-reporting tool 284 may prompt or request self-reported symptom information from the user, as described above.

According to various embodiments, user input response generator 286 may be generally responsible for providing feedback to the user. In one such embodiment, user input response generator 286 may be configured to analyze user input for user data, such as utterances or audio recordings, and may also include user instruction generator 282 and/or sample recording inspection. In conjunction with device 2608, feedback may be provided to the user based on the user's input. In one embodiment, user input response generator 286 may analyze the user's response to determine whether the user provided a good audio sample and then indicate that determination to the user. For example, a green light, check mark, smiley face, thumbs up, bell or chirp, or similar indicator may be provided to the user to indicate that the recorded sample is good. . Similarly, a red light, grimace, buzzer, or similar indicator may be provided to alert the user that the sample is incomplete or defective. In some embodiments, user input response generator 286 may determine whether the user has failed to comply with instructions 231 from user instruction generator 282. In some embodiments of user input response generator 286, if a problem is detected, a chatbot software agent may be invoked to provide contextual help or assistance to the user.

Embodiments of the user input response generator 286 may detect if past audio samples have insufficient acoustic characteristics such as sound level, if there is too much background noise, or if the sound recorded in the sample is not long enough. In this case, the user may be notified. For example, after the user provides the first audio sample, the user input response generator 286 may output, "I couldn't catch you. Please repeat. Say 'aaaaa' for 5 seconds." good. In one embodiment, the user input response generator 286 may provide an indication of the volume level that the user should achieve during recording and/or provide feedback to the user regarding whether the audio sample is acceptable. Often, this may be determined according to sample record examiner 2608.

In some embodiments, the user input response generator 286 may utilize aspects of the user interface to provide feedback to the user regarding sound levels, background noise, or timing durations for acquiring audio samples. good. For example, a visual or audible countdown clock or timer may be used to signal when to start or stop speaking for recording audio samples. One embodiment of a timer is shown in FIG. 5A as GUI element 5122. A similar example of providing user input responses is shown in FIG. 5B as GUI elements 5222, including a timer and an indicator of background noise. Other examples (not shown) may relate to audio input levels, background noise, word color changes or balls bouncing along with the words the user is reading as the words are spoken, or similar auditory or visual indicators. Examples include GUI elements.

User input response generator 286 may provide the user with an indication of the progress of a particular speech-related task (eg, vocalization) or audio session. For example, as discussed above, the user input response generator 286 may count the number of seconds (as displayed in the graphical user interface or audio user interface) during which the user makes a sustained utterance, And/or the timing of stopping may be indicated to the user. Some embodiments of user input response generator 286 (or user instruction generator 282) provide indicators regarding upcoming or completed speech-related tasks for a particular session, time frame, or date. It may also be provided.

As discussed above, some embodiments of the user input response generator 286 may check feedback of the audio sample provided by the user, such as the volume level of the sample, whether the sample is acceptable, and /Or a visual indicator may be generated for the user, such as an indicator as to whether the sample was correctly captured.

Utilizing the voice information collected and determined by user voice monitor 260 (alone or in conjunction with user interaction manager 280), respiratory disease tracker 270 may generate information regarding the user's respiratory disease and/or the user's A prediction regarding future respiratory disease may be determined. In one embodiment, respiratory disease tracker 270 generates a set of phoneme features (e.g., one or more phoneme feature vectors) that can be associated with a particular time or time frame and timestamped with date and/or time information. You may also receive it. For example, the phoneme feature set may be received from a user voice monitor 260 or a patient record 240 (such as a phoneme feature vector 244) associated with the user. As described herein, the temporal information associated with the phoneme feature set corresponds to the date and/or time at which the speech sample (or speech-related data) used to determine the phoneme feature set was obtained from the user. You can leave it there. Respiratory disease tracker 270 may also receive context information associated with the audio recording or voice sample for which phonemic characteristics have been determined, which may also be associated with patient record 240 and/or user voice monitor 260 ( Or specifically, it may be received from the context information determiner 2616). Embodiments of the respiratory disease tracker 270 utilize one or more classifiers to identify the user based on multiple sets of phoneme features (vectors) and, in some embodiments, context information. A currently likely respiratory disease score or determination may be generated. Additionally or alternatively, respiratory illness tracker 270 may utilize a predictor model to predict likely future respiratory illnesses for the user. Embodiments of respiratory disease tracker 270 may include a feature vector time series assembler 272, a phoneme feature comparator 274, a self-report data evaluator 276, and a respiratory disease inference engine 278.

The feature vector time series assembler 272 may be employed to assemble a time series of consecutive phoneme feature vectors (or feature sets) of the user. The time series may be assembled in chronological or reverse chronological order according to time information (or timestamps) associated with the characteristic vectors. In some embodiments, the time series includes phoneme feature vectors generated for collected speech samples of a user or patient, within a time period during which the patient is ill (i.e., has a respiratory infection). or a time within a set or predetermined time interval, such as the past 3-5 weeks, the past 2 weeks, or the past 1 week. It may include all associated phoneme feature vectors. In other embodiments, the time series includes only two characteristic vectors. In one such embodiment, the first phoneme feature vector of the time series may be associated with the most recent time period or point in time according to the corresponding timestamp, thus providing information regarding the user's current respiratory illness. represents. On the other hand, the second characteristic vector may be associated with an earlier time period or point in time. In some embodiments, the earlier time period corresponds to a time interval (ie, a time when the user was sick or well) during which the user's respiratory illness was different from the most recent time period or point in time.

Furthermore, the phoneme feature comparator 274 may generally be responsible for determining differences in the user's phoneme feature vectors 244 (or differences in the values of features in different feature sets). The phoneme feature comparator 274 may determine the difference by comparing two or more phoneme feature vectors. For example, a comparison may be performed between phoneme feature vectors 244 associated with any two different time periods or time periods or between a feature vector associated with the most recent time period or time point and a feature vector associated with an earlier time period or time point. It may be possible to do so. Each phoneme feature set (or vector) being compared is , may be associated with different time periods or points in time. In some embodiments, it is also contemplated that two or more characteristic vectors that are compared may have the same duration, or that each vector has a corresponding characteristic (i.e., the same dimension) for comparison. It will be done. In some cases, only some of the characteristic vectors (or a subset of characteristics) may be compared. In one embodiment, multiple characteristic vectors (which may include three or more vectors, each associated with a different time period or point in time) are used to perform an analysis characterizing changes in a characteristic over time frames spanning different time periods or points in time. It may also be used by the phoneme characteristic comparator 274. For example, this analysis may include determining rate of change, regression or curve fitting, cluster analysis, discriminant analysis, or other analysis. As mentioned above, herein the terms "feature set" and "feature vector" may be used interchangeably to facilitate comparisons between feature sets; The individual properties of the set can be thought of as a property vector.

In some embodiments, an average or combination of the characteristic vector for the most recent time period or time point (e.g., the characteristic vector determined from the most recently acquired audio sample) and the characteristic vector corresponding to multiple earlier time periods or time points. (eg, a boxcar moving average based on multiple prior characteristic vectors or audio samples). Optionally, this average may take into account the maximum number of feature vectors associated with the user's previous period or point in time (e.g., feature vectors corresponding to 10 previous sessions in which audio samples were obtained). The maximum number of characteristic vectors from a predetermined earlier time interval, such as the past week or two weeks, may be taken into account. Alternatively or additionally, the phoneme feature comparator 274 may be used to detect a user such as the general public or other users similar to the monitoring user (e.g., with a similar respiratory disease or monitoring The user's feature vector for the most recent time segment may be compared against phoneme feature criteria that may be based on other similar cohorts for the user. Additionally, in some cases, the comparison includes statistics about the criterion (or, in embodiments that do not utilize a criterion, the characteristic set), such as the statistical variance or standard deviation of the characteristic set (or corresponding to the characteristic set) corresponding to the criterion. The information may also be used. In some embodiments, an average, in particular a rolling average or a moving average, is used to act as a smoothing function for the preceding characteristic vector (i.e., the characteristic vector corresponding to the audio samples taken at an earlier time period or point in time). may be considered. This variation in voice-related data does not account for respiratory infections that may occur in early samples (e.g., whether voice samples were taken in the morning when the user first woke up, at the end of a long day, or if the user The amount of time spent cheering or singing can be minimized. Additionally, some embodiments of the phoneme feature comparator 274 may compare the average of the most recent feature vectors to the mean of earlier feature vectors or feature vectors associated with a single earlier time period or point in time. You might also say that. Similarly, the statistical variance of the characteristic value (or part of the characteristic value) of the most recent characteristic is determined and compared against the variance of the earlier characteristic value (or part thereof). Good too.

Some embodiments of phoneme feature comparator 274 may utilize phoneme feature comparison logic 235 to determine comparisons of phoneme feature vectors. Phoneme feature comparison logic 235 may include computer instructions (e.g., functions, routines, programs, libraries, etc.) for performing comparisons of features or feature vectors or for facilitating interpretation by comparing or processing comparisons. including, but not limited to, one or more rules, conditions, processes, models, or other logic. In some embodiments, phoneme feature comparison logic 235 is utilized by phoneme feature comparator 274 to compute distance metrics or difference measurements for phoneme feature vectors. In an exemplary aspect, the distance measurement results may be considered to quantify changes in the acoustic characteristic space of the audio information over time for the user. In this way, changes in a user's respiratory illness can be observed based on quantifiable changes detected in the acoustic feature space (e.g., phoneme features) between two or more times at which the user's speech information is acquired. and may be quantified. In one embodiment, phoneme feature comparator 274 may determine a Euclidean measurement or L2 distance for two feature vectors (or an average of feature vectors) to determine the distance measurement. Optionally, phoneme feature comparison logic 235 includes logic to perform flattening, normalization, or other processing operations for multidimensional vectors prior to or as part of the comparison operation. You can stay there. In some embodiments, phoneme feature comparison logic 235 may include logic to implement other distance metrics (eg, Manhattan distance). For example, Mahalanobis distance may be utilized to determine the distance between the most recent feature vector and a set of feature vectors associated with earlier time periods or points in time. In some embodiments, Levenshtein distances may be determined, such as embodiments that compare readings of sentences by users. For example, according to one embodiment, a speech-to-text algorithm may be utilized to generate text from a user's reading of a text aloud. Also, a chronology of one or more entries including syllables or words of the sentence and corresponding timestamps of when the user read these words may be determined. This time series (or timestamp) information may be adapted to generate a characteristic vector for comparison (e.g. using a Levenshtein distance algorithm) against a similarly determined reference characteristic vector. (Alternatively, it may be used as a characteristic).

In some embodiments, phoneme feature differences (or distance metrics) may be determined for multiple pairs of times for a patient. For example, calculating the distance between the phoneme feature vector of the most recent date and the phoneme feature vector of the date before the most recent date, and/or calculating the distance between the phoneme feature vector of the most recent date and the phoneme feature vector of a sample collected one week ago. The distance between the characteristic vector or the phoneme characteristic vector representing the reference may be calculated. Additionally, in some embodiments, different types of distance measurements may be computed for different phoneme feature vectors or features.

In some embodiments, phoneme feature differences (or distance metrics) may indicate differences in particular acoustic features over time or points in time. For example, the phoneme characteristic comparator 274

A distance metric regarding the harmonicity of the phoneme may be computed.

A different distance metric may be calculated for the shimmer. Additionally or alternatively, a distance metric (or indicator of change) may be determined for a combination of acoustic characteristics over a period or time.

In some embodiments, phoneme feature comparison logic 235 (or phoneme feature comparator 274) includes computer instructions that generate or utilize feature criteria for a user. The criteria may represent a health condition, an illness condition (eg, an influenza condition or a respiratory infection condition), a recovery condition, or any other condition of the user. Examples of other states include the user's state at a point or time interval (e.g., 30 days ago), the user's state associated with an event (e.g., before surgery or injury), the user's state in response to conditions ( For example, the user's condition when on medication or living in a polluted city) or in conjunction with other criteria. For example, the criteria for health status may be determined using one or more sets of characteristics that correspond to one or more time periods (e.g., days) during which the user was in good health. .

In this specification, a criterion determined based on a plurality of characteristic sets, each of which corresponds to a different time period, may be referred to as a multi-reference or multi-day criterion. Optionally, the multiple references include multiple feature sets or a group of feature sets, each corresponding to a different time segment. Alternatively, a multi-reference criterion may include a single representative set of characteristics based on multiple characteristic sets from multiple time periods (e.g., as discussed above, for characteristic set values at different time periods or points in time). (including average or composite). In some embodiments, the criteria may include statistical data, supplemental data, or metadata about the characteristics. For example, if multiple characteristic sets are used (e.g., multiple reference standards), the criteria may include the characteristic set (which may represent multiple time periods) as well as the statistical variance or standard deviation of the characteristic values. good. Supplemental data may include contextual information that may be associated with the time segment of the feature set used to determine the criteria. The metadata may include information about the set of characteristics used to determine the criteria, such as information about the user's respiratory illness in the time period (e.g., whether the user is healthy, sick, recovering, etc.) or other information about the criteria. You can stay there. In some embodiments, a set of criteria may be determined to perform different comparisons based on various criteria, as described herein.

The characteristic vector generated from the collected audio samples may be compared to a reference for a particular state to indicate the user's tone or state as compared to a known tone or state. In an exemplary embodiment, a criterion is determined for a particular user such that comparison to the criterion indicates whether the user's condition or condition has changed. Alternatively or in addition to this, the criteria may be determined for the general public or from a cohort of similar users. In some embodiments, different types of criteria are used for different sets of characteristics. For example, some characteristics may be compared to user-specific criteria, while other characteristics are compared to standard criteria determined by data from many patients. In some embodiments, a user may specify (eg, via settings 249) a particular audio sample, date, or time period to use in determining the criteria. For example, the user may specify a date or range of dates through the GUI, such as selecting a date on a calendar that corresponds to the user's known condition or condition, and may also specify information regarding the known condition or condition. (eg, "Please select at least one date on which you were healthy"). Similarly, in a recording session in which a voice sample is obtained, the user may indicate that the voice sample is to be used for determining the criteria and provide a corresponding indication of the user's tone or condition. For example, during a recording session, a GUI checkbox may be presented to use the sample as a measure of healthy (or sick or recovering) status.

In some embodiments, phoneme feature comparison logic 235 may include computer instructions that generate and utilize multiple dates or multiple references. The multi-day criterion may be variable or fixed, for example. In particular, by performing a comparison of the most recent feature vector against this reference, the phoneme feature comparator 274 provides information indicating that the user's respiratory disease has changed and whether the user is sick or healthy. It may be decided. Details regarding determining a user's respiratory illness based on the comparisons performed by the phoneme feature comparator 274 are described in connection with the respiratory illness inference engine 278. Similarly, the phoneme feature comparison logic 235 may determine (e.g., by the respiratory disease inference engine 278) that the user's respiratory disease has changed and that the user is sick (or healthy) or that the user is doing well or Instructions for performing multiple comparisons utilizing the most recent phoneme feature vector and a set of early vectors (or multiple references) and for comparing the difference measurements with each other so that a downward trend can be determined. It may include. Further details of performing multiple comparisons, including comparing distance measurements, are described in connection with respiratory disease inference engine 278.

In some embodiments, criteria may be defined dynamically and automatically as more information about the user is obtained. For example, as the user's normal variability of voice information changes over time, the user's criteria may also change to reflect the user's current normal variability. In some embodiments, the most recent feature set or multiple current feature sets (corresponding to multiple time periods (e.g., days)) may be determined from a baseline measure (e.g., healthy, sick, recovered). Adaptive criteria may be used that are updated when a new set of characteristics is determined that match the criteria. For example, multiple feature sets utilized in an adaptive criterion can be configured on a first-in, first-out basis so that when a new feature set for the criterion is determined (e.g., from a more recent date), the feature set from an older time is no longer considered. FIFO) data flow may be followed. In this way, the adaptive criteria may exclude small fluctuations or slow changes and adaptations that may occur in the user's voice. In some embodiments that utilize an adaptive criterion, the parameters of this criterion (e.g., the number of feature sets included or the time window of the most recent feature set included) are set in the application settings (e.g., settings 249). You can leave it there. In some embodiments where feature sets from multiple time periods (e.g., dates) are utilized as a criterion, the more recently determined feature sets are weighted to have more significance so that the criteria is more recent. You can do it like this. Alternatively or additionally, older (ie, "stale") characteristic sets corresponding to earlier time periods or points in time may be weighted such that they decay or contribute less to the criterion over time.

In some embodiments, certain characteristics within the user criteria may be adapted for the particular user. Thus, different users may have different combinations of phoneme characteristics within their respective criteria, and thus different phoneme characteristics are determined and utilized for respiratory disease monitoring for each user. You can. For example, in a first user's healthy voice sample, certain acoustic characteristics (in general or for specific phonemes) may vary naturally and be useful for detecting changes in the user's respiratory illness. However, it may be useful to other users and may be included in their criteria.

In some embodiments, the user's criteria may be correlated with contextual information such as weather, time of day, and/or season (ie, time of year). For example, the user's baseline may be generated from samples recorded during periods of high humidity. This criterion may be adapted to be compared to phoneme feature vectors generated from samples recorded during periods of high humidity. Conversely, different criteria may be compared to phoneme feature vectors generated from samples taken during periods of relatively low humidity. Thus, there may be multiple criteria determined for a given user and utilized in different contexts.

Furthermore, in some embodiments, the criteria may be determined not for a particular user, but for a particular cohort, such as patients, that share a set of common characteristics. In an exemplary embodiment, data from patients known to have the same respiratory disease (e.g., influenza, rhinovirus, COVID-19, asthma, chronic obstructive pulmonary disease (COPD), etc.) The criteria may be specific to respiratory diseases in that they can be determined using the following: In some embodiments, criteria may be defined dynamically as more information about the user is obtained, such as providing initial criteria based on phoneme characteristic data from the general public or a cohort similar to the user. It may be . Over time, as more phoneme feature sets for a user are determined, the criteria may be updated and adapted to the user using the user's phoneme feature sets.

Some embodiments of respiratory illness tracker 270 include self-reporting information that may be correlated with or taken into account in a user's diagnosis (e.g., determining a user's current respiratory illness) and/or self-predicting future health. A self-report data evaluator 276 may be included that may collect reporting information from the user. Self-report data evaluator 276 may collect this information from self-report tool 284 and/or context information determiner 2616. This information may be user-provided or user-derived data (eg, from sensors indicating body temperature, breathing rate, blood oxygen, etc.) regarding how the user feels or how the user is currently doing. In one embodiment, this information includes the user's self-reported perceived severity of various symptoms with respect to the respiratory illness. For example, this information may include the user's severity score for post-nasal drip, nasal congestion, runny nose, mucus-filled nasal discharge, cough, sore throat, and need to blow the nose.

Self-report data evaluator 276 may utilize the input data to determine a symptom score indicative of the severity of the respiratory disease or condition. For example, self-report data evaluator 276 may output a composite symptom score (CSS) that may be computed by combining scores for multiple symptoms. Furthermore, individual symptom scores may be summed or averaged to obtain a composite symptom score. For example, in one embodiment, a composite symptom score may be determined by summing symptom scores (range 0-5) for seven respiratory disease-related symptoms, resulting in a composite symptom score. Scores range from 0 to 35. Higher symptom scores may indicate more severe symptoms. In one embodiment, symptoms may include postnasal drip, nasal congestion, runny nose, mucous nasal discharge, cough, sore throat, and the need to blow the nose. In some embodiments, separate symptom scores may be generated for all symptoms, such as symptoms related to nasal congestion and symptoms not related to nasal congestion.

In some embodiments, self-report data evaluator 276 associates the determined symptom score to phoneme characteristics determined from audio samples that correspond to the same time window as the user input that generated the determined symptom score. It's okay. In other embodiments, self-report data evaluator 276 may correlate the symptom score against a phoneme feature vector or a distance metric determined by a comparison of phoneme feature vectors. By fitting an exponential decay model and correlating acoustic feature values with decay rates, symptom scores, such as composite symptom scores for all symptoms, including nasal congestion-related or non-nasal congestion-related symptoms, can be generated for phonemic features. They may be correlated. Decay models can be utilized to estimate the magnitude and rate of change in symptoms. In one embodiment, the score ≈ ae ^{- b (days - 1)} + ε is utilized for an exponential decay model. Here, a represents the magnitude of change, and b represents the attenuation rate. This exponential decay model used subjects as a random effect in the R-project for Statistical Computing, package nlme (version 3.1.144), accessible through the Comprehensive R Archive Network (CRAN). It may be implemented by using a non-linear mixed effects model. Examples of correlations between phoneme feature vectors and symptom scores and between phoneme feature vectors and/or derived distance metrics are shown in FIG. 9 and FIGS. 11A and 11B, respectively. Symptom scores generated by self-report data evaluator 276 and, in some embodiments, associations and/or correlations with phoneme feature vectors or distance metrics may be stored in the user's patient record 240. .

In some embodiments, self-reporting is initiated based on a detected change (eg, the user's condition is worsening) or if the user is already sick. Initiation of self-reporting may also be based on user-configured preferences, such as settings 249 in patient record 240. In some embodiments, self-reporting is initiated based on respiratory illness detected from the user's collected voice samples. For example, the self-report data evaluator 276 may prompt the user to obtain self-report symptom information based on the detection of the user's tone through speech analysis, which may be determined based on the feature vector comparison performed by the phoneme feature comparator 274. You may also decide to prompt.

Further, the respiratory illness inference engine 278 may generally be responsible for determining or inferring a user's current respiratory illness and/or predicting a user's future respiratory illness. This determination may be based on the user's acoustic characteristics, including detected changes in characteristic values. To this end, the respiratory disease inference engine 278 may receive information regarding changes in the user's phonemic characteristics and/or detected characteristics that may be determined as a distance metric. Some embodiments of respiratory disease inference engine 278 may include user self-reported data or Further use may be made of analysis of self-reported data. In one embodiment, the longest utterance time or

The duration of a user's duration of one or more particular phonemes such as, another basic vowel utterance, or other utterances may be adapted for use by the respiratory disease inference engine 278. For example, a short maximum phonation time indicates shortness of breath and/or decreased vital capacity and may be associated with worsening of respiratory disease. Additionally, respiratory illness inference engine 278 may compare the acoustic characteristics to one or more criteria to determine a user's respiratory illness. For example, to determine whether the user's respiratory capacity is increasing or decreasing, the user's longest phonation time may be compared to the user's baseline maximum phonation time, and if the maximum phonation time is shorter; may indicate worsening of respiratory disease. Similarly, a low percentage of voiced frames of phonemes extracted from a voice sample of a predetermined duration may also indicate worsening of a respiratory disease. As a non-limiting example, for audio samples of reading sentences, a decrease in speech rate, an increase in average pause length, an increase in the number of pauses, and/or a decrease in global SNR may indicate worsening of a respiratory disease. Determining any of these changes may be accomplished by comparing the most recent sample to a criterion, such as a user-specific criterion, as described herein.

Respiratory disease inference engine 278 utilizes this input information to generate one or more respiratory disease scores or classifications representative of the user's current respiratory disease and/or future disease (i.e., prediction). You can do it like this. The output from the respiratory disease inference engine 278 may be stored in the results/inference state 246 of the user's patient record 240 and as described in connection with the example GUI 5300 of FIG. 5C. , may be presented to the user.

In some embodiments, the respiratory illness inference engine 278 may determine a respiratory illness score that corresponds to the detected quantitative change in the user's respiratory illness. Alternatively or in addition to this, the respiratory illness score or inference for the user's respiratory infection illness may be based on the detected value (i.e., a single measurement rather than a change) of one or more specific phonemic features or one or more It may be based on a combination of specific characteristic values, detected changes in characteristic values, and different rates of change. In one embodiment, the respiratory illness score measures the likelihood or probability that the user has (or does not have) a respiratory illness (e.g., with respect to any disease in general or a particular respiratory infection). can be shown. For example, a respiratory illness score may indicate that the user has a 60% chance of having a respiratory infection. In some embodiments, a respiratory illness score may include a composite score or a set of scores (eg, a set of probabilities that a user has a set of respiratory illnesses). For example, the respiratory disease inference engine 278 may determine whether a user has a specific respiratory disease with a corresponding probability of each disease being allergies (0.2), rhinovirus (0.3), COVID-19 (0.04), etc. Vectors of organic diseases may also be generated. Alternatively or additionally, the respiratory illness score may be indicative of a difference in the user's current condition relative to a known health condition, or may be indicative of a user's current condition relative to a user's baseline or health condition as described herein. may be based on a comparison of the current state of.

In many cases, the respiratory disease inference engine 278 may determine the change or difference (or probability of respiratory infection) from the user's health status in which the user does not experience symptoms (or the probability of a respiratory infection). A disease score may indicate this). This feature is an advantage and improvement over conventional techniques that rely on subjective data. On the other hand, rather than relying on subjective data, embodiments of the techniques provided herein may detect the onset of a respiratory infection before a user feels any symptoms. These embodiments may be particularly useful in combating respiratory pandemics such as SARS-CoV-2 (COVID-19) by providing earlier warning of respiratory infections than conventional approaches. For example, a respiratory illness score indicating the likelihood of infection (or a determination by the respiratory illness inference engine 278 regarding a user's respiratory illness) may cause the user to self-quarantine, socially distance, wear a mask, or or other precautions may be notified to the user.

In some embodiments, a respiratory illness score that may indicate or correspond to a probability that the user has a respiratory infection is expressed as a value for the health status of the user. You can. For example, if the respiratory disease score is 90 out of 100 (100 represents a healthy state), then the detected change in the user's respiratory disease is 90% of the user's normal or healthy state (i.e. , 10% change). In this example, although a user may feel healthy with a respiratory illness score of 90, the score may indicate that the user has developed a respiratory infection (or is still recovering). Similarly, a respiratory illness score of 20 indicates that the user is likely to be ill (i.e., the user is likely to have a respiratory infection); If the user has a respiratory disease score of 20, it may indicate that the user is likely to be sick, but not as sick (or not as likely) as a respiratory disease score of 20. For example, if a respiratory illness score corresponds to a probability, a respiratory illness score of 20 may indicate a higher probability that the user has an infection than a respiratory illness score of 40. However, if the respiratory illness score reflects the difference between the user's current condition and the health baseline, a respiratory illness score of 40 will result in a lower detected change from the baseline than a respiratory illness score of 20. may be small, indicating that the user may not be sick. In some cases, the user's respiratory disease score may be indicated by a color or symbol as an alternative to or addition to a number. For example, green color may indicate that the user is healthy, while yellow, orange, and red may represent a greater deviation from the user's health status and may indicate that the user has a respiratory infection. This may indicate that the quality of the product is increasing. Similarly, emoticons (eg, smiling and frowning or pale faces) may be utilized to represent respiratory disease scores.

Embodiments herein may include phoneme feature information (including changes in phoneme features) and, in some embodiments, contextual information (such as measured physiological data) and/or self-reported symptom scores from users. It shall be understood that it may be used to further characterize the user's respiratory infection status based on. Thus, in some cases, both severe and mild respiratory infections may represent the same phonemic characteristics (or changes in characteristics). Therefore, in these cases, different respiratory disease scores are not considered useful in indicating whether a user is "severely sick" or "less sick," but instead indicate whether the user has a respiratory infection (or It may indicate only that the user does not have a respiratory infection (i.e., a binary indicator), it may indicate the probability that the user is sick, or it may indicate the relationship between the user's current condition and health status. (could be indicative of a respiratory infection).

Additionally, monitoring changes in respiratory disease scores when correlated with a user's respiratory infection treatment (which may be received as contextual information), such as taking prescribed medications, can indicate the efficacy of the treatment. . For example, a user diagnosed with a respiratory infection may be prescribed antibiotics by a clinician and use a respiratory infection monitoring app on a smartphone, such as the respiratory infection monitoring app 5101 described in connection with FIG. 5A. be instructed to do so. A first respiratory illness score (or first set of respiratory illness scores) may then be determined from the user voice samples collected as described herein. After a time interval, such as one week, the second respiratory illness score may indicate a change in the user's respiratory illness. Changes that indicate that the user's disease is improving (which may be determined as described below) may imply that the antibiotic is working. Changes that indicate that the user's disease is not improving, or remaining the same, imply that the antibiotics are not working, and the user's clinician may wish to administer a different treatment. think. Thus, embodiments of the technology described herein can determine objectives such as quantifiable information regarding changes in a user's respiratory disease and whether the patient has been prescribed for the treatment of a respiratory infection. Antibiotics can be used more carefully and purposefully to extend their effectiveness and minimize antimicrobial resistance.

In some embodiments, respiratory disease inference engine 278 may utilize user disease inference logic 237 to determine a respiratory disease score or make inferences and/or predictions regarding a user's respiratory disease. . User disease inference logic 237 may include rules, conditions, associations, machine learning models, or other criteria for inferring and/or predicting the likelihood of respiratory disease from voice-related data. User disease inference logic 237 may take different forms depending on the mechanism of use and intended output. In one embodiment, user disease inference logic 237 includes one or more classifier models that determine or infer the user's current (i.e., most recent) respiratory disease and/or the likelihood of the user's future respiratory disease. may include one or more predictor models that predict . Examples of classifier models include decision trees or random forests, Naive Bayes, neural networks, pattern recognition models, other machine learning models, other statistical classifiers, or combinations thereof (e.g., ensembles). However, it is not limited to these. In some embodiments, user disease inference logic 237 may include logic to perform clustering or unsupervised classification techniques. Examples of predictive models include regression techniques (e.g., linear or logistic regression, least squares, generalized linear models (GLMs), multivariate adaptive regression splines (MARS), or other regression processes), neural networks, decisions Examples include, but are not limited to, trees or random forests, or other predictive models or combinations of models (eg, ensembles).

As discussed above, some embodiments of the respiratory disease inference engine 278 may determine the probability that a user has or will develop a respiratory infection. In some cases, this probability may be based on acoustic characteristics of the user (including detected changes in characteristics), the output of a classifier or predictive model, or rules or conditions that are met. For example, according to one embodiment, user disease inference logic 237 determines whether a change in a phoneme feature value or one or more phoneme feature values meets a particular threshold (e.g., a disease change threshold, as described herein). may include rules for determining the probability of respiratory infection based on the degree of detected change occurring in the respiratory infection. In one embodiment, user illness inference logic 237 includes rules that interpret detected changes or differences between the user's current respiratory illness and a baseline to determine the likelihood that the user has a respiratory infection. It's okay to stay. In another embodiment, multiple recent assessments of the user's respiratory illness (ie, multiple comparisons of recent times to earlier times) may contribute to the probability. As a non-limiting example, if a user exhibits a change in respiratory illness for two days in a row, the probability of a respiratory infection may be higher than if the user exhibits a change for only one day. In one embodiment, a set of thresholds applied to one or more patterns or characteristic changes of a set of known phoneme characteristic changes for a particular respiratory infection, corresponding to the known respiratory infection; The detected change and/or rate of change may be compared against the probability of infection determined based on the comparison. Further, in some embodiments, the user disease inference logic 237 utilizes contextual information, such as physiological information or information regarding the regional prevalence of respiratory infectious diseases, to estimate the probability that the user has a respiratory infectious disease. It may be decided.

User disease inference logic 237 includes one or more determined changes in the acoustic characteristic information (e.g., changes in characteristic set values, characteristic vector distance measurements, and other data) or rates of change in the acoustic characteristic information. may include computer instructions and rules or conditions for comparison to a threshold (sometimes referred to herein as a disease change threshold). For example, distance measurements of two characteristic vectors corresponding to a recent time interval and an earlier time interval, respectively, may be compared to a disease change threshold. Based on the comparison, the disease change threshold is determined by the detector (e.g., outlier detection It may also be used as a container. The disease change threshold is such that a significant change in the user's disease can be detected, while a small variation, which is a trivial change, will not be detected as a change (or determined as a change) in the user's respiratory disease. It may be determined that there is no such thing. For example, some embodiments utilizing a multi-day criterion may employ a disease change threshold determined as two standard deviations of the characteristic value of the multi-day criterion, as described elsewhere herein. .

In some embodiments, the disease change threshold is specific to the user's disease state (e.g., infected or uninfected), and if the magnitude of change between the characteristic vectors satisfies the disease change threshold, the user's disease may be determined to have changed. Further, the threshold value may be used not only to determine the trend of respiratory diseases in general but also to determine the possibility of respiratory diseases. In one embodiment, if the comparison (which may be performed by phoneme feature comparator 274) meets (e.g., exceeds) a disease change threshold, then the user's respiratory disease has a certain magnitude (as defined by the disease change threshold). Since the user's disease is changing, it can be determined that the user's disease is improving or worsening (i.e., a trend). Thus, in this embodiment, minor changes that do not meet the disease change threshold cannot be considered. Alternatively, it may indicate that the user's disease has not changed substantially.

In some embodiments, disease change thresholds are weighted, applied to only a subset of phoneme features, and/or set to a value that characterizes changes in each phoneme feature or subset of features of a feature vector (or phoneme feature set). The inclusion of sets of threshold values is possible. For example, a change in the first phoneme characteristic can be significant even if it is small, whereas a change in the second phoneme characteristic cannot be significant if it is small, or it is thought that it generally occurs. For this reason, it is considered useful to know when the first characteristic value has changed even slightly, and it is useful to know when the second characteristic value has changed significantly. Therefore, a smaller first disease change threshold (or weighted threshold) may be used for this first phoneme characteristic so that even a small change can be satisfied. A larger (second) disease change threshold (or a differently weighted threshold) may be used for the second phoneme characteristic. Application of such weighted or modified disease change thresholds determines that a particular phoneme feature is more susceptible (i.e., a change in this phoneme feature is more representative of a change in the user's respiratory disease). ) may be adapted for use in the detection or monitoring of certain respiratory infections.

In some embodiments, the disease change threshold is based on the standard deviation of the reference used for comparison against the user's most recent acoustic characteristic value. For example, a criterion, such as a multi-day criterion, may be determined (eg, by phoneme feature comparison logic 235) to include characteristic information for multiple time periods in which the user began to become healthy (or ill), for example. The standard deviation may be determined based on characteristic values of the characteristic from different time periods (eg, days) used as a reference. The disease change threshold may be determined based on standard deviation (eg, a threshold of two standard deviations is utilized). For example, if a comparison of the most recent phoneme feature set to a health standard (or similar detected change in the user's phoneme feature values over a period or point in time) satisfies two standard deviations of the standard, then the user It can be determined that the patient has a disease. In this way, the above comparison is more robust. As a non-limiting example, minor changes in the user's acoustic characteristics that may occur on a daily basis in a healthy case are factored into the disease change threshold. In some cases, multiple thresholds may be utilized based on standard deviations to determine or quantify the degree of difference between the user's current respiratory illness and a baseline. For example, in one embodiment, a user has a low probability of having a respiratory infection if a comparison to a health criterion (or a similar detected change in the user's phoneme feature value over time) meets two standard deviations of the criterion. If it is determined that the comparison satisfies three standard deviations of the standard, it may be determined that the user has a high probability of having a respiratory infection.

In some embodiments, the disease change threshold determined according to user disease inference logic 237 may be subject to modification (e.g., by the user, clinician, or caregiver of the user) and (e.g., , a clinician, a caregiver, or an application developer). Further, the disease change threshold may be based on reference population data or may be determined for a specific user. For example, the disease change threshold may be based on the user's specific health information (e.g., medical exam, medication, or health record data) and/or personal information (e.g., the user's behavior or behaviors such as age, singing, or smoking). It may be set. Additionally or alternatively, the user (or caregiver) may set or adjust the disease change threshold as a setting, such as settings 249 in the personal record 240. In some embodiments, the disease change threshold may be based on the particular respiratory infection being monitored or detected. For example, user disease inference logic 237 may include logic that utilizes different thresholds (eg, a set of thresholds) for monitoring different possible respiratory infections or diseases. Therefore, if a user's disease is known or suspected (e.g., through a diagnosis), certain thresholds may be utilized, which may be based on contextual information or self-reported symptom information. It may be determined by In some embodiments, more than one disease change threshold may be applied.

In some embodiments, user disease inference logic 237 may include computer instructions for performing outlier (or exception) detection and detecting a possible respiratory infection in the user. It may take the form of an outlier detector (or it may utilize an outlier detection model). For example, in one embodiment, user disease inference logic 237 determines the standard deviation of a set of baseline characteristics (e.g., a multi-day basis) and utilizes a set of thresholds for outlier detection, as described elsewhere herein. may include rules for In other embodiments, user disease inference logic 237 may be in the form of one or more machine learning models that utilize outlier detection algorithms. For example, user disease inference logic 237 may include one or more probabilistic models, linear regression models, or proximity models. In some aspects, such models may be trained on user data to detect user-specific variability. In other embodiments, the model may be trained to utilize baseline information from respiratory disease-specific cohorts. For example, data from patients known to have certain respiratory diseases, such as influenza, asthma, and chronic obstructive pulmonary disease (COPD), trains models to detect such diseases. Thus, user disease inference logic 237 may be specific to the type of respiratory disease being monitored, determined, or anticipated.

In some embodiments, the output of respiratory disease inference engine 278 utilizing user disease inference logic 237 is a prediction or prediction. This prediction may be determined based on detected changes, rates of change, and/or patterns of change in phonemic features or respiratory disease scores, and trend analysis as described herein. , regression, or other predictive models may be used. In some embodiments, the prediction may include a corresponding predicted probability and/or a future time interval for the prediction (e.g., the likelihood that the user will develop a respiratory infection by the next week is 70%). In one embodiment, the user predicts when the user is likely to become healthy again based on a detected rate of change in the user's phonemic characteristics that indicates a trend of improvement in the user's respiratory disease (e.g., in an example illustrating this embodiment). (see Figure 4E). In some cases, predictions may be provided in the form of trends or views of the user (e.g., is the user getting better or getting worse?), or the probability/likelihood that the user will get sick or recover. Predictions may be provided automatically. In some embodiments, a reference population (e.g., The general public or a population similar to the user, such as a cohort with similar respiratory diseases) may be determined. In some embodiments, respiratory disease inference engine 278 or user disease inference logic 237 may include functionality to assemble one or more patterns of user phoneme feature vectors. These patterns may be correlated with self-reported input or symptom scores, or with judgments generated from self-reported input, such as composite symptom scores. The user's phoneme characteristic patterns may then be analyzed to predict a particular user's future respiratory illness. or other users (reference populations representing the general public, populations of patients with specific respiratory diseases (e.g., cohorts for influenza, asthma, rhinovirus, chronic obstructive pulmonary disease (COPD), COVID-19, etc.) , or a population of patients similar to the user) may be used to predict future respiratory illness for a particular user. Exemplary diagrams illustrating prediction of respiratory disease are provided in FIG. 4E (element 447) and FIG. 5C (element 5316).

User disease inference logic 237, in some embodiments, considers patterns or rates of change in phoneme feature vectors and/or considers geographically specific information, such as infectious disease prevalence in the area where the user is located. is possible. For example, a particular pattern (or rate of change) of changes in all or specific phonemic features may be indicative of a particular respiratory infection (e.g., after several days of nasal congestion) that represents the development of a respiratory disease or condition. usually causes a sore throat, which is usually followed by laryngitis).

In some embodiments, user disease inference logic 237 may include computer instructions for determining and/or comparing multiple changes or rates of change in phoneme feature information. For example, a first comparison (or set of comparisons) between the most recent phoneme feature vector and the first earlier phoneme feature vector may indicate that the user's respiratory illness has changed. In one embodiment, whether the change indicates improvement or worsening of the user's disease may be determined by performing another comparison. For example, a second comparison between the most recent phoneme feature vector and a health baseline feature vector or a second earlier phoneme feature vector for a period or point in time during which the user is known to have been healthy may be determined. You can leave it there. Furthermore, a third comparison between the first early phoneme feature vector and a reference or second early phoneme feature vector may be determined. A comparison of the changes detected between the second and third comparisons (fourth comparison) indicates whether the user's respiratory disease is improving (e.g., the comparison between the latest phoneme feature vector and the health standard). whether the difference is smaller than the difference between the first early phoneme feature vector and the health criterion) or worsening (e.g., the difference between the latest phoneme feature vector and the health criterion is less than the difference between the first early phoneme feature vector and the health criterion) It may be determined whether the difference is larger than the difference from the health standard. Additionally, the extent to which the user's respiratory illness has worsened or improved, how close the user is to recovery (e.g., if the phoneme feature values have returned to or near the healthy baseline), or the user's expectation that the user will be in recovery; Additional comparisons to thresholds indicative of the degree of change may be utilized to determine when a change may occur (e.g., based on the amount or change in the user's condition in a trend indicating improvement).

In some embodiments, user disease inference logic 237 includes user self-report and/or contextual data (in some cases, physiological data, such as user sleep information (if available), and related to recent user activity). information, or a user's location information). For example, if a user's voice-related data indicates hoarseness, and context information determines that the user's location the previous night was at an arena venue, and the previous night's agenda item was titled "Playoff Tournament," the user's disease Reasoning logic 237 may determine that the observed changes in the user's audio data are likely the result of the user participating in a sporting event rather than a respiratory infection.

In some embodiments, user disease inference logic 237 may include computer instructions for determining the risk of a user's likelihood of transmitting a detected respiratory-related pathogen. For example, transmission risk is determined based on rules or conditions applied to the respiratory illness or likelihood of future illness as determined by the respiratory illness inference engine 278 or the user's clinician diagnosis of a respiratory infection. It is also possible to make it look like this. Transmission risk can be binary (e.g., user is likely to transmit infection/not), categorical (e.g., low, medium, or high risk of transmission), or probability or transmission, which can indicate the likelihood of transmission. It may be determined as a risk score. In some cases, the risk of transmission may be based on a particular respiratory infection that the user has or is likely to contract (eg, influenza, rhinovirus, COVID-19, certain types of pneumonia, etc.). To this end, rules may specify that a user with a particular disease (eg, COVID-19) is susceptible to infection for a set period of time that may be fixed or variable based on the user's condition. For example, the rule may specify that the user remains susceptible to infection for 24 hours after the respiratory illness inference engine 278 determines that the user is no longer likely to have a respiratory infection. . Additionally, the risk of transmission may be constant over the entire period of a user developing (or likely to develop) a respiratory infection, or may be based on the status or evolution of the user's respiratory infection. It may also change depending on the situation. For example, the risk of transmission may be determined by the user's respiratory illness (i.e., over the most recent time period (e.g., in the past week or since the user's likelihood of having a respiratory infection was first determined by the respiratory illness inference engine 278). It may change based on a detected change, trend, pattern, rate of change, or analysis of audio-related data). Transmission risk may be provided to the user or recommended to the user (e.g., by respiratory disease inference engine 278, another component of system 200, or a clinician). Avoidance of close contact or wearing of masks, etc.) An example of a transmission risk determined according to one embodiment of user disease inference logic 237 by respiratory disease inference engine 278 is shown in element 5314 of FIG. 5C.

In some embodiments, user disease inference logic 237 includes rules, conditions, etc. that determine and/or provide recommendations corresponding to respiratory disease, prediction, transmission risk, or other determinations by respiratory disease inference engine 278. Or it may contain instructions. This recommendation may be provided (eg, as a decision support recommendation) to an end user, such as a patient, caregiver, or clinician associated with the user. For example, a recommendation determined for a user or caregiver may include one or more recommended practices that minimize transmission, control a respiratory infection, or minimize the likelihood of exacerbation of an infection. May include matters. In some embodiments, user disease inference logic 237 accesses a database of health information that may be associated with a determined respiratory infection or other determination by respiratory disease inference engine 278 to determine at least some of this information. may include computer instructions for providing the copy to a user, caregiver, or clinician. Additionally or alternatively, these recommendations may be determined using (or selected or assembled from) information in a health information database.

In some embodiments, the user's current and/or historical information (e.g., historical voice-related data, previously determined respiratory illnesses, trends or changes in the user's respiratory illness, etc.) and/or symptoms; Recommendations may be tailored to the user based on physiological data or contextual information such as geographic location. For example, in one embodiment, information about the user may be utilized as a selection or filtering criterion to identify relevant information in a database of health information used to determine recommendations that are matched to the user. .

The recommendation may be provided to the user, caregiver, or clinician, and/or stored in the patient record 240 associated with the user, such as results/inference status 246. In some embodiments of accessing a health information database, the database may be stored in storage 250 and/or in a remote server or cloud environment. An example of a recommendation determined according to one embodiment of user disease inference logic 237 by respiratory disease inference engine 278 is shown in element 5315 of FIG. 5C.

Also, as shown in FIG. 2, the example system 200 includes determination or storage (e.g., decision support tools that may include various computer applications or services that capture output determinations of the components of system 200, such as the user's respiratory illness or predictions obtained from results/inference states 246) of the user's patient record 240; 290. According to some embodiments, decision support tool 290 may utilize this information to enable therapeutic and/or preventive actions. In this manner, decision support tool 290 may be adapted for use by a monitoring user and/or a caregiver of a monitoring user. Decision support tool 290 may take the form of a stand-alone application on a client device, a web application, a distributed application or service, and/or a service on an existing computer application. In some embodiments, one or more decision support tools 290 are part of a respiratory infection monitoring or tracking application, such as respiratory infection monitoring app 5101 described in connection with FIG. 5A.

An exemplary decision support tool includes disease monitor 292. Disease monitor 292 may include an app that runs on a user's smartphone (or user equipment such as a smart speaker). The Disease Monitor 292 app monitors the user's speech to alert the user and/or the user's care provider whether the user has or is recovering from a respiratory infection, such as rhinovirus or influenza. You may also do so. In some embodiments, disease monitor 292 may listen to the user and request permission to collect audio-related data or, in some aspects, other data. The disease monitor 292 may generate notifications or alerts to the user indicating whether the user is sick, at risk of becoming sick, or is recovering. In some embodiments, disease monitor 292 may initiate and/or schedule treatment recommendations based on the determination and/or prediction of respiratory disease. The notification or alert may include recommended actions for interventional actions, such as treatment, based on the determination and/or prediction of the respiratory disease. Treatment recommendations may include, by way of non-limiting example, recommended actions for the user to take (e.g., wearing a mask), over-the-counter medications, consultation with a clinician, and/or checking for the presence of a respiratory infection. Treatment of the disease and/or resultant symptoms may include tests in which treatment is recommended. For example, disease monitor 292 may recommend that the user schedule a visit with a health care provider and/or undergo a test to confirm a respiratory illness. In some embodiments, disease monitor 292 may initiate or facilitate scheduling of doctor appointments and/or test appointments. Alternatively or additionally, disease monitor 292 may recommend or prescribe treatment, such as over-the-counter medications.

Embodiments of the disease monitor 292 may recommend that the user notify others in the home to take precautions, such as minimum distancing, to prevent the spread of infectious diseases. In some embodiments, disease monitor 292 recommends this notification and, if the user affirmatively approves this notification, sends the notification to user equipment associated with other users in the infected user's home. You may start it. Disease monitor 292 may identify associated user equipment, such as user account/equipment 248, through information stored in the user's patient record 240. In some embodiments, disease monitor 292 includes other sensed data (e.g., physiological data such as heart rate, body temperature, sleep, etc.), other contextual data such as information regarding the prevalence of respiratory infections in the user's region. , or user-input data (such as symptom information provided via self-reporting tool 284) may be correlated with the determination and/or prediction of respiratory disease to make recommendations.

In one embodiment, the disease monitor 292 may be part of or may be adapted to work in conjunction with an infectious disease contact tracing application. In this way, information regarding early detection of a possible respiratory infection of the first user may be automatically communicated to others with whom the first user has come into contact. Additionally or alternatively, the information may be used to initiate respiratory infection monitoring of these others. For example, another person may be notified of a possible exposure to an infected person and may download and use a respiratory infection monitoring application, such as disease monitor 292 or respiratory infection monitoring app 5101 described in connection with FIG. 5A. You may be prompted to do so. In this way, it is possible to notify others and start monitoring even before the first user becomes unwell (that is, before the first user feels symptoms).

As shown in FIG. 2, another example decision support tool 290 is a prescription monitor 294. Prescription monitor 294 may utilize determinations and/or predictions regarding the user's respiratory illness, such as whether the user has a respiratory infection, to determine whether the prescription should be refilled. good. Prescription monitor 294 may determine, for example, from the user's personal record 240, whether the user has a current prescription for a detected or predicted respiratory condition. The prescription monitor 294 may also determine prescription instructions regarding the frequency of dosing of the medication, the date of the last fill of the medication, and/or the number of refills available. Prescription monitor 294 determines whether a prescription needs to be refilled based on a determination that the user currently has a respiratory infection or a prediction that the user will develop a respiratory infection or exhibit symptoms in the near future. It may be determined whether or not.

Additionally, some embodiments of prescription monitor 294 may determine whether a user is taking a medication through sensing data or user input via self-reporting tool 284. Information indicating whether the user is taking the prescribed medication is used by the prescription monitor 294 to determine if or when the current prescription will run out. Prescription monitor 294 may issue an alert or notification indicating to the user that the prescription has been refilled. In one embodiment, the prescription monitor 294 issues a notification recommending refilling the prescription after the user takes active action to request a refill. Prescription monitor 294 may initiate refill orders through the pharmacy, which pharmacy information may be stored in the user's patient record 240 or entered by the user at the time of refill. Good too. Aspects of an exemplary prescription monitoring service, such as prescription monitor 294, are shown in FIG. 4F.

As shown in FIG. 2, another example decision support tool 290 is a drug efficacy tracker 296. Medication efficacy tracker 296 utilizes determinations and/or predictions regarding the user's respiratory illness, such as whether the user's illness is improving or worsening, to determine whether the medication being taken by the user is effective. It may be determined whether or not. To this end, drug efficacy tracker 296 may determine from the user's patient record 240 whether the user has a current prescription. Medication efficacy tracker 296 may determine whether the user is actually taking the medication through sensing data or user input via self-reporting tool 284 . The drug efficacy tracker 296 may also determine prescription instructions and determine whether the user is taking the medication according to the prescription instructions.

In some embodiments, drug efficacy tracker 296 determines whether the user is taking a drug by correlating inferences or predictions regarding respiratory illness based on the utilization of audio-related data; , it may be determined whether the drug is effective or not. For example, if a user is taking a medication as prescribed, but the respiratory illness is worsening or not improving, it may be determined that the prescribed medication is not effective for the particular user at this time. To this end, the drug efficacy tracker 296 may recommend that the user consult a clinician to change the prescription, or may automatically communicate an electronic notification to the user's physician or clinician. , may allow the clinician to consider modifications to the default treatment.

In some embodiments, drug efficacy tracker 296 operates on or interfaces with a monitoring user's clinician's equipment, such as clinician user equipment 108 of FIG. 1, in addition to or as an alternative to the above. For example, a clinician may prescribe a drug, such as an antibiotic for a respiratory infection, to a sick patient by instructing the patient to use a drug efficacy tracking application (e.g., 296). , patient audio-related data may be monitored in accordance with embodiments of the present disclosure. Determining that the user is getting worse or not improving may cause the drug efficacy tracker 296 to inform the clinician of an inference or prognosis of the patient's respiratory illness. In some cases, drug efficacy tracker 296 may further make recommendations to change the default treatment for the patient.

In another embodiment, the drug efficacy tracker 296 may be utilized as part of a drug study or trial and may be used to determine and/or predict respiratory disease in multiple participants. The analysis may determine whether a study drug is effective for a group of participants. Additionally or alternatively, in some embodiments, respiratory-related side effects (e.g., cough, stuffy nose, runny nose) or non-respiratory-related side effects (e.g., fever, nausea, inflammation, swelling, itching), etc. The drug efficacy tracker 296 may be utilized as part of a study or test in conjunction with a sensor (eg, sensor 103) to determine if there are side effects of the drug.

Some embodiments of the decision support tool 290 described above include aspects of treating a user's respiratory illness. The treatment may be aimed at reducing the severity of the respiratory disease. Treating respiratory diseases may include new therapeutic agents, new drug dosages or new dosages of existing drugs being taken by the user, and/or new methods of administering the drug or taking the user. may include determining new treatment procedures, which may include new dosing methods for existing drugs being used. Recommendations for new treatment procedures may be provided to the user or the user's caregiver. In some embodiments, the prescription may be sent to the user, the user's caregiver, or the user's pharmacy. In some cases, treatment may include refilling the existing prescription without modification. Other embodiments may include dosing the user with a recommended treatment in accordance with a recommended treatment procedure and/or tracking application or use of the recommended treatment. Thus, embodiments of the present disclosure allow for better control, monitoring, and/or management of the use or application of therapeutic agents to treat respiratory diseases, which may depend on the user's condition. Not only is it beneficial, it can also help healthcare providers and pharmaceutical manufacturers, as well as others in the supply chain, better comply with regulations and recommendations set by the Food and Drug Administration and other governing bodies.

In an exemplary embodiment, the treatment includes:
PLpro inhibitors (apiromod, EIDD-2801, ribavirin, valganciclovir, β-thymidine, aspartame, oxprenolol, doxycycline, acetophenazine, iopromide, riboflavin, leproterol, 2,2'-cyclotidine, chloramphenicol, chlor Phenesine carbamate, levodropropidine, cefamandole, floxuridine, tigecycline, pemetrexed, L(+)-ascorbic acid, glutathione, hesperetin, ademethionine, masoprocol, isotretinoin, dantrolene, sulfasalazine antibacterial, silibinin, nicardipine , sildenafil, platycodin, chrysin, neohesperidin, baicalin, sugetriol-3,9-diacetate, (-)-epigallocatechin gallate, phytantrin D, 2-(3,4-dihydroxyphenyl)-2-[ [2-(3,4-dihydroxyphenyl)-3,4-dihydro-5,7-dihydroxy-2H-1-benzopyran-3-yl]oxy]-3,4-dihydro-2H-1-benzopyran-3 , 4,5,7-tetrol, 2,2-di(3-indolyl)-3-indolone, (S)-(1S,2R,4aS,5R,8aS)-1-formamide-1,4a-dimethyl- 6-methylene-5-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-2-amino-3-phenylpropanoate, piceatannol, rosmarinic acid, and/or magnol),
3CLpro inhibitors (rimesycline, chlorhexidine, alfuzosin, cilastatin, famotidine, almitrin, progabid, nepafenac, carvedilol, amprenavir, tigecycline, montelukast, carminic acid, mimosine, flavin, lutein, cefpiramide, pheneticillin, candoxatril, nicardipine, Estradiol valerate, pioglitazone, conivaptan, telmisartan, doxycycline, oxytetracycline, (1S,2R,4aS,5R,8aS)-l-formamide-l,4a-dimethyl-6-methylene-5-((E)-2- (2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl 5-((R)-1,2-dithiolan-3-yl)pentanoate, vetronal, chrysin-7- Ο-β-glucuronide, andrographiside, (1S,2R,4aS,5R,8aS)-l-formamide-l,4a-dimethyl-6-methylene-5-((E)-2-(2-oxo- 2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-2-nitrobenzoate, 2β-hydroxy-3,4-secofriederolactone-27-oic acid (S)-(1S, 2R,4aS,5R,8aS)-1-formamido-1,4a-dimethyl-6-methylene-5-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl) Decahydronaphthalen-2-yl-2-amino-3-phenylpropanoate, isodecortinol, cerebisterol, hesperidin, neohesperidin, andrograpanin, 2-((1R,5R,6R,8aS)-6 -Hydroxy-5-(hydroxymethyl)-5,8a-dimethyl-2-methylenedecahydronaphthalen-1-yl)ethylbenzoate, cosmosyin, cleistokalton A, 2,2-di(3-indolyl)-3- Indolone, violobin, gunizicin, filaembrinol, theaflavin 3,3'-di-Ο-gallate, rosmarinic acid, koichenside I, oleanolic acid, stigmast-5-en-3-ol, deacetylcentapicrin, and / or Berkemol),
RdRp inhibitors (valganciclovir, chlorhexidine, ceftibuten, fenoterol, fludarabine, itraconazole, cefuroxime, atovaquone, chenodeoxycholic acid, cromolyn, pancuronium bromide, cortisone, tibolone, novobiocin, silibinin, idarubicin bromocriptine, diphenoxylate, benzylpenicilloyl G) , dabigatran etexilate, vetronal, gunidicine, 2β,30β-dihydroxy-3,4-secofriederolactone-27-lactone, 14-deoxy-11,12-didohydroandrographolide, gnidithrin, theaflavin 3,3 '-di-Ο-gallate, (R)-((1R,5aS.6R,9aS)-1,5a-dimethyl-7-methylene-3-oxo-6-((E)-2-(2-oxo -2,5-dihydrofuran-3-yl)ethenyl)decahydro-1H-benzo[c]azepin-1-yl)methyl 2-amino-3-phenylpropanoate, 2β-hydroxy-3,4-secofurie Derolactone-27-oic acid, 2-(3,4-dihydroxyphenyl)-2-[[2-(3,4-dihydroxyphenyl)-3,4-dihydro-5,7-dihydroxy-2H-1- benzopyran-3-yl]oxy]-3,4-dihydro-2H-1-benzopyran-3,4,5,7-tetron, filaemblycin B, 14-hydroxycyperotandone, andrographiside, 2-( (1R,5R,6R,8aS)-6-hydroxy-5-(hydroxymethyl)-5,8a-dimethyl-2-methylenedidecahydronaphthalen-1-yl)ethylbenzoate, andrographolide, sugetriol- 3,9-diacetate, baicalin, (1S,2R,4aS,5R,8aS)-1-formamide-1,4a-dimethyl-6-methylene-5-((E)-2-(2-oxo-2 ,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-5-((R)-1,2-dithiolan-3-yl)pentanoate, l,7-dihydroxy-3-methoxyxanthone, l,2,6-trimethoxy-8-[(6-Ο-β-D-xylopyranosyl-β-D-glucopyranosyl)oxy]-9H-xanthen-9-one, and/or 1,8-dihydroxy-6- Methoxy-2-[(6-Ο-β-D-xylopyranosyl-β-D-glucopyranosyl)oxy]-9H-xanthen-9-one, 8-(β-D-glucopyranosiloxy)-l,3, 5-trihydroxy-9H-xanthene-9-one),
including one or more of the following therapeutic agents:

In an exemplary embodiment, treatment includes one or more therapeutic agents that treat viral infections such as SARS-CoV-2, which causes COVID-19. Therefore, therapeutic agents may include one or more SARS-CoV-2 inhibitors. In some embodiments, treatment includes a combination of one or more SARS-CoV-2 inhibitors with one or more of the therapeutic agents listed above.

In some embodiments, the treatment includes any of the above specific therapeutic agents and Diosmin, Hesperidin, MK-3207, Venetoclax, Dihydroergocristine, Borazine, R428, Diterqualinium, Etoposide, Teniposide, UK-432097, Irinotecan, lumacaftor, velpatasvir, eluxadrine, ledipasvir, lopinavir/ritonavir + ribavirin, alferon, prednisone,
dexamethasone, azithromycin and remdesivir and boceprevir, umifenovir and favipiravir,
Zhang, L. ;Lin,D. ; Sun, X. ; Rox, K. ;Hilgenfeld,R. ;X-ray Structure of Main Protease of the Novel Coronavirus SARS-CoV-2 Enables Design of a-Ketoamide Inhibitors It is possible to design α-ketoamide inhibitors from the X-ray structure of V-2 main protease. ); bioRxiv preprint doi: https://doi. α-ketoamide compounds Hr, 13a, and 13b as described in org/10.1101/2020.02.17.952879,
RIG1 pathway activators as described in U.S. Pat. No. 9,884,876;
Dai W, Zhang B, Jiang X-M, et al. Structure-based design of antiviral drug candidates targeting the SARS-CoV-2 main protease. Science. 2020;368(6497):1331-1335 (including compounds designated as DC402234), and/or remdesivir, galidesivir, favilavir/avifavir, molnupiravir (MK-4482/EIDD 2801), AT-527, AT-301, BLD-2660, Favipiravir, Camostat, SLV213 Emtricabine/Tenofivir, Clevudine, Dalcetrapib, Boceprevir, ABX464, (3S)-3-({N-[(4-methoxy-1H-indole-2) -yl)carbonyl]-L-leucyl}amino)-2-oxo-4-[(3S)-2-oxopyrrolidin-3-yl]butyl dihydrogen phosphate and/or a pharmaceutically acceptable salt thereof , solvate, or hydrate (PF-07304814), (1R,2S,5S)-N-{(1S)-1-cyano-2-[(3S)-2-oxopyrrolidin-3-yl] ethyl}-6,6-dimethyl-3-[(3-methyl-N-(trifluoroacetyl)-L-valyl]-3-azabicyclo[3.1.0]hexane-2-carboxamide or its solvate or hydrate (PF-07321332), and/or antiviral agents such as S-217622, glucocorticoids such as dexamethasone and hydrocortisone, convalescent plasma, recombinant human plasma such as Gelsolin (Rhu-p65N), regdanvimab (Requirona) , monoclonal antibodies such as ravlizumab (Ultomiris), VIR-7831/VIR-7832, BRII-196/BRII-198, COVI-AMG/COVI DROPS (STI-2020), bamlanivimab (LY-CoV555), mavrilimab, leronlimab (PROMO) ), AZD7442, lenzilumab, infliximab, adalimumab, JS 016, STI-1499 (COVIGUARD), lanadelumab (Taczylo), canakinumab (Ilaris), gimcilumab, otilimab, casirivimab/imdevimab (REGN-Cov2), etc. antibody cocktail, MK-7110 (CD24Fc/SACCOVID), anticoagulants such as heparin and apixaban, IL-6 receptor agonists such as tocilizumab (Actemra) and/or sarilumab (Kevzara), PIKfyve inhibitors such as apilimod dimethylate, RIPK1 inhibitors such as DNL758, DC402234, VIP receptor agonists such as PB1046, SGLT2 inhibitors such as dapaglifodin, TYK inhibitors such as abivertinib, ATR-002, bemcentinib, acalabrutinib, rosmapimod, baricitinib, and/or tofacitinib, etc. Kinase inhibitors, H2 inhibitors such as famotidine, anthelmintics such as niclosamide, furin inhibitors such as diminazene,
one or more therapeutic agents selected from:

For example, in one embodiment, the treatment comprises (3S)-3-({N-[(4-methoxy-1H-indol-2-yl)carbonyl]-L-leucyl}amino)-2-oxo-4- [(3S)-2-oxopyrrolidin-3-yl]butyl dihydrogen phosphate, and pharmaceutically acceptable salts, solvates, or hydrates thereof (PF-07304814). In another embodiment, the treatment includes (IR,2S,5S)-N-{(1S)-1-cyano-2-[(3S)-2-oxopyrrolidin-3-yl]ethyl}-6, 6-dimethyl-3-[3-methyl-N-(trifluoroacetyl)-L-valyl]-3-azabicyclo[3.1.0]hexane-2-carboxamide or its solvate or hydrate (PF -07321332).

Continuing with FIG. 2 and system 200, presentation component 220 of system 200 generally provides user instructions and/or feedback for obtaining detected respiratory disease information, user voice data and/or self-report data, and related It may also be responsible for providing information. Presentation component 220 may include one or more applications or services on a user device, on multiple user devices, or in a cloud environment. For example, in one embodiment, presentation component 220 may manage the provision of information, such as notifications and alerts, to a user on multiple user devices associated with the user. Based on the presentation logic, context, and/or other user data, the presentation component 220 determines the user equipment on which the content is provided, as well as the method of presentation (e.g., format and content (which may be determined by the user equipment or context)). The context of the provision, such as the timing of the provision, or other aspects regarding the provision of the information, may be determined.

In some embodiments, the presentation component 220 is associated with aspects of other components of the system 200, such as a user voice monitor 260, a user interaction manager 280, a respiratory disease tracker 270, and a decision support tool 290. generating user interface functions that are used to facilitate the presentation of such aspects to a user, who may be the patient being monitored or the clinician of the monitoring patient; You may also do so. Such functionality may include graphical or audio interface elements such as icons or indicators, graphic buttons, sliders, menus, sounds, audio prompts, alerts, alarms, vibrations, pop-up windows, notification bars or status bar items, in-app notifications, or (such as other similar functions for interfacing with the user), queries, and prompts. Some embodiments of presentation component 220 may employ speech synthesis, text-to-speech conversion, or similar functionality to generate and present utterances to a user, such as embodiments operating on a smart speaker. Good too. For an example of a graphical user interface (GUI) and a description of example audio user interface elements that may be generated by presentation component 220 and provided to a user (i.e., a monitoring patient or clinician), see FIGS. 5A-5E. It is described as follows. Embodiments that utilize audio user interface functionality are illustrated in the embodiments of FIGS. 4C-4F. Some embodiments of an audio user interface provided by presentation component 220 include a voice user interface (VUI), such as a VUI on a smart speaker. Also, an example of a graphical user interface (GUI) and a description of example audio user interface elements that may be generated by presentation component 220 and provided to a user (i.e., a monitoring patient or clinician), such as smart watch 402a of FIG. Also shown and described in connection with a wearable device.

Storage 250 of example system 200 generally stores data, computer instructions (e.g., software program instructions, routines, or services), logic, profiles, and/or models used in embodiments described herein. The information may be stored. In one embodiment, storage 250 may comprise a data store (or computer data memory), such as data store 150 of FIG. Additionally, although shown as a single data store component, storage 250 may be implemented as one or more data stores or in a cloud environment.

As shown in example system 200, storage 250 includes speech-to-phoneme extraction logic 233, phoneme feature comparison logic 235, and user disease inference logic 237, all of which are described above. Additionally, storage 250 may include one or more patient records (such as patient record 240, as shown in FIG. 2). Patient records 240 are associated with a particular monitoring patient/user, such as profile/health data (EHR) 241, voice samples 242, phoneme feature vectors 244, results/inference states 246, user accounts/equipment 248, and settings 249. May contain information. As described herein, the information stored in the patient record 240 may be stored in the data collection component 210, the user voice monitor 260, the user interaction manager 280, the respiratory disease tracker 270, the decision support tool 290, or may be available to other components of the system 200.

Profile/Health Data (EHR) 241 may provide information regarding the health of the monitored patient. Embodiments of profile/health data (EHR) 241 may include some or all of a patient's EHR, or may include only some health data related to respiratory disease. For example, the profile/health data (EHR) 241 may include past or current diagnostic conditions, such as influenza, rhinovirus, COVID-19, chronic obstructive pulmonary disease (COPD), asthma, or diseases that affect the respiratory system; Medications, weight, or age associated with respiratory disease treatment or underlying symptoms of respiratory disease may be indicated. Profile/health data (EHR) 241 may include user self-reported information, such as self-reported symptoms described in conjunction with self-reporting tool 284.

Audio samples 242 may include raw data and/or processed audio-related data, such as data received from sensor 103 (shown in FIG. 1). This sensor data may include data used for respiratory infection tracking, such as voice recordings or samples collected. In some cases, the audio samples 242 may be temporarily stored until phoneme vector analysis is performed on the collected samples and/or until a predetermined period of time has elapsed. good.

Furthermore, phoneme characteristic vector 244 may include phoneme characteristics and/or phoneme characteristic vectors determined for a particular user. The phoneme feature vector 244 is correlated to other information in the patient record 240 such as contextual or self-reported information or a composite symptom score (which may be part of the profile/health data (EHR) 241). It may be as follows. Further, the phoneme characteristic vector 244 may include information that defines a phoneme characteristic standard for a specific user as described in conjunction with the phoneme characteristic comparison logic 235.

Results/inference states 246 may include the user's expectations and the user's inferred respiratory conditions. The results/inference state 246 may be an output by the respiratory disease inference engine 278 and thus include the monitoring user's score and/or likelihood of a respiratory disease at the current or future time interval. Good too. Results/inference state 246 may be utilized by decision support tool 290, as described above.

User account/equipment 248 may generally include information regarding user computer equipment that is accessed, used, or otherwise associated with the user. Examples of such user equipment include user equipment 102a-102n in FIG. may include other equipment that may be communicatively coupled to.

In one embodiment, user accounts/devices 248 include accounts associated with the user (e.g., online or cloud-based accounts (e.g., online health record portals, networks/health providers, network websites, decision support applications, social It may also include information related to media, email, telephone, e-commerce websites, etc.). For example, user accounts/equipment 248 may include monitoring patient accounts for decision support applications such as decision support tools 290, care provider site accounts (e.g., available for electronic booking), and treatment online may include an online e-commerce account, such as Amazon or drugstore, that can be used to place orders.

User accounts/devices 248 may also include the user's calendar, reservations, application data, other user accounts, and the like. Some embodiments of user account/device 248 may store information across one or more databases, knowledge graphs, or data structures. As discussed above, the information stored in user account/device 248 may be determined by data collection component 210.

Further, settings 249 generally include one or more steps of monitoring user voice data, including collecting voice data, collecting self-reported information, or inferring and/or predicting respiratory illness in the user. It may include user settings or preferences that are associated or associated with one or more decision support applications, such as decision support tool 290. For example, in one embodiment, settings 249 may include settings for collecting audio-related data, such as settings for collecting audio information as the user speaks on a daily basis. Settings 249 may include settings or preferences for contextual information, including settings for obtaining physiological data (eg, information associating a wearable sensor device). Settings 249 may further include privacy settings, as described herein. Some embodiments of settings 249 may specify specific phonemes or phoneme characteristics to detect or monitor respiratory disease, and further specify detection or inference thresholds (e.g., disease change thresholds). You may also specify it. Also, as described herein, settings 249 may include settings for a user to set a baseline condition for each respiratory disease. As a non-limiting example, other settings include user notification acceptance thresholds that may define when and how the user would like to be notified of a determination or prediction of a respiratory disease. In some aspects, settings 249 may include user preferences for applications, such as notifications, preferred caregivers, preferred stores such as pharmacies, and over-the-counter medications. Settings 249 may include treatment instructions for the user, such as prescription drugs. In one embodiment, settings 249 may also store calibration, initialization, and settings for a sensor (such as sensor 103 in FIG. 1).

Referring now to FIG. 3A, a diagrammatic representation of an example process 3100 incorporating at least some of the components of system 200 is shown. The example process 3100 shows one or more users 3102 providing data via a voice-symptom application 3104 that may operate on user equipment such as a smart mobile device and/or a smart speaker. There is. The data provided via the voice-symptom application 3104 may include an audio recording (e.g., audio sample 242 of FIG. 2) from which phonemes may be extracted, as described with respect to the user audio monitor 260 of FIG. . The received data also includes symptom ratings that may be manually entered by the user, as described in conjunction with user interaction manager 280.

Based on receiving the recorded voice samples and symptom values, a computer system residing on a server (e.g., server 106 of FIG. 1) and accessible via a network (e.g., network 110 of FIG. Operations 3106 may include interacting with a user, executing a symptom algorithm, extracting audio characteristics, and applying an audio algorithm. User interactions may include prompts and feedback to gather usable data, as described in conjunction with user interaction manager 280. The symptom algorithm may include generating a composite symptom score (CSS) based on the user's self-reported symptom values, as described in conjunction with self-report data evaluator 276. Extracting voice characteristics may include extracting acoustic characteristic values for phonemes detected in the voice sample, as described in conjunction with user voice monitor 260 and, more specifically, acoustic characteristic extractor 2614. The extracted acoustic features are then subjected to an audio algorithm, which may include comparing feature vectors from different days of the patient (i.e., computing a distance metric), as described in conjunction with the phoneme feature comparator 274. You can leave it there.

Based on at least some of the operations 3106, reminders and notifications may be configured to be electronically sent by user equipment, such as user equipment 102a of FIG. 1, to one or more users 3102. . The reminder may inform the user that additional information may be required, such as an audio sample or self-reported symptom assessment. As described with respect to user interaction manager 280, the notification provides feedback to the user indicating whether a longer duration, louder volume, or less background noise is required when providing the audio sample. can be given. The notification may also indicate whether and to what extent the user followed predefined procedures for providing audio samples and, in some cases, symptom information. For example, the notification may indicate that the user has completed 50% of the voice exercise to provide a voice sample.

Also, based at least in part on acts 3106, the collected information and/or analysis results thereof are transmitted to one or more user equipment associated with the clinician, such as clinician user equipment 108 of FIG. It may be possible to do so. Clinician dashboard 3108 is also generated by a computer software application, such as decision support app 105a or 105b, operating on or in conjunction with clinician user equipment 108 (of FIG. 1). It may be as follows. Clinician dashboard 3108 provides access and reception of information regarding a particular patient or set of patients being monitored (i.e., monitoring users 3102) and, in some embodiments, direct or indirect interactions with patients. It may include a graphical user interface (GUI) to enable communication. Clinician dashboard 3108 may include views that present information to multiple users (such as a chart with each row containing information about a different user). Additionally or alternatively, clinician dashboard 3108 may present information to a single monitored user.

In one embodiment, clinician dashboard 3108 may be utilized by a clinician to monitor data collection of user 3102 via audio-symptom application 3104. For example, the clinician dashboard 3108 may indicate whether the user has provided available audio samples and, in some embodiments, a symptom severity rating. The clinician dashboard 3108 may notify the clinician if the user is not following predefined procedures for providing audio samples and/or other information. In some embodiments, clinician dashboard 3108 allows clinicians to communicate reminders to users to follow procedures or revised procedures for collecting data (e.g., sending electronic messages). ) functions may be included.

In some embodiments, operation 3106 may include determining a respiratory illness of the user from the collected audio samples (e.g., determining whether the user is ill). , which may be performed by one embodiment of a respiratory disease tracker 270 in general, and a respiratory disease inference engine 278 more specifically, as described in conjunction with FIG. In these embodiments, a notification may be sent to user 3102 indicating the determined respiratory illness. In some embodiments, the notification to the user 3102 may include a recommendation for action, as described in conjunction with the decision support tool 290. Further, if the user's voice-related information is utilized to determine the user's respiratory illness, some embodiments of the clinician dashboard 3108 may be used by the clinician to track the user's respiratory illness. You can. In some embodiments of clinician dashboard 3108, the user's respiratory illness status (e.g., respiratory illness score, whether the user has a respiratory infection or not) and/or the user's illness trend (e.g., , whether the user's disease is worsening, improving, or unchanged. Alerts or notifications can be used to indicate whether the user's illness is particularly bad (if the respiratory illness score is below a threshold score), whether a new infection has been detected in the user, and/or whether the user's illness has changed. It may be provided to a doctor.

In some embodiments, clinician dashboard 3108 allows clinicians to monitor the status and efficacy of predefined treatments (including side effects of such treatments), as described with respect to decision support tool 290 and drug efficacy tracker 296. may be utilized for close monitoring of users who have been prescribed a drug for a respiratory infection and/or who have been diagnosed with a respiratory disease by a clinician. Thus, embodiments of clinician dashboard 3108 may identify prescribed medications or predefined treatments and whether a user is taking a prescribed medication or performing a predefined treatment.

Further, in some embodiments, the clinician dashboard 3108 includes information about recommended or required voice sample collection procedures (e.g., how often the user should provide voice samples), the user's default treatment or prescribed medications, and additional information for the user. may include functionality that allows clinicians to set clinical recommendations (e.g., fluid intake, rest, avoidance of exercise, self-quarantine). Clinician dashboard 3108 may also be used by clinicians to configure or adjust monitoring settings (e.g., thresholds that generate alerts for clinicians and, in some embodiments, users). You can leave it there. In some embodiments, the clinician dashboard 3108 also allows the clinician to determine whether the voice-symptom application 3104 is operating properly and to perform diagnostics on the voice-symptom application 3104. May contain functions.

FIG. 3B illustratively illustrates a diagrammatic representation of an example process 3500 for collecting data for monitoring respiratory disease. In this example process 3500, a monitoring patient may perform multiple collection checkpoints where audio samples and symptom assessments are provided. Collection checkpoints include one "in-sick" laboratory visit where the patient has experienced symptoms of a respiratory infection or, in some embodiments, a diagnosis of a respiratory infection; May include one "wellness" laboratory visit where the patient has recovered from an infectious disease. Patients may also have twice-daily (or once-daily or periodic) collection checkpoints at home between two laboratory visits. Home checkpoints are conducted over a period of at least two weeks, and may be longer than two weeks if the patient's recovery period is longer than two weeks. At each collection checkpoint, the patient may provide a voice sample and symptom assessment.

The laboratory visit may be a visit by the clinician, such as the clinician's office or a laboratory conducting research. During the laboratory visit, monitoring patient voice samples may be simultaneously recorded through the smartphone and computer coupled to the headset. However, it is also contemplated that embodiments of process 3500 may utilize only one of these methods for collecting audio samples during a laboratory visit. Patients may utilize smartphones, smart watches, and/or smart speakers for home collection to record voice samples and provide symptom assessments.

For voice samples from both laboratory and home visits, patients may be asked to record sustained vocalizations of 5 to 10 seconds each, both nasal and fundamental vowels. In one embodiment, four vowels and three nasals are recorded. The four vowels using the International Phonetic Alphabet (IPA) are:

The patient may have a more familiar beginning.

In some cases, you may be asked to pronounce the words. The three nasal sounds are

It may be. Individuals may also be asked to record planned and unplanned utterances. The audio recording system may use lossless compression and have a bit depth of 16. In some embodiments, audio data may be sampled at 44.1 kilohertz (kHz). In another embodiment, the audio data may be sampled at 48kHz.

During the home recovery period, patients may be asked to provide voice samples and report symptoms every morning and night. For symptom assessment during the home period, patients may be asked to rate the severity (0-5) of perceived symptoms associated with airway disease for 19 symptoms in the morning and 16 symptoms in the evening. . In one embodiment, the four sleep questions are only included in the morning list, and the end-of-day fatigue question is asked only in the evening. An exemplary list of symptom questions may be provided in conjunction with self-report tool 284. A composite symptom score (CSS) may be determined by summing the scores of at least some of these symptoms. In one embodiment, CSS is the sum of seven symptoms (postnasal drip, nasal congestion, runny nose, mucus nasal discharge, cough, sore throat, need to blow the nose).

4A-4F each illustratively illustrate example patient (ie, user 410) scenarios utilizing embodiments of the present disclosure. User 410 may run a computer software application (e.g., decision support application 105a of FIG. 1) running on user equipment (e.g., any of user computing equipment 102a-102n), as described with respect to presentation component 220 of FIG. It may interact with one or more user interfaces (eg, a graphical user interface and/or an audio user interface). Each scenario is represented by a series of scenes (boxes) that are intended to be ordered chronologically (from left to right). Although different scenes (boxes) are not necessarily different discrete interactions, they are considered part of one interaction between the user 410 and the user interface component.

4A, 4B, and 4C may include one or more audio-symptom application 3104 of FIG. 3A and/or one embodiment of respiratory infection monitoring app 5101 of FIGS. 5A-5E, as described below. It shows data, such as user voice information, being collected from the user 410 through interaction with an app or program running on the user device. The embodiments shown in FIGS. 4A-4C may be performed by one or more components of system 200, such as user interaction manager 280, data collection component 210, and presentation component 220. good.

For example, referring to FIG. 4A, in scene 401, a user 410 using a smartphone 402c (which may be an embodiment of user equipment 102c of FIG. 1) is provided with instructions 405 to provide a sustained vocalization. It will be done. The content of the instruction 405 is ``Let's start evaluating the voice state. From now on, please continue to say the sound ``Oooh'' for 5 seconds.'' These instructions 405 may be provided by one embodiment of the user instruction generator 282 of FIG. 2. The instructions 405 may be displayed as text via a graphical user interface on the display screen of the smartphone 402c. Additionally or alternatively, the instructions 405 may be provided as audible instructions utilizing a voice user interface on the smartphone 402c. Scene 402 shows a user 410 providing a voice sample 407 by saying the words "Oooh..." to a smartphone 402c, and the microphone (not shown) of the smartphone 402c captures this voice sample 407. can be picked up and recorded.

FIG. 4B similarly shows that in scene 411 instructions 415 are provided to user 410. Instructions 415 may be generated by an embodiment of user instruction generator 282, which may also be generated by smartwatch 402a (which may be an exemplary embodiment of user equipment 102a of FIG. 1). given through. As such, instructions 415 may be displayed as text via a graphical user interface on smartwatch 402a. Additionally or alternatively, instructions 415 may be provided as audible instructions via an audio user interface. In scene 412, user 410 responds to instruction 415 by producing an audio sample 417 ("Aaaa...") by speaking to smartwatch 402a.

FIG. 4C shows that user 410 is prompted to provide audio samples through a series of instructions (sometimes referred to as prompts) from smart speaker 402b (which may be one embodiment of user equipment 102b of FIG. 1). It shows that you are being guided. These instructions may be output from smart speaker 402b via an audio user interface, and responses from user 410 may be communicated to a microphone (not shown) on smart speaker 402b or to smart speaker 402b. It may also be an audible response picked up by another device to which it is coupled.

Also, according to some embodiments of the present disclosure, FIG. 4C illustrates an audio recording session initiated by an application or program operating on or in conjunction with smart speaker 402b. For example, in scene 421, smart speaker 402b states aloud an intention 424 to start an audio recording session. The content of the intention 424 is "Let's begin the audio state evaluation. Are you sure now?" to which the user 410 gives an audible response 425 of "yes."

In scene 422, smart speaker 402b provides an audible instruction 426 that causes user 410 to provide an audio sample. User 410 provides an audible response 427 that includes a normal affirmation ("OK") and a directed sound ("aaaa..."). If it is determined that the user has responded, the next set of instructions shall be given to obtain another voice sample. Determining the user's 410 response and providing appropriate feedback or next steps to the user 410 may be performed by one embodiment of the user input response generator 286. In scene 423, an instruction 428 for the next audio sample is emitted from smart speaker 402b, to which user 410 responds with an audible audio sample 429, "Oooh!" This back and forth instruction between smart speaker 402b and user 410 may continue until all of the required audio samples have been collected.

As described herein, audio information collected from a user may be utilized to monitor or track a user's respiratory illness. To this end, FIGS. 4D, 4E, and 4F illustrate scenarios in which the user is informed of various aspects of tracking his or her respiratory illness. The audio data utilized in the inferences and predictions of FIGS. 4D-4F may be collected in a variety of ways on various devices such as those shown in FIGS. 4A-4C. In some embodiments, the inference and prediction decisions underlying the scenarios of FIGS. 4D-4F may be made by the respiratory disease inference engine 278 of FIG. Notifications and requests for additional information may be provided by embodiments of user interaction manager 280 and/or decision support tool 290 (such as disease monitor 292).

FIG. 4D shows that the user 410 is notified of a determination of respiratory disease. In scene 431, smart speaker 402b provides an audible message 433 indicating that user 410 is determined to be potentially ill based on the latest audio data. This determination of the possibility that the user is sick may be made according to an embodiment of the respiratory disease tracker 270. Audible message 433 further requests confirmation of symptoms consistent with respiratory illness (e.g., "Do you feel stuffy, tired, or...?"), which may The input response generator 286 may be configured to do so in accordance with an embodiment of the input response generator 286. User 410 may provide an audible response 435 of "a little." In scene 432 of FIG. 4D, in response to user 410's response 435 that he feels stuffy, an additional message 437 is provided by smart speaker 402b. Additional message 437 requests symptom feedback from user 410 by asking user 410 to rate nasal congestion. This scenario of FIG. 4D may follow if the user provides a response assessing nasal congestion and/or any other symptoms.

FIG. 4E illustrates another interaction between the user 410 and the smart speaker 402b where the user's 410 voice data may continue to monitor the user's 410 respiratory illness. In an audible message 443 shown in scene 441, smart speaker 402b notifies user 410 that a previously detected respiratory illness (i.e., a cold) is being tracked, as well as updates made based on more recent data. The user 410 is notified of the determined respiratory disease. Specifically, the content of the message 443 is "...the frequency of coughing seems to be decreasing, and voice analysis shows improvement. How are you doing?" User 410 then provides an audible response 445 indicating that he is fine. In scene 442, smart speaker 402b provides an audio message 447 informing user 410 of a prediction of user's 410 future respiratory illness. Specifically, message 447 notifies user 410 that user 410 is expected to recover with respect to his respiratory illness within three days. Message 447 also provides a recommendation to continue to rest and follow doctor's instructions. In FIG. 4E, the determination that user 410's voice is improving and that the user is likely to recover within three days is determined by an embodiment of respiratory disease inference engine 278, as described in conjunction with FIG. It may be possible to do so.

FIG. 4F illustrates a scenario in which user 410's respiratory illness continues to be monitored (e.g., as indicated by message 455 in scene 451 that reads "In disease monitoring mode..."). . In scene 451, smart speaker 402b is in disease monitoring mode and produces an audible message 455 indicating that user 410 does not appear to be getting better based on an analysis of audio samples collected over the past few days. Output. Also, in message 455, smart speaker 402b asks if user 410 is taking antibiotics. Determining whether user 410 has been prescribed a medication may be made by one embodiment of prescription monitor 294. User 410 provides a response 457 ("yes") indicating that he is taking medication. In scene 452, smart speaker 402b communicates with one or more other computers over the network, as illustrated by cloud 458, based on user 410's response 457 confirming that user 410 is taking medication. Communicate to a system or device. In one embodiment, smart speaker 402b may directly or indirectly communicate to user 410's care provider that user 410's prescription will be refilled since user 410 remains ill. As a result, in scene 453, smart speaker 402b outputs an audible message 459 informing user 410 that the user's care provider has been contacted and a refill of the antibiotic prescription has been ordered.

5A-5E illustrate various example screenshots from computer equipment that illustrate aspects of an example graphical user interface (GUI) of a computer software application (or app). In particular, the exemplary embodiments of GUIs (such as GUI 5100 of FIG. 5A) shown in the screenshots of FIGS. It's called an app. Although the example apps shown in FIGS. 5A-5E are described as monitoring respiratory infections, the present disclosure generally applies to applications that monitor respiratory diseases and changes in respiratory diseases as well. This is considered to be the case.

The example respiratory infection monitoring app 5101 includes one implementation of a user voice monitor 260, a user interaction manager 280, and/or other components or subcomponents as described in connection with FIG. May contain. Additionally or alternatively, some aspects of the respiratory infection monitoring app 5101 may include one implementation of the decision support app 105a or 105b and/or one of the decision support apps 105a or 105b, as described in connection with FIGS. 1 and 2, respectively. One implementation of one or more decision support tools 290 may be included. Exemplary respiratory infection monitoring app 5101 is running on user computer equipment (or user equipment) 5102a, which may be embodied as any of user equipment 102a-102n, as described in connection with FIG. (and the GUI may be displayed on user computer equipment 5102a). Some of the GUI elements of the example GUI shown in the screenshots of FIGS. 5A-5E (such as hamburger menu icon 5107 in FIG. 5A) are selectable by the user, such as by touching or clicking on the GUI elements. Good too. Some embodiments of user computing equipment 5102a may include a touch screen or display that facilitates user interaction with the GUI, such as in conjunction with a stylus or mouse.

In some embodiments, the default or recommended standard of care for patients diagnosed with a respiratory disease (e.g., influenza, rhinovirus, COVID-19, asthma, etc.) includes (as described herein) may include the use of an embodiment of a respiratory infection monitoring app 5101 that may run on the user's/patient's own computing device, such as a mobile device or other user equipment 102a-102n) It is also contemplated that it may be provided to the user/patient through the patient's healthcare provider or pharmacy. In particular, traditional solutions for monitoring and tracking respiratory diseases are subjective (i.e., self-tracking of symptoms), and early detection is not possible or practical, among other drawbacks. There is. However, embodiments of the technology described herein can provide an objective, non-invasive, and more accurate means of monitoring, detecting, and tracking a user's respiratory disease data. . As a result, these embodiments ensure that the technology can be used for patients prescribed specific medications for respiratory conditions. In this way, a physician or health care provider can help users take medications and use a computer decision support app (e.g., respiratory infection monitoring app 5101) to, among other things, more accurately determine the efficacy of a given treatment. Instructions can be issued that can be tracked and determined. Similarly, a physician or healthcare provider may require that a patient make a computer decision prior to taking a drug so that the drug can be prescribed based on consideration of the analysis, recommendations, or output provided by the computer decision support app. Instructions can be issued (or may be specified by standard of care) including monitoring or tracking the user's respiratory illness by using the decision support app. For example, a doctor may prescribe a particular antibiotic, and a computer decision support app may determine that the user likely has a respiratory illness and does not appear to be recovering. You may also do so. Additionally, the use of computer decision support apps (e.g., respiratory infection monitoring apps 5101) as part of the standard of care for patients administered or prescribed certain medications may be helpful in determining the effectiveness (including side effects) of prescribed medications. By enabling healthcare providers to better understand, modify or change the dosage of certain prescribed medications, or instruct patients to discontinue use because the user's/patient's disease has improved and is no longer needed. aids in the effective treatment of

Referring to FIG. 5A, an example GUI 5100 is shown that illustrates aspects of an example respiratory infection monitoring app 5101 that may be used for respiratory disease monitoring and decision support of a user. For example, the respiratory infection monitoring app 5101 may be used to facilitate the acquisition of respiratory disease data for users and/or to determine, view, track, complete, or report information regarding respiratory diseases, among other purposes. An embodiment of may be used. The example respiratory infection monitoring app 5101 shown in the GUI 5100 includes a header area located near the top of the GUI 5100 that includes a hamburger menu icon 5107, a descriptor 5103, a supply icon 5104, a stethoscope icon 5106, and a cycle icon 5108. 5109 may be included. Selection of the hamburger menu icon 5107 may provide the user with access to a menu of other services, features, or features of the respiratory infection monitoring app 5101, as well as help, app version information, and secure user account signature. It may further include access to in/sign-off functionality. Descriptor 5103 may indicate the current date in this example GUI 5100. This date is the date that is associated with any audio-related data obtained by the user when the user initiates the audio data collection process, as described in connection with audio analyzer 5120 and FIG. 5B. In some cases, the descriptor 5103 may indicate a past date, or may be blank, such as when the user is accessing historical data, modes or features of the respiratory infection monitoring app 5101, user notifications, etc. You can.

Share icon 5104 may be selected to share various data, analyzes or diagnostics, reports, user-provided annotations, or observations (eg, notes) via electronic communication. For example, the share icon 5104 may facilitate a user emailing, uploading, or sending a report consisting of the latest phonemic feature data, respiratory disease changes, inferences or predictions, or other data to a caregiver. In some embodiments, the share icon 5104 indicates the sharing on social media or other similar users of various data aspects captured, determined, displayed, or accessed via the respiratory infection monitoring app 5101. It can be easily shared with other users. In one embodiment, the share icon 5104 provides information about respiratory disease by sharing the user's respiratory disease data and, in some cases, related data (e.g., location, historical data, or other information) with a government agency or health department. It can facilitate monitoring of infectious disease outbreaks. This shared information may be de-identified and encrypted prior to communication to protect user privacy.

Selection of the stethoscope icon 5106 may provide the user with various communication or contact options with the user's health care provider. For example, selection of the stethoscope icon 5106 may result in a remote appointment (or request for an appointment), a user's medical record (e.g., Profile/Health Data (EHR) 241 of FIG. ) or facilitate access to a healthcare provider's online portal for additional services. In some embodiments, selection of the stethoscope icon 5106 may initiate a feature that communicates certain data to the user's health care provider, such as what the user is currently viewing; A user's health care provider may be requested to view the user's data. Finally, selection of the cycle icon 5108 causes the view and/or data displayed through the respiratory infection monitoring app 5101 to be refreshed or updated so that the view is current with respect to available data. You can. In some embodiments, selection of cycle icon 5108 causes a sensor (or a computer application associated with collecting data from a sensor, such as sensor 103 of FIG. 1) and/or a cloud data store ( For example, data retrieved from an online data account may be refreshed.

The example GUI 5100 also includes various user-selectable icons 5111, 5112, 5113, 5114, and 5115 corresponding to additional various features provided by the example embodiment of the respiratory infection monitoring app 5101. It may also include an icon menu 5110 that includes. In particular, selecting these icons can navigate the user to various services or tools provided via the respiratory infection monitoring app 5101. As a non-limiting example, selection of the home icon 5111 may cause one of the exemplary GUIs described in connection with FIGS. 5A-5E, one provided by the respiratory infection monitoring app 5101, or The user can be navigated to a home screen that can include a welcome screen (such as GUI 5510 of FIG. 5E) that can include multiple commonly used services or tools, the user's account information, or any other views (not shown). .

In some embodiments, an audio analyzer 5120 or the like includes application functionality that facilitates obtaining audio samples from a user by selecting an "audio recording" icon 5112 shown as selected in the example GUI 5100. can navigate the user to the audio data acquisition mode. Embodiments of audio analyzer 5120 may be used with one or more components of system 200, including user audio monitor 260 (or one or more of its subcomponents), as described in FIG. In some cases, it may be performed by user interaction manager 280 (or one or more of its subcomponents), also as shown in FIG. For example, the functionality of audio analyzer 5120 to obtain user audio sample data may be performed as described in conjunction with audio sample collector 2604.

In some embodiments, the audio analyzer 5120 may provide instructions to guide the user through the audio data collection process as shown in GUI element 5105 of FIG. 5A and described elsewhere in connection with FIG. 5B. . In particular, GUI element 5105 illustrates a repeat sound exercise that prompts the user to repeat the sound for a set duration. Here, for example, the user is required to make a "woom" sound for 5 seconds. In some embodiments, instructions provided by speech analyzer 5120 are determined or generated in accordance with one or more of the subcomponents, such as user interaction manager 280 or user instruction generator 282. It may be.

Descriptor 5103 indicates the current date, which will be associated with the collected audio sample. A timer (GUI element 5122) may be provided to facilitate instructing the user when to start or end recording audio samples. A visual audio sample recording indicator (GUI element 5123) may also be displayed to provide feedback to the user regarding the audio sample recording. In one embodiment, the operations of GUI elements 5122 and 5123 may be performed by user input response generator 286, described in connection with FIG. 2. Other visual indicators (not shown) include background noise level, microphone level, volume, progress indicators, or other indicators described in connection with user input response generator 286, including but not limited to: Not limited.

In some embodiments (not shown), audio analyzer 5120 may display a user's progress in acquiring audio-related data within a certain time period (eg, a day or half a day). For example, if voice-related data is obtained through daily interactions or reading aloud a text, the voice analyzer 5120 may include indicators of the user's progress, such as percentage complete, dials, or sliding progress bars, or the user's It may also indicate an index of phonemes that have been successfully acquired for utterances or phonemes that have not yet been acquired. Another GUI and details of an exemplary audio data collection process performed by audio analyzer 5120 are described in connection with FIG. 5B.

Referring again to FIG. 5A, continuing with GUI 5100 and icon menu 5110, selection of view icon 5113 can navigate the user to GUIs and features that provide the user with tools and information regarding the user's respiratory disease. This may include, for example, information regarding the user's current respiratory illness, trends, forecasts, or recommendations. Additional details of the functionality associated with view icon 5113 are described in connection with FIG. 5C. Additionally, selection of log icon 5114 (FIG. 5A) can navigate the user to logging tools that include features to facilitate tracking or monitoring of respiratory diseases, such as those described in connection with FIGS. 5D and 5E. be. In one embodiment, the functionality associated with the logging tool or log icon 5114 may include a GUI and tools or services that receive and display a user's physiological data, symptom data, or other contextual information. . For example, one embodiment of a logging tool includes a self-reporting tool that records user symptoms such as those described in connection with FIGS. 5D and 5E.

In some embodiments, selection of the settings icon 5115 configures various user preferences, settings, or settings of the respiratory infection monitoring app 5101, aspects of audio-related data (e.g., sensitivity thresholds, phoneme feature comparison settings, phoneme or other settings related to the acquisition or analysis of voice-related data), user account, information about the user's caregivers, caregivers, insurance, diagnosis or condition, user care/treatment, or specifying other settings. The user can be navigated to a user settings mode that may be enabled. In some embodiments, at least some of the settings may be configured by the user's healthcare provider or clinician. Some of the settings accessible via settings icon 5115 may include the settings described in connection with settings 249 in FIG.

5B, example GUIs 5210, 5220, 5230 illustrating aspects of an example process for the acquisition of speech-related data in which a user is guided to provide audio samples of various utterances; A sequence 5200 of and 5240 is provided. The process depicted in the GUI of sequence 5200 may be provided by a respiratory infection monitoring app 5101 running on user computer equipment 5102a and capable of displaying GUIs 5210, 5220, 5230, and 5240. In one embodiment, the functionality shown in GUIs 5210, 5220, 5230, and 5240 is provided by an audio data acquisition mode of respiratory infection monitoring app 5101, such as audio analyzer 5120 shown in FIG. 5112 (FIG. 5A). The instructions shown in GUIs 5210, 5220, 5230, and 5240 to guide the user (e.g., instructions 5213) may be provided by one or more of the subcomponents, such as user interaction manager 280 or user instruction generator 282. may be determined or generated according to the

As shown in GUI 5210, instructions 5213 are shown to guide the user in producing continuous sounds as part of a repeated sound exercise. A repetitive sound exercise may include one or more vocal tasks performed by a user. In this example, a user may initiate an exercise (or task within an exercise) by selecting a start button 5215. GUI 5210 also shows a progress indicator 5214, which is a sliding bar that indicates the user's progress toward providing audio sample data for this session or time segment (eg, 60% complete).

GUIs 5220, 5230, and 5240 continue to illustrate how to guide the user in producing continuous sounds as part of a repeated sound exercise. As shown in sequence 5200, example GUIs 5220, 5230, and 5240 include various visual indicators that facilitate guiding the user or providing feedback to the user. For example, GUI 5220 includes GUI element 5222 that shows a countdown timer and background noise confirmation indicator. A countdown timer in GUI element 5222 indicates the amount of time until the user begins speaking. GUI 5230 includes another example of a timer, in this case a GUI element 5232 that indicates the duration that the user continues to utter the "aah" sound. Similarly, GUI 5240 includes an example timer, in this case a GUI element 5242 that indicates that the user has uttered a "woom" sound for 5 seconds. GUI 5240 also includes a GUI element 5243 that provides feedback to the user regarding the audio sample recording of the "ooh" sound. As discussed above, functionality associated with visual indicators, such as the progress indicator 5214, the countdown timer and background indicator in GUI element 5222, the timer in GUI elements 5232 and 5242, or the audio sample recording indicator in GUI element 5243, may be triggered by user input. may be provided by response generator 286. Another example of visual indicators and user feedback actions that can be provided is described in connection with user input response generator 286.

Continuing with sequence 5200, GUI 5240 may represent the final stage of a repeated sound exercise to obtain audio sample data or the last of one of multiple stages of a process to obtain audio sample data. For example, there may be additional vocal tasks or exercises that are performed later. The user who provided the audio sample may end the exercise (or task within the exercise) by selecting a finish button 5245. Alternatively, if the user wishes to redo the task and provide another audio sample, the user may select GUI element 5244 to begin the task again. In some embodiments, as described in connection with the sample recording examiner 2608 and the user input response generator 286, an indicator or instruction to retry the task may be provided, such as if the audio sample is determined to be defective. It may be given to the user.

An example process illustrated in sequence 5200 for collecting audio-related data includes prompting the user for instructions as part of a repeated sound exercise. However, other embodiments of the respiratory infection monitoring app 5101 may obtain audio-related data from everyday interactions, as described herein. Additionally, in some embodiments, voice-related data may be collected from a combination of daily interactions and repeated sound exercises, such as the example of FIG. 5B. For example, if routine interactions do not result in sufficient usable voice-related data over a given time interval (e.g., a day or half a day) or a particular type of usable voice-related data, repeat sound The user may be notified (eg, via the respiratory infection monitoring app 5101) of providing additional voice-related data by or similar interaction. In some embodiments, the user may configure options for how audio-related data can be obtained via settings icon 5115 or as described in connection with settings 249 of FIG. .

Referring now to FIG. 5C, another aspect of a respiratory infection monitoring app 5101 that includes a GUI 5300 is shown. GUI 5300 includes various user interface (UI) elements that display a user's respiratory disease perspective (e.g., perspective 5301), and the functionality shown in GUI 5300 is accessed or initiated by selecting a perspective icon 5113 in GUI 5100. (FIG. 5A). The example GUI 5300 includes a descriptor 5303 that indicates the current date on which the user is accessing the views feature of the respiratory infection monitoring app 5101 (e.g., today is May 4th) and a and a user view 5301 indicating that the user is in a view operating mode (or accessing view functionality) of the app 5101. As shown in FIG. 5C, the icon menu 5110 indicates that the view icon 5113 has been selected, allowing the user to be presented with a GUI 5300 showing the user's view 5301. The opinion 5301 may include the user's respiratory disease determination and/or prediction and related information. For example, the view 5301 may include a respiratory disease score 5312, a transmission risk 5314 that may include associated recommendations 5315, and trend information such as a trend descriptor 5316 and GUI element 5318.

As described herein, the respiratory illness score 5312 may quantify or characterize the user's respiratory illness, including the user's current respiratory illness, a change in the user's respiratory illness, or the user's respiratory illness. can represent the possibility of future respiratory disease. As described elsewhere herein, the respiratory disease score 5312 is based on voice-related data obtained through the example process shown in FIG. 5B or described in connection with the user voice monitor 260 of FIG. It may be based on the user's voice-related data such as. In some cases, the respiratory illness score 5312 further includes user observations (e.g., self-reported symptom scores), health or physiological data (e.g., data provided by wearable sensors or the user's health record), weather, location, region. It may be based on contextual information, such as infection information (eg, current infection rate in the user's geographic location), or other context. Additional details for determining respiratory disease score 5321 are provided in connection with respiratory disease inference engine 278 of FIG. 2 and method 6200 of FIG. 6B.

Transmission risk 5314 of GUI 5300 may indicate the risk of the user transmitting the detected respiratory-related pathogen. Transmission risk 5314 may be determined as described with respect to respiratory disease inference engine 278 and user disease inference logic 237 of FIG. Transmission risk may be a quantitative or categorical indicator, such as "medium-high" indicating medium to high risk in the example GUI 5300. In conjunction with Transmission Risk 5314, Opinion 5301 includes recommended practices to reduce transmission risk, such as wearing masks, social distancing, self-quarantining, or consulting a health care provider. Recommendations 5315 to obtain may be provided.

These recommendations 5315 may include predetermined recommendations, which in some embodiments are determined based on the particular detected respiratory disease and/or transmission risk 5314 according to a set of rules. It may be possible to do so. In some embodiments, recommendations 5315 may be tailored to the user based on the user's historical information, such as historical audio-related information, and/or contextual information, such as geographic location. Additional details of determining recommendations 5315 are described with respect to respiratory disease inference engine 278 and user disease inference logic 237 of FIG.

Views 5301 may provide trend information such as trend descriptors 5316 and, in some embodiments, GUI elements 5318 that visualize trends or changes in a user's respiratory disease over time. Trend descriptor 5316 may indicate previously detected or currently detected changes in the user's respiratory disease. Here, trend descriptor 5316 indicates that the user's respiratory disease is worsening. Additionally, GUI element 5318 may include a graph or chart of the user's data or other visual indicators of changes in the user's respiratory illness, such as changes in phoneme characteristics detected from the audio sample over the past 14 days. Good too. In other embodiments, the view 5301 additionally or alternatively provides a prediction of the user's likely future respiratory disease trends. For example, in some embodiments, GUI element 5318 may indicate a future date and predict future changes in the user's respiratory illness, as described with respect to respiratory illness inference engine 278. . In one embodiment, the opinion 5301 provides a prediction indicating when the user is likely to recover from a respiratory infection (eg, "will be normal within 3 days"). Another example forecast that may be provided by opinion 5301 is an early warning forecast, such as a forecast that indicates that a user may be expected to become ill at some time in the future upon initial detection of a possible respiratory infection. (e.g., ``You appear to have a respiratory infection and may feel unwell by the end of the week.'')

In some cases, the respiratory infection monitoring app 5101 may generate or provide electronic notifications to the user (or caregiver or clinician) regarding the forecast or other information provided by the opinion 5301. The information provided by view 5301 (which may include trend or forecast information utilized in generating trend descriptor 5316 and/or GUI element 5318) may be used in respiratory disease tracker 270 of FIG. 2 or any of its subcomponents. may be determined by one or more embodiments (such as respiratory disease inference engine 278). Additional details for determining respiratory disease information, transmission risk 5314, recommendations 5315, forecasts, or trend information 5316 are described in connection with respiratory disease tracker 270 of FIG.

Referring now to FIG. 5D, another aspect of a respiratory infection monitoring app 5101 that includes a GUI 5400 is shown. The GUI 5400 includes UI elements that display or accept respiratory disease-related information (respiratory symptoms, etc.), and corresponds to a log function indicated by a log icon 5114. In particular, GUI 5400 illustrates an example of a logging tool 5401 for recording, displaying, and, in some aspects, annotating current or historical user data. Log tool 5401 may be accessed by selecting log icon 5114 in icon menu 5110. In some embodiments, the logging tool 5401 (or the self-reporting tool 5415 described below) may be presented to the user due to a determination that the user has or may have a respiratory infection. (or the user may receive a notification that the log tool 5401 is accessed). The example GUI 5400 further includes a descriptor 5403 indicating that the information displayed by the log tool 5401 is for the date of Monday, May 4th. In some embodiments of log tool 5401, for example, by selecting date arrow 5403a or by selecting a particular calendar date from a calendar view (not shown) after selecting history tab 5440, Users can navigate to past dates and access historical data.

As shown in the example GUI 5400 of this respiratory infection monitoring app 5101, the logging tool 5401 has five selectable tabs: Add Symptom 5410, Notes 5420, Report 5430, History 5440, and Action 5450. include. These tabs may correspond to additional functionality provided by logging tool 5401. For example, as shown in GUI 5400, when the Add Symptoms 5410 tab is selected, various UI components are presented to the user to enable them to self-report symptoms that may be associated with their respective respiratory illnesses. In particular, the functionality corresponding to adding symptoms 5410 includes a self-reporting tool 5415 that includes a symptom list and user-selectable sliders to accept user input regarding the severity the user feels about each symptom. For example, self-report tool 5415 shown in GUI 5400 indicates that the user is experiencing moderate shortness of breath and nasal congestion and severe cough. In some embodiments, a user may utilize a self-reporting tool 5415 to enter this symptom data daily or multiple times per day (eg, every morning and night, etc.). In some cases, symptom data may be input at or near the time period during which voice-related data is collected from the user.

In some embodiments, the add symptom 5410 (or logging tool 5401) also provides a selectable option 5412 for the user to enter data from another computing device, such as a wearable smart device or similar sensor. May contain. For example, a user may select to input data from a fitness tracker so that it can be received by logging tool 5401. In some embodiments, data may be received directly and/or automatically from a smart device or a database (eg, an online account) associated with the device. In some cases, the user may be required to link or associate a device with the respective respiratory infection monitoring app 5101 (or a user account associated with the respiratory infection monitoring app 5101) to enter data. be. In some embodiments, the user may configure various parameters for inputting data from another device in the application settings (e.g., by selecting the settings icon 5115 as shown in FIG. 5A). Good too. For example, when data is entered, the user may specify the data to be entered (e.g., the user's sleep data captured by a smartwatch), or may specify permission settings, account linking, or other You may also make settings.

As a non-limiting example, entering such data and utilizing selectable options 5412 may be utilized in conjunction with self-reporting tool 5415 or They may also be used without being combined. For example, data imported from a linked smart device may provide an initial severity assessment of symptoms based on information entered by the user into the linked smart device, but the user may utilize self-reporting tools 5415 to , these initial evaluations may be adjusted. Add symptom 5410 may also include another selectable option 5418 indicating that the symptom has not changed since the user last recorded the symptom (such as the previous day). The functionality and UI elements associated with adding symptoms 5410 of GUI 5400 are one embodiment of user interaction manager 280 or one or more subcomponents thereof (such as self-reporting tool 284) described in conjunction with FIG. It may be generated by using .

In conjunction with the GUI 5400 shown in FIG. 5D, the memo 5420 tab is used by a respiratory infection monitoring app 5101 that accepts or displays observation data from the user or caregiver on the particular day (in this case, May 4). The functionality (or, more specifically, the logging functionality associated with logging tool 5401) is navigable to the user. Examples of observational data include notes 5420 documenting the user's respiratory illness (symptoms, etc.) or notes 5420 related to the user's respiratory illness. In some embodiments, the memo 5420 includes a UI that accepts text (or audio or video recordings) from the user. In some aspects, the UI functionality of the note 5420 includes a human body configured to accept input from a user indicating an area of the user's body that is affected by a potential or known respiratory disease, symptom, or side effect. It may also include GUI elements shown. In some embodiments, a user may enter contextual information such as, for example, the user's geographic location, weather, and any physical activity performed that day.

The Reports 5430 tabs can navigate the user to a GUI for viewing and generating various reports of respiratory disease related data detected by embodiments described herein. For example, the report 5430 may include historical or trend information regarding the user's respiratory illness or the prediction of the user's respiratory illness. In another example, report 5430 may include a report of respiratory related information for a larger population. For example, report 5430 may indicate many other users of respiratory infection monitoring app 5101 where the same or similar respiratory illness was detected. In some embodiments, the functionality provided by report 5430 includes respiratory disease-related data communicated to a caregiver or clinician (e.g., via share icon 5104 or stethoscope icon 5106 in FIG. 5A) or Operations may include formatting or creating respiratory disease-related data to be shared with a caregiver or clinician.

The history 5440 tab allows the user to navigate to a GUI that displays the user's historical data related to respiratory disease monitoring. For example, selection of history 5440 can display a GUI with a calendar view. A calendar view may facilitate access or display of a user's respiratory disease-related data detected and interpreted on different dates. For example, by selecting a particular past date in a displayed calendar, the user may be presented with a summary of data for that date. In some embodiments of the calendar view GUI that is displayed when the history 5440 tab is selected, each date in the calendar displays an indicator or information that indicates detected or predicted respiratory disease information associated with that date. It may be possible to do so.

Selection of the tab showing treatment 5450 on GUI 5400 directs the user to a GUI within respiratory infection monitoring app 5101 that includes the ability for the user to specify details such as whether the user received treatment and/or had side effects on that date. It is possible to navigate. For example, a user may specify whether they took a prescribed antibiotic or received a respiratory procedure on a particular date. Additionally, in some embodiments, a smart pillbox or smart container, which may include so-called Internet of Things (IoT) functionality, automatically detects when a user has accessed medication stored within the container, and when the user It is also conceivable that an indicator indicating that the patient received treatment on the date may be transmitted to the respiratory infection monitoring application 5101. In some embodiments, the Treatments 5450 tab may include, for example, treatments that the user (or the user's caregiver or clinician) underwent on that date (e.g., taking a prescription drug, taking an over-the-counter drug, taking a large amount of The user may include a UI that allows the user to specify each treatment by selecting a check box indicating the type of treatment (intake of fluids, rest, etc.).

Referring to FIG. 5E, a sequence 5500 of an example GUI 5510, 5520, and 5530 is provided that illustrates aspects of an example process for user-initiated symptom reporting. GUIs 5510, 5520, and 5530 may be generated according to one embodiment of self-reporting tool 284 described in connection with FIG. 2. In some cases, when a user launches respiratory infection monitoring app 5101 on user computer equipment 5102a, GUI 5510 may be provided as a welcome/login screen. As described herein, the respiratory infection monitoring app 5101 may be associated with a particular user and specified by a user account. As illustrated, the GUI 5510 includes user credentials (i.e., email address and password, etc.) that identify the user so that user-specific information can be accessed and user input can be properly stored in association with the user. includes a UI element for the user to input the user identifier (user identifier). When a user logs in via GUI 5510, a GUI 5520 may be provided that includes initial instructions prompting the user to report symptoms. GUI 5520 may include a selectable "Symptom Report" button that may cause GUI 5530 to be presented that includes UI elements that facilitate input of user symptom information. In an exemplary embodiment of GUI 5530, a user may rate the severity of each symptom displayed within GUI 5530 by moving a slider to the appropriate severity level. Additional details of user input of symptom information are described with respect to GUI 5400 of FIG. 5D.

6A and 6B are flow diagrams of exemplary methods utilized to monitor respiratory disease in a user. For example, FIG. 6A is a flow diagram illustrating an example method 6100 for obtaining phoneme characteristics, according to one embodiment of the present disclosure. FIG. 6B is a flow diagram illustrating an example method 6200 for monitoring respiratory disease in a user based on phoneme characteristics, according to an embodiment of the present disclosure. Each block or step of methods 6100 and 6200 includes a computer process that may be performed using any combination of hardware, firmware, and/or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Additionally, these methods may be embodied as computer-usable instructions stored on a computer storage medium. These methods may be provided by a standalone application, service or hosting service (standalone or in combination with another hosting service), or a plug-in to another product, to name a few. Accordingly, methods 6100 and 6200 may be adapted to be performed by one or more computing devices, such as user equipment such as a smartphone, a server, or a distributed computing platform, such as in a cloud environment. Exemplary aspects of computer program routines covering implementations of phoneme feature extraction are illustrated in FIGS. 15A-15M.

Referring to the method 6100 of FIG. 6A, the method 6100 includes detecting phoneme characteristics according to one embodiment of the present disclosure, and an embodiment of the method 6100 includes the step of detecting a phoneme characteristic of the user speech monitor 260 described in connection with FIG. etc., may be configured to be performed by embodiments of one or more components of system 200. At step 6110, audio data is received. In some embodiments, step 6110 is performed by one embodiment of the audio sample collector 2604 described in connection with FIG. 2. Another embodiment of step 6110 is described in connection with audio sample collector 2604 and user audio monitor 260.

The audio data received in step 6110 may include records (eg, audio samples, speech samples) of individual phonemes or combinations of phonemes, such as planned or unplanned utterances, uttered by the user. Thus, the audio data includes audio information regarding the user. Audio data is collected during a user's daily interactions or daily interactions with user equipment, such as user equipment 102a-102n of FIG. 1, having a sensor such as a microphone (such as one embodiment of sensor 103 of FIG. 1). It may be possible to do so.

Some embodiments of method 6100 include operations performed prior to receiving the audio data in step 6110. For example, determining and/or modifying the acoustic parameters (signal strength, directivity, sensitivity, frequency, and signal-to-noise ratio (SNR), etc.) of a sensor (e.g., microphone) to obtain usable audio data. An operation may be performed to determine proper or optimized settings. These operations may be associated with recording optimizer 2602 of FIG. Similarly, these operations may include identifying and, in some aspects, removing or reducing background noise as described in connection with background noise analyzer 2603 of FIG. These steps include comparing the noise intensity level to a maximum threshold, checking for speech within a predetermined frequency, and checking for intermittent spikes or similar acoustic artifacts. Good too.

In some embodiments, user instructions may be provided to facilitate receiving audio data. For example, a user may be guided to providing audio data by following speech-related tasks. The user instructions may also include feedback based on the most recently provided sample, such as instructing the user to speak louder or maintain vocal phonemes for longer durations. User interaction to facilitate the reception of audio data is generally performed by an embodiment of user interaction manager 280 or a subcomponent thereof, user instruction generator 282, described in connection with FIG. It may be .

In step 6120, a date and time value corresponding to the time segment is determined. The date and time value may be the time the audio data was received or recorded from the user's utterance. In some embodiments, step 6120 is performed by one embodiment of the audio sample collector 2604 described in connection with FIG. 2.

In step 6130, phonemes are determined by processing at least a portion of the audio data. Some embodiments of step 6130 may be adapted to be performed by one embodiment of phoneme segmenter 2610 described in connection with FIG. 2. Determining a phoneme from a portion of the audio data includes detecting the phoneme by performing automatic speech recognition (ASR) on the portion of the audio data, and associating the detected phoneme with the portion of the audio data; May contain. ASR may determine text (eg, words) from a portion of the audio data, and phonemes may be determined based on the recognized text. Alternatively, determining the phoneme may include receiving an indication of a phoneme corresponding to the portion of the audio data and associating the phoneme with the portion of the audio data. This process may be particularly useful when the audio data is of sustained phoneme utterances based on a speech-related task given to the user. For example, users are now instructed to say "aah" for 5 seconds, "yee" for 5 seconds, "mm" for 5 seconds, and finally "umm" for 5 seconds. These instructions may be based on the expected phoneme order for the audio data (i.e.

) can be shown.
Processing audio data to determine phonemes may include detecting and isolating particular phonemes. In one embodiment,

The phoneme corresponding to is detected. In another embodiment,

only detected. Alternatively, processing the audio data may include detecting the presented phonemes and separating all detected phonemes. Phonemes may be detected by applying an intensity threshold to separate background noise from the user's speech, as described elsewhere in conjunction with phoneme segmenter 2610 of FIG. 2.

Some aspects of processing the audio data in step 6130 may include additional processing steps that may be performed by one embodiment of signal creation processor 2606 of FIG. For example, frequency filtering, such as high-pass or band-pass filtering, may be applied to remove or attenuate frequencies of the audio data representing background noise. In one embodiment, a bandpass filter of, for example, 1.5 to 6.4 kilohertz (kHz) is applied. Step 6130 may also include performing audio normalization to achieve a targeted signal amplitude level, applying a bandpass filter and/or amplifier to improve SNR, or other signal conditioning or preprocessing. good.

In step 6140, a set of phoneme characteristics is determined based on the determined phonemes. Some embodiments of step 6140 are performed by the embodiment of acoustic feature extractor 2614 described in conjunction with FIG. The phoneme feature set includes at least one acoustic feature that characterizes the processed portion of the audio data. The feature set may include measures of power and power variability, pitch and pitch variability, spectral structure, and/or formants, as described elsewhere in connection with acoustic feature extractor 2614. In some embodiments, different sets of characteristics (i.e., different combinations of acoustic characteristics) are determined for different phonemes detected in the audio data. For example, in one exemplary embodiment, the phoneme

Twelve characteristics were determined for the phoneme

Twelve characteristics were determined for the phoneme

Eight characteristics are determined for . was detected

The phoneme property set includes the standard deviation of formant 1 (Fl) bandwidth, pitch interquartile range, spectral entropy, jitter, and mel determined for frequencies from 1.6 to 3.2 kilohertz (kHz). It may include the standard deviation of the frequency cepstral coefficients MFCC9 and MFCC12, the average of the mel frequency cepstral coefficient MFCC6, and the spectral contrast determined for frequencies from 3.2 to 6.4 kHz. was detected

The phoneme property set includes harmonicity, standard deviation of F1 bandwidth, interquartile range of pitch, spectral entropy determined for frequencies 1.5-2.5 kHz and 1.6-3.2 kHz, 1 Spectral flatness determined for frequencies from .5 to 2.5 kHz, standard deviation of Mel frequency cepstral coefficients MFCC1, MFCC2, MFCC3, and MFCC11, mean of Mel frequency cepstral coefficient MFCC8, and 1.6 to 3.2 kHz. may include a spectral contrast determined for the frequency of . was detected

Hz and spectral entropy determined for frequencies from 1.6 to 3.2 kHz, spectral flatness determined for frequencies from 1.5 to 2.5 kHz, standard deviation of mel frequency cepstral coefficients MFCC2 and MFCC10, It may include the average of the mel frequency cepstral coefficients MFCC8, the shimmer, the spectral contrast determined for frequencies from 3.2 to 6.4 kHz, and the standard deviation of the 200 hertz (Hz) 1/3 octave band. Also, in some embodiments, the values of one or more properties in the property set may be transformed. In one exemplary embodiment, a log transform is applied to the interquartile range of pitch, standard deviation of MFCC, spectral contrast, jitter, and standard deviation within the 200 Hz 1/3 octave band.

In step 6155, it is determined whether the audio data is to be additionally processed. In some embodiments, step 6155 is performed by one embodiment of user audio monitor 260. As mentioned above, the received audio data may have multiple phonemes, as it may be a recording of multiple sustained phonemes or utterances (planned or unplanned). In this way, different parts of the audio data may be processed for the detection of different phonemes. For example, a first part is processed to determine a first phoneme, a second part is processed to determine a second phoneme, and a third part is processed to detect a third phoneme. The first, second, and third phonemes may each be processed such that the first, second, and third phonemes are

It may correspond to In some aspects, the fourth portion is processed to detect a fourth phoneme, and the fourth phoneme is

It may be. These phonemes may be recorded in one recording by a user who utters these three phonemes. Thus, the additional audio data of step 6155 may include additional portions of the same partially processed audio sample. Additionally or alternatively, step 6155 includes determining whether to additionally process the audio data from additional audio samples recorded in the same session (i.e., acquired in the same time frame). You can stay there. For example, three phonemes may be recorded separately from the same session.

If the audio data is additionally processed in step 6155, steps 6130 and 6140 may be performed on the additional portion of the audio data. FIG. 6A shows that step 6155 occurs after the first portion of the audio data has been processed and a feature set has been determined for the detected phonemes. However, embodiments of method 6100 may include determining whether to additionally process the audio data for detection of additional phonemes in step 6155 before any feature sets are extracted. .

Without further processing the audio data or determining a feature set, the method 6100 proceeds to step 6160 where the phoneme feature set extracted from the audio data is stored in a record associated with the user. . The stored phoneme characteristic set includes an index of date and time value. In some embodiments, step 6160 is performed by one embodiment of user audio monitor 260 and, more particularly, acoustic feature extractor 2614. The phoneme feature set may be stored in a user's patient record, such as patient record 240. More specifically, the phoneme feature set may be stored as a vector, or may be stored as the phoneme feature vector 244 in FIG. 2.

Some embodiments of method 6100 include additional acts of monitoring the user's respiratory illness over time and, in some aspects, detecting changes in the user's respiratory illness. For example, steps 6110-6160 are performed for a first audio data sample recording of a first time segment and repeated for a second audio data sample recording of a second subsequent time segment. You can. For this reason, a first set of phoneme characteristics may be determined and stored for the first time segment, and a second set of phoneme characteristics may be determined and stored for the second time segment. . The method 6100 may then include an act of monitoring the user's respiratory illness over time using the first and second sets of phoneme characteristics. For example, the first and second phoneme feature sets may be compared to detect a change. This comparison operation may be performed by one embodiment of the phoneme feature comparator 274 and includes a feature distance measurement (e.g., Euclidean distance) between the feature set vectors of the first and second time intervals. may include determining. Based on the characteristic distance measurement result (e.g., the magnitude and/or whether the measurement result is positive or negative), it is determined whether the user's respiratory disease has changed between the second and first time intervals. It may be .

In some embodiments, method 6100 includes receiving context information associated with a time segment (e.g., a first time segment and/or a second time segment) and determining for the associated time segment. storing context information in a record associated with the set of characteristics identified. These operations may be performed by one embodiment of the context information determiner 2616 of FIG. Contextual information may include the user's physiological data, which may be self-reported, received from one or more physiological sensors, and/or the user's electronic health record (e.g., profile/ The determination may be made from health data (EHR) 241). Additionally or alternatively, the context information may include location information of the user in the relevant time segment or other context information associated with the first time segment. Embodiments of step 6140 may include determining a further determined set of phoneme characteristics based on the context data of the associated time segment.

Referring to FIG. 6B, a method 6200 includes monitoring a user's respiratory illness based on phoneme characteristics, according to an embodiment of the present disclosure. Method 6200 may be adapted to be performed by an embodiment of one or more components of system 200, such as respiratory disease tracker 270 described in connection with FIG. Step 6210 includes receiving phoneme feature vectors (sometimes referred to as phoneme feature sets) representing the user's speech information at different times. Thus, a first phoneme feature vector (i.e., a first set of phoneme features) is associated with a first date and time value, and a second phoneme feature vector (i.e., a second set of phoneme features) is associated with a first date and time value. associated with a second datetime value that occurs after the value. For example, the first phoneme feature vector may correspond to audio data captured in a first interval (corresponding to a first date/time value) that is within approximately 24 hours (e.g., 18 to 36 hours) of the capture of the audio data. and is used to determine the second phoneme characteristic vector in the second division (corresponding to the second date and time value). The time between the first and second datetime values can be shorter (e.g., 8-12 hours) or longer (e.g., 3 days, 5 days, 1 week, 2 weeks). it is conceivable that. Step 6210 may generally be performed by respiratory disease tracker 270 and, more specifically, by feature vector time series assembler 272 or phoneme feature comparator 274.

Determining the first and second phoneme feature vectors may be performed according to one embodiment of method 6100 of FIG. 6A. In some embodiments, determining the first and/or second sets of phoneme characteristics includes processing audio information including phonetic information to determine the first and/or second sets of phonemes, and determining each set of phonemes in the set. This may be done by extracting, for a phoneme, a set of characteristics that characterize the phoneme. In some embodiments, the first and second feature sets are phonemes.

Contains acoustic characteristic values characterizing the In an exemplary embodiment, the first and second feature vectors each represent a phoneme.

8 phonemes for

12 phonemes for

Contains 12 characteristics for . phoneme

The characteristics are the standard deviation of formant 1 (Fl) bandwidth, pitch interquartile range, spectral entropy, jitter, and mel frequency cepstrum determined for frequencies from 1.6 to 3.2 kilohertz (kHz). It may include the standard deviation of the coefficients MFCC9 and MFCC12, the average of the Mel frequency cepstral coefficient MFCC6, and the spectral contrast determined for frequencies from 3.2 to 6.4 kHz. phoneme

The properties of are harmonicity, standard deviation of F1 bandwidth, interquartile range of pitch, spectral entropy determined for frequencies 1.5-2.5 kHz and 1.6-3.2 kHz, 1.5 Spectral flatness determined for frequencies ~2.5 kHz, standard deviations of Mel frequency cepstral coefficients MFCC1, MFCC2, MFCC3, and MFCC11, mean of Mel frequency cepstral coefficients MFCC8, and frequencies from 1.6 to 3.2 kHz may include a spectral contrast determined for. phoneme

The properties of are harmonicity, standard deviation of F1 bandwidth, interquartile range of pitch, spectral entropy determined for frequencies 1.5-2.5 kHz and 1.6-3.2 kHz, 1.5 Spectral flatness determined for frequencies ~2.5 kHz, standard deviation of Mel frequency cepstral coefficients MFCC2 and MFCC10, mean of Mel frequency cepstral coefficients MFCC8, shimmer, determined for frequencies 3.2 to 6.4 kHz and the standard deviation of the 200 hertz (Hz) 1/3 octave band. In some embodiments,

One or more of these properties are extracted to characterize the phoneme.
In some embodiments, the first phoneme feature vector determined for the first time interval is based on a plurality of phoneme feature sets from the plurality of audio samples captured before the second date and time value. . The first feature vector may represent a combination (such as an average) of multiple phoneme feature vectors. These multiple audio samples are used when it is known or estimated that the patient is healthy (i.e., not suffering from a respiratory infection), such that the first characteristic vector may represent a health criterion. It may be one that has been acquired. Alternatively, the audio samples used to determine the first phoneme feature vector are obtained when it is known or estimated that the patient is ill (i.e., has a respiratory infection). The first phoneme feature vector may represent a disease criterion.

Step 6220 includes performing a comparison of the first and second phoneme feature vectors to determine a phoneme feature set distance. In some embodiments, step 6220 may be performed by one embodiment of phoneme feature comparator 274 of FIG. In some embodiments, this comparison includes determining a Euclidean distance between the first and second sets of phoneme features. Each characteristic represented by a characteristic vector may be compared with a corresponding characteristic in the other characteristic vector. For example, the first characteristic of the first phoneme characteristic vector (e.g., the phoneme

jitter) of the second phoneme feature vector (e.g., the jitter of the phoneme

jitter).
In step 6230, it is determined that the user's respiratory disease has changed based on the phoneme feature set distance between the first and second phoneme feature vectors. In some embodiments, step 6230 is performed by one embodiment of the respiratory disease inference engine 278 described in connection with FIG. 2. Determining that a user's respiratory illness has changed may be based on a prior decision by a caregiver or clinician or the user's physiological data (e.g., self-reported), user settings, or historical information about the user's respiratory illness. It may be determined that the phoneme characteristic set distance satisfies a determinable threshold distance (for example, a disease change threshold). Alternatively, the disease change threshold may be preset based on a reference population of monitoring patients.

In some embodiments, determining that the user's respiratory illness has changed may include determining whether the user's respiratory illness is improving, worsening, or unchanged (e.g., improving or worsening). It may also include a determination as to whether the This includes disease change criteria for the determined phoneme feature set distance (which may be a general standard determined from information about the reference population or determined for each user based on past user data). may also include comparisons with For example, a third phoneme feature vector representative of a health criterion may be determined from audio data captured at a time when the user is determined not to have a respiratory infection, and a second (i.e. A second phoneme feature set distance is determined by performing a second comparison between the most recent) and third (i.e., reference) phoneme feature vectors. A third phoneme feature set distance may also be determined by performing a third comparison between the first (i.e., early) and third (i.e., reference) phoneme feature vectors. . The third phoneme feature set distance (representing the change between the health criterion and the first phoneme feature vector) is the distance between the health criterion and the second phoneme feature vector (representing the change between the health criterion and the second phoneme feature vector according to data captured after the first phoneme feature vector). is compared with the second phoneme feature set distance (representing the change between . If the second phoneme feature set distance is shorter than the third feature set distance (the vector from the most recently acquired data is close to the health standard), it is determined that the user's respiratory disease is improving. can be done. If the second phoneme feature set distance is longer than the third feature set distance (the vector from the most recently acquired data is far from the health standard), it is determined that the user's respiratory disease is worsening. can be done. If the second phoneme feature set distance is equal to the third feature set distance, it may be determined that the user's respiratory disease has not changed (i.e., is not improving or worsening, at least generally).

At step 6240, actions are initiated based on the determined change in the user's respiratory illness. Exemplary actions include actions and recommendations to treat respiratory diseases and/or symptoms of diseases. Step 6240 is adapted to be performed by an embodiment of decision support tool 290 (including disease monitor 292, prescription monitor 294, and/or drug efficacy tracker 296) and/or presentation component 220 of FIG. Good too.

The above operations may include sending an alert or notification to a user via user equipment, such as user equipment 102a-102n of FIG. 1, or to a clinician via clinician user equipment, such as clinician user equipment 108 of FIG. This may include sending or transmitting electronically. The notification may indicate whether there is a change in the user's respiratory condition and, in some embodiments, whether the change is an improvement. The notification or alert may include a respiratory illness score that quantifies or characterizes the change in the user's respiratory illness and/or the current state of the respiratory illness.

In some embodiments, the operations may include processing the respiratory disease information to further make decisions, including recommending treatment and support based on the user's respiratory disease. Good too. Such recommendations may include consultation with a health care provider, continuation of existing prescriptions or over-the-counter medications (e.g., prescription refills), modification of current treatment dosages or medications, and/or treatment of respiratory conditions. Continuing monitoring is recommended. One or more of these recommended actions may be implemented in response to a detected change (or no change) in respiratory disease. For example, based on the determined change (or no change), embodiments of the present disclosure may cause the user's healthcare provider appointment and/or prescription refill.

7-14 illustrate various aspects of exemplary embodiments of the present disclosure as actually implemented. For example, FIGS. 7-14 illustrate analysis of acoustic properties, correlations between acoustic properties and user respiratory disease (including symptoms), and aspects of self-reported information. The information reflected in these drawings may have been collected for many collection checkpoints (eg, at the clinic/laboratory and/or home) of multiple users. An example process for collecting information is described in conjunction with FIG. 3B.

In one embodiment, FIG. 7 shows representative changes in exemplary acoustic properties over time. In this embodiment, acoustic characteristics are extracted from audio samples acquired at two collection checkpoints (Visit 1 and Visit 2). Visit 1 may represent a collection checkpoint when the user is ill, and visit 2 may represent a collection checkpoint when the user is well (ie, after recovering from illness). As shown in FIG. 7, properties have been measured for seven phonemes, and graphs 710, 720, and 730 show the change in the acoustic properties of each phoneme between two visits. Graph 710 shows the change in jitter (a measure of pitch variability), graph 720 shows the change in shimmer (a measure of amplitude), and graph 730 shows the change in spectral contrast. Graphs 710 and 720 demonstrate that a patient's speech may be more stable after recovery from a respiratory illness, as jitter and shimmer decrease during recovery (i.e., between Visits 1 and 2) for all phonemes. It shows. The graph 730 shows the nasal sound (

) show increased spectral contrast at high frequencies, consistent with nasal resonance becoming more pronounced with less nasal congestion during recovery.
FIG. 8 shows a graphical representation of the decay constant of respiratory infection symptoms. Histogram 810 shows the attenuation constants for all symptoms, histogram 820 shows the attenuation constants for nasal congestion symptoms, and histogram 830 shows the attenuation constants for symptoms other than nasal congestion. Examples of nasal congestion symptoms include the need to blow the nose, nasal congestion, and postnasal drip. On the other hand, symptoms other than nasal congestion include runny nose, sore throat, and mucus-rich nasal discharge. The exponential decay model utilized for histograms 810, 820, and 830 is the score ≈ ae ^{− b (days − 1)} + ε, which in turn translates into daily symptom phenotypes (i.e., nasal congestion, Fitted for symptoms other than nasal congestion or all symptoms). Positive values in histograms 810, 820, and 830 correspond to a reduction in symptoms, zero values correspond to no change, and negative values correspond to worsening of symptoms. Histograms 810, 820, and 830 show that the recovery profile of self-reported symptoms is variable. Two examples of recovery profiles are described in conjunction with FIG.

Figure 9 shows the correlation between acoustic properties and self-reported respiratory infection symptoms. Graph 900 shows separate scores computed for the sum of all symptom ratings (e.g., composite symptom score), the sum of all nasal congestion-related symptom ratings, and the sum of all non-nasal congestion-related symptom ratings. Based on the damping constant of Spearman's correlation coefficients are calculated and all correlation values with a significant trend (p<0.1) are shown in graph 900 as a function of symptom group. In graph 900, the absolute value of the correlation is plotted.

For most acoustic characteristics, the direction of correlation is the same across symptom groups. However, while formant 1 bandwidth variability (bw1sdF) is positively correlated with symptoms other than nasal congestion, it is negatively correlated with nasal congestion symptoms (therefore, it is not correlated with the sum of all symptoms). do not have). Graph 900 shows that there is a strong correlation between changes in high frequency spectral structure associated with the nasal congestion phenotype and changes in self-reported symptoms compared to non-nasal congestion phenotypes.

Figure 10 shows the change in self-reported symptom scores for two patients over time. Graph 1010 shows the changes for one patient (Subject 26), with a slow decay of composite symptom score (CSS) upon recovery. In contrast, graph 1020 shows a relatively rapid decay of CSS during recovery for another patient (subject 14).

FIGS. 11A and 11B show graphical representations of rank correlations between distance metrics and self-reported symptom scores computed for different acoustic characteristics. Graph 1100 of FIG. 11A represents a rank correlation for a first set of acoustic characteristics, while graph 1150 of FIG. 11B represents a rank correlation for a second set of acoustic characteristics. Graphs 1100 and 1150 show the feature vector distance metric and the seven phonemes (

) shows the distribution of Spearman's rank correlations between self-reported symptom scores (e.g., CSS) across groups of monitoring patients for all possible combinations of. The phoneme combinations are sorted in ascending order based on the interquartile coefficient of variation (IQR/median).

The acoustic characteristics of these graphs 1100 and 1150 may be extracted from audio samples collected on different days according to embodiments of the present disclosure. Even if one voice sample was collected from each patient on a day when the patient was sick, and another voice sample was collected from each patient on a later day when the patient was well (i.e., not sick). good. This distance method calculation may be performed in conjunction with the phoneme characteristic comparator 274 as described. The distance metric is correlated to the patient's self-report symptom score (eg, Spearman's r), which may be determined as described in conjunction with the self-report data evaluator 2746. Graphs 1100 and 1150 represent phonemes

The subset containing , had the lowest interquartile coefficient of variation, indicating its association with respiratory disease detection. In one embodiment of the present disclosure, based on the results shown in graphs 1100 and 1150, further refinement is performed using sparse PCA to identify a subset of the acoustic properties of each of the three phonemes. A total of 32 characteristics (

12 pieces for

12 pieces for, and

A subset of 8 characteristics) may be selected.
FIG. 12A is a graph 1200 showing rank correlation values between distance metrics and self-reported symptom scores for different patients. The distance metric used to calculate the rank correlation value is the distance metric used to calculate the rank correlation value.

) may be based on 32 phoneme characteristics derived from . In graph 12200, patients are sorted from left to right in descending order of change in symptoms (this may not necessarily match the degree of rank correlation indicated by the bars in graph 1200). (*) indicates that the indicated rank correlation was determined to be statistically significant (eg, p<0.05). Graph 1200 shows that correlations are generally higher for patients with more rapid recovery (ie, larger values of b). The mean rank correlation for patients with b-values higher than the median is 0.7 (±0.13), compared to 0.46 (±0.33) for patients with b-values lower than the median. . The median correlation between the computed distance metric and self-reported composite symptom score (CSS) is 0.63.

FIG. 12B shows the results of a paired T-test (p-value) for changes between sick and well visits to indicate a statistically significant correlation, according to an embodiment of the present disclosure. Table 1210 only includes values when p<0.05. Table 1210 shows the results for all studied patients and only for patients in the high recovery group (measured by the decay constant b). In table 910, standard deviation is indicated by "sd" and logarithmic transformation is indicated by "LG".

FIG. 13 depicts a graphical representation of relative changes in acoustic characteristics and self-reported symptoms over time for three exemplary patients, identified as Subjects 17, 20, and 28, according to some embodiments. It shows. Graphs 1310, 1320, and 1330 depict, respectively, each patient's self-reported composite symptom score (CSS) (indicated by vertical bars) and distance metrics computed from phoneme feature vectors (indicated by dashed lines) over time. It shows change. Graph 1310 shows that subject 17 exhibits a significant and relatively monotonic reduction in symptoms over time, which is also reflected in the distance metric. Graph 1320 shows that Subject 28's symptom reduction was more gradual and less monotonous than Subject 17, with Subject 28's recovery stabilizing around days 7 to 12, and then symptoms slightly decreasing on day 13. It shows that you are doing it. Graph 1320 also shows moderate agreement with the distance metric, allowing the transition from illness to recovery to be observed. In contrast to graphs 1310 and 1320, graph 1330 shows that subject 20's self-reported symptoms were initially mild (CSS=5 on day 1) and that symptoms other than nasal congestion (cough and sore throat) were It shows that it gets worse over time. As a result, graph 1330 has fewer matches with the distance metric than graphs 1310 and 1320.

Graph 1340 of FIG. 13 includes a box plot of distance metrics computed over time for a group of monitoring patients including subjects 17, 20, and 28. Graph 1340 shows a tendency for the distance to decrease as the patient approaches a recovered (or "healthy") state, which may be around 14 days.

FIG. 14 shows an exemplary representation of the performance of a respiratory infection detector. Specifically, FIG. 14 illustrates quantification of the ability of one embodiment of the present disclosure to detect changes in respiratory disease as measured by self-reported symptom scores (eg, CSS). Graph 1410 plots the change in the distance metric against the change in self-reported symptom scores and shows that the greater the difference in self-reported symptoms on a given day, the greater the distance between the phoneme feature vectors. Graph 1420 illustrates receiver behavior for detecting different magnitude changes in self-reported symptom scores using phoneme features (and distances computed between phoneme feature vectors), according to embodiments of the present disclosure. Figure 2 shows characteristic (ROC) curves and associated area under the curve (AUC) values. As shown, for a 7 point change (corresponding to 20% of the composite symptom score range of 0-35), the AUC value is 0.89.

15A-15M illustrate an exemplary embodiment of a computer program routine for extracting phonemic features from audio data to track respiratory disease, as described herein. To this end, the computer program routines of FIGS. 15A-15M may be adapted for use by user audio monitor 260 or one or more of its subcomponents. The computer program routines of FIGS. 15A-15M may also be adapted to perform one or more aspects of methods 6100 and 6200 of FIGS. 6A and 6B, respectively.

From the above, various aspects of the technology are provided that are directed to systems and methods for monitoring respiratory disease in a user. It is understood that various features, subcombinations, and improvements of the embodiments described herein are useful and may be adopted in other embodiments without reference to other features or subcombinations. Furthermore, the sequence and order of steps depicted in the example method or process is in no way intended to limit the scope of the present disclosure; in fact, these steps may be varied within the scope of the embodiments herein. may appear in different orders. Such modifications and combinations are also considered to be within the scope of the embodiments of the present disclosure.

Having described various implementations above, exemplary computer environments suitable for implementing embodiments of the present disclosure are described below. Referring to FIG. 16, an exemplary computer device is provided, generally referred to as computer device 1700. Computer equipment 1700 is one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present disclosure. Nor should computing device 1700 be construed as having any dependency or requirement regarding any one or combination of illustrated components.

Embodiments of the present disclosure describe computer code or machine usage, such as program modules executed by a computer or other machine, such as a personal digital assistant, a smartphone, a tablet PC, or other handheld or wearable device, such as a smart watch. may be described in the general context of computer-enabled instructions (including computer-usable or computer-executable instructions). Generally, program modules, including routines, programs, objects, components, data structures, etc., represent code that performs particular tasks or performs particular abstract data types. Embodiments of the present disclosure may be implemented in a variety of system configurations, including hand-held devices, consumer electronics, general purpose computers, or special purpose computing equipment. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

Referring to FIG. 16, computing equipment 1700 includes memory 1712, one or more processors 1714, one or more presentation components 1716, one or more input/output (I/O) ports 1718, one or more A bus 1710 is provided that directly or indirectly couples various equipment, including a plurality of I/O components 1720 and an exemplary power supply 1722. Some embodiments of computing equipment 1700 may further include one or more radios 1724. Bus 1710 represents one or more buses (address bus, data bus, or a combination thereof). For clarity, the various blocks in FIG. 16 are shown as lines; in fact, these blocks represent logical components and not necessarily actual components. For example, a presentation component such as a display device that serves as an I/O component may also be considered. Additionally, the processor may include memory. FIG. 16 is merely an illustration of example computer equipment that may be used in connection with one or more embodiments of the present disclosure. There is no distinction between categories such as "workstation," "server," "laptop," or "handheld device." This is because, within the scope of FIG. 16, everything is considered by reference to "computer equipment".

Computer equipment 1700 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computing device 1700 and includes both volatile and nonvolatile media, removable and non-removable media. As one non-limiting example, computer-readable media may include computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. including. Computer storage media include memory technologies such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and compact disk read-only memory (CD-ROM). , optical disk storage such as a digital versatile disk (DVD), magnetic storage devices such as magnetic cassettes, magnetic tapes, magnetic disk storage, or any other medium for storing desired information. and can be accessed by computer equipment 1700. Computer storage media do not themselves contain signals. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. As one non-limiting example, communication media includes wired media, such as a wired network or direct wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media. Any combinations of these should also be included within the scope of computer-readable media.

Memory 1712 includes computer storage media in the form of volatile and/or non-volatile memory. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include, for example, solid state memory, hard drives, and optical disk drives. Computing device 1700 includes one or more processors 1714 that read data from various devices, such as memory 1712 or I/O components 1720. Presentation component 1716 presents the data indicators to a user or other device. Exemplary presentation components 1716 include displays, speakers, printing components, vibration components, and the like.

I/O ports 1718 enable logical coupling of computing equipment 1700 to other equipment, including I/O components 1720, some of which may be embedded. Exemplary components include a microphone, joystick, game pad, satellite dish, scanner, printer, or wireless device. I/O component 1720 may provide a natural user interface (NUI) to process air gestures, speech, or other physiological input generated by a user. In some cases, the input may be sent to an appropriate network element for further processing. The NUI includes speech recognition, touch and stylus recognition, facial recognition, biometrics, gesture recognition (both on-screen and screen-adjacent), air gestures, head and eye tracking, and touch recognition associated with the display on computing device 1700. may be implemented in any combination. Computing device 1700 may be equipped with a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, and combinations thereof, for gesture detection and recognition. Computing equipment 1700 may also be equipped with an accelerometer or gyroscope to enable detection of motion. Accelerometer or gyroscope output may be provided to a display of computing device 1700 to render immersive augmented reality or virtual reality.

Some embodiments of computing equipment 1700 may include one or more radios 1724 (or similar wireless communication components). The wireless device 1724 transmits and receives wireless communications. Computing device 1700 may be a wireless terminal configured to receive communications and media over various wireless networks. Computing device 1700 may communicate via wireless protocols such as code division multiple access (“CDMA”), Global System for Mobile (“GSM”), time division multiple access (“TDMA”), or other wireless means. This allows communication with other devices. Wireless communications may be short-range connections, long-range connections, or a combination thereof. As used herein, the "short" and "long" connection types do not represent a spatial relationship between two devices. Instead, these connection types generally represent short-range and long-range as different categories or connection types (i.e., primary and secondary connections). An example of a short-range connection is a Wi-Fi connection to a device that provides access to a wireless communication network (e.g., a mobile hotspot), such as a wireless local area network (WLAN) connection using the 802.11 protocol. ) connection, another example of a short-range connection includes, but is not limited to, a Bluetooth connection to another computer device, or near-field communication. Long-range connections include, by way of example and without limitation, connections using one or more of CDMA, General Packet Radio System (GPRS), GSM, TDMA, and 802.16 protocols.

Many different configurations of the various illustrated and non-illustrated components are possible without departing from the scope of the following claims. Embodiments of the present disclosure are described to be illustrative rather than restrictive. Alternate embodiments will be apparent to a reader of this disclosure after reading this disclosure. Alternative means of implementing the foregoing may be accomplished without departing from the scope of the following claims. Certain features and subcombinations are useful and can be employed without reference to other features and subcombinations and are contemplated within the scope of the claims.

Claims (47)

1. A computerized system for monitoring respiratory disease in a human subject, comprising: one or more processors; receiving audio data of a first phoneme; determining a first set of phoneme characteristics including at least one acoustic characteristic characterizing a first portion of the first audio data including a first phoneme; Computer-executable instructions are stored for performing operations comprising: monitoring the respiratory disease by comparing the first set of phoneme characteristics to a second set of phoneme characteristics determined from audio data. A computerized system comprising computer memory and. The computerized system of claim 1, further comprising an acoustic sensor configured to capture audio information. 3. The computerized system of claim 2, wherein the acoustic sensor is integrated into a smart speaker. The first phoneme feature set is

2. The computerized system of claim 1, comprising an acoustic characteristic characterizing at least one phoneme comprising: . the first phoneme feature set includes a first phoneme associated with the first portion of the first audio data; a second phoneme associated with the second portion of the first audio data; and an acoustic characteristic characterizing a third phoneme associated with a third portion of the first audio data, wherein the first phoneme is

and the second phoneme is

and the third phoneme is

2. The computerized system of claim 1, comprising: Said

The acoustic properties of the phoneme include the standard deviation of the bandwidth of formant 1 (Fl), the interquartile range of pitch, the spectral entropy determined for frequencies from 1.6 to 3.2 kilohertz (kHz), jitter, at least one of the standard deviations of the Mel frequency cepstral coefficients MFCC9 and MFCC12, the average of the Mel frequency cepstral coefficients MFCC6, and the spectral contrast determined for frequencies from 3.2 to 6.4 kHz;

The acoustic properties of the phonemes are harmonicity, standard deviation of F1 bandwidth, interquartile range of pitch, spectral entropy determined for frequencies 1.5-2.5 kHz and 1.6-3.2 kHz; Spectral flatness determined for frequencies from 1.5 to 2.5 kHz, standard deviations of Mel frequency cepstral coefficients MFCC1, MFCC2, MFCC3, and MFCC11, average of Mel frequency cepstral coefficients MFCC8, and 1.6 to 3. at least one of the spectral contrasts determined for a frequency of 2 kHz;

The acoustic properties of the phonemes are harmonicity, standard deviation of F1 bandwidth, interquartile range of pitch, spectral entropy determined for frequencies 1.5-2.5 kHz and 1.6-3.2 kHz; Spectral flatness determined for frequencies from 1.5 to 2.5 kHz, standard deviation of Mel frequency cepstral coefficients MFCC2 and MFCC10, mean of Mel frequency cepstral coefficients MFCC8, shimmer, for frequencies from 3.2 to 6.4 kHz 6. The computerized system of claim 5, comprising at least one of a spectral contrast determined for a 200 hertz (Hz) 1/3 octave band standard deviation. The operations include determining a first phoneme by performing automatic speech recognition on the first portion of the first audio data; 2. The computerized system of claim 1, further comprising: associating with the first phoneme. performing automatic speech recognition includes determining text corresponding to the first portion of the first audio data; and determining the first phoneme based on the text. 8. Computerized system according to claim 7. the first audio data is associated with a first time segment that corresponds to a first date and time value; the second audio data is associated with a second time segment that corresponds to a second date and time value; Monitoring the respiratory disease in the human subject comprises determining a characteristic distance measurement of at least some of the characteristics in the first phoneme feature set and the second phoneme feature set; and determining that the respiratory disease of the human subject has changed between the second date and time value and the first date and time value based on the measurement result. computerized system. 10. The computerized system of claim 9, wherein the second date and time value occurs 18 to 36 hours after the first date and time value. The operations include receiving first physiological data of the human subject associated with a first time segment associated with the first audio data, and storing the physiological data in a recording. The computerized system of claim 1, further comprising: the first audio data is associated with a first time segment, the motion is associated with a first time segment, physiological data regarding the human subject, the human subject in the first time segment; further comprising determining first contextual data of the human subject, the first contextual data comprising at least one of information regarding a location of the human subject, or contextual information associated with the first time segment; The computerized system of claim 1, wherein a set is further determined based on the first context data. the first set of phoneme characteristics is determined from a plurality of other sets of phoneme characteristics each associated with a first datetime value occurring before a second time interval associated with the second audio data; , the computerized system of claim 1. Comparing the first set of phoneme characteristics to the second set of phoneme characteristics includes determining the difference between at least a portion of the first set of phoneme characteristics and at least a portion of the second set of phoneme characteristics. 2. The computerized system of claim 1, comprising determining a Euclidean distance or a Levenshtein distance. Comparing the first set of phoneme characteristics to the second set of phoneme characteristics includes at least a first characteristic of the first set of phoneme characteristics and a corresponding second set of phoneme characteristics of the second set of phoneme characteristics. 2. The computerized system of claim 1, comprising performing a comparison with a characteristic. monitoring the respiratory disease in the human subject comprises determining a first feature set distance by performing a comparison of the first phoneme feature set and the second phoneme feature set; and determining that the respiratory disease of the human subject has changed by comparing a first characteristic set distance to a threshold distance. Claim wherein the threshold distance is predetermined by a clinician or automatically determined based on one or more of a user's physiological data, user settings, or user's respiratory disease history information. 17. Computerized system according to paragraph 16. The operations further include receiving a third set of phoneme characteristics representative of criteria at which the human subject is determined to be free of the respiratory disease, and monitoring the human subject for the respiratory disease. determining a first feature set distance by performing a comparison between the first phoneme feature set and the second phoneme feature set; determining a second feature set distance by performing a second comparison with a feature set; and performing a third comparison between the first phoneme feature set and the third phoneme feature set. determining a third characteristic set distance by; performing a fourth comparison of the second characteristic set distance and the third characteristic set distance; and based on the fourth comparison, determining the third characteristic set distance. providing an indicator that the respiratory disease of the human subject is improving when the characteristic aggregation distance of the second characteristic aggregation distance is shorter than the third characteristic aggregation distance; providing an indication that the human subject's respiratory disease is worsening if the second trait aggregation distance is greater than the third trait aggregation distance; 17. The computerized system of claim 16, comprising: providing an indication that the human subject's respiratory disease is unchanged when the human subject's respiratory disease is unchanged. The third phoneme feature set representing the reference includes phoneme features having feature values determined based on an average of a set of phoneme feature values, and each of the set of phoneme feature values is unique to the human subject. 3. The computerized system of claim 2, wherein: is determined from different time segments during the time when the respiratory disease is determined to be absent. 2. The action further comprises initiating action based on a change in the respiratory disease determined by comparing the first set of phoneme characteristics to the second set of phoneme characteristics. computerized system. initiating an action based on a change in the respiratory disease of the human subject, issuing a notification to at least one of a user equipment associated with the human subject or a clinician of the human subject; making an appointment between the human subject and the human subject's clinician; providing recommendations to modify treatment for the respiratory illness; and requesting refills of prescription drugs. 21. The computerized system of claim 20. further comprising a user equipment associated with the human subject, wherein monitoring the respiratory disease in the human subject is based on a comparison of at least the first set of phonemic characteristics with the second set of phonemic characteristics. 2. The computerized system of claim 1, further comprising: determining a respiratory illness score using a computer, said act further comprising causing said respiratory illness score to be displayed on a user interface of said user equipment. further comprising a user equipment associated with the human subject, wherein monitoring the respiratory disease in the human subject is based on a comparison of at least the first set of phonemic characteristics with the second set of phonemic characteristics. determining a transmission risk level indicative of a risk of the human subject transmitting a pathogen associated with the respiratory disease, the act causing the transmission risk level to be displayed on a user interface of the user equipment. The computerized system of claim 1, further comprising: further comprising a user equipment associated with the human subject, wherein monitoring the respiratory disease in the human subject is based on a comparison of at least the first set of phonemic characteristics with the second set of phonemic characteristics. determining the trend of the respiratory disease in the human subject, the act further comprising causing the trend of the respiratory disease in the human subject to be displayed on a user interface of the user equipment. , the computerized system of claim 1. 2. The computerized system of claim 1, wherein the first portion of the first audio data includes sustained utterances of elementary vowel phonemes, and wherein the first set of phoneme characteristics is based on maximum utterance times. The first audio data includes a recording of a spoken sentence including a plurality of phonemes, and the first set of phoneme characteristics is one of speech rate, average pause length, number of pauses, and global signal-to-noise ratio. 2. The computerized system of claim 1, comprising: or more than one. A method of treating a respiratory disease utilizing an acoustic sensor device, the method comprising: receiving first audio data associated with a first time segment and including speech information of a human subject; and a first phoneme. determining a first set of phoneme characteristics comprising at least one acoustic characteristic characterizing a first portion of the first audio data comprising: and determining from the second audio data associated with a second time segment treating the respiratory disease by performing a comparison of the first set of phoneme characteristics to a second set of phoneme characteristics determined by the human subject; and initiating a treatment procedure for the human subject based at least on the comparison. A method, including steps for doing so. 28. The method of claim 27, wherein initiating the treatment procedure includes determining at least one of a therapeutic agent, a dosage, and a method of administering the therapeutic agent. The therapeutic agent may include PLpro inhibitors (apiromod, EIDD-2801, ribavirin, valganciclovir, β-thymidine, aspartame, oxprenolol, doxycycline, acetophenazine, iopromide, riboflavin, leproterol, 2,2'-cyclotidine, Ramfenicol, chlorphenesin carbamate, levodropropizine, cefamandole, floxuridine, tigecycline, pemetrexed, L(+)-ascorbic acid, glutathione, hesperetin, ademethionine, masoprocol, isotretinoin, dantrolene, sulfasalazine Antibacterial, silibinin, nicardipine, sildenafil, platycodin, chrysin, neohesperidin, baicalin, sugetriol-3,9-diacetate, (-)-epigallocatechin gallate, phytantrin D, 2-(3,4-dihydroxyphenyl) )-2-[[2-(3,4-dihydroxyphenyl)-3,4-dihydro-5,7-dihydroxy-2H-1-benzopyran-3-yl]oxy]-3,4-dihydro-2H- 1-benzopyran-3,4,5,7-tetrol, 2,2-di(3-indolyl)-3-indolone, (S)-(1S,2R,4aS,5R,8aS)-1-formamide-1 ,4a-dimethyl-6-methylene-5-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-2-amino-3- phenylpropanoate, piceatannol, rosmarinic acid and magnol), 3CLpro inhibitors (rimesycline, chlorhexidine, alfuzosin, cilastatin, famotidine, almitrin, progavid, nepafenac, carvedilol, amprenavir, tigecycline, montelukast, carminic acid) , mimosine, flavin, lutein, cefpiramide, pheneticillin, candoxatril, nicardipine, estradiol valerate, pioglitazone, conivaptan, telmisartan, doxycycline, oxytetracycline, (1S,2R,4aS,5R,8aS)-l-formamide-l,4a- Dimethyl-6-methylene-5-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl 5-((R)-1,2 -dithioran-3-yl)pentanoate, vetronal, chrysin-7-Ο-β-glucuronide, andrographiside, (1S,2R,4aS,5R,8aS)-l-formamide-l,4a-dimethyl-6-methylene -5-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-2-nitrobenzoate, 2β-hydroxy-3,4-seco Friederolactone-27-oic acid (S)-(1S,2R,4aS,5R,8aS)-1-formamide-1,4a-dimethyl-6-methylene-5-((E)-2-(2- Oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-2-amino-3-phenylpropanoate, isodecortinol, cerebisterol, hesperidin, neohesperidin, andrograpanin , 2-((1R,5R,6R,8aS)-6-hydroxy-5-(hydroxymethyl)-5,8a-dimethyl-2-methylenedecahydronaphthalen-1-yl)ethylbenzoate, cosmosyin, cleistocaltone A, 2,2-di(3-indolyl)-3-indolone, violobin, gunizicin, filaembrinol, theaflavin 3,3'-di-Ο-gallate, rosmarinic acid, kouichenside I, oleanolic acid, stigmast RdRp inhibitors (valganciclovir, chlorhexidine, ceftibuten, fenoterol, fludarabine, itraconazole, cefuroxime, atovaquone, chenodeoxycholic acid, cromolyn, pancuronium bromide, cortisone, Tibolone, novobiocin, silibinin, idarubicin bromocriptine, diphenoxylate, benzylpenicilloyl G, dabigatran etexilate, vetronal, gunizicin, 2β,30β-dihydroxy-3,4-secofriederolactone-27-lactone, 14-deoxy -11,12-didohydroandrographolide, gnidithrin, theaflavin 3,3'-di-Ο-gallate, (R)-((1R,5aS,6R,9aS)-1,5a-dimethyl-7-methylene -3-oxo-6-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydro-1H-benzo[c]azepin-1-yl)methyl 2-amino -3-phenylpropanoate, 2β-hydroxy-3,4-secofriederolactone-27-oic acid, 2-(3,4-dihydroxyphenyl)-2-[[2-(3,4-dihydroxyphenyl) )-3,4-dihydro-5,7-dihydroxy-2H-1-benzopyran-3-yl]oxy]-3,4-dihydro-2H-1-benzopyran-3,4,5,7-tetron, fila Embricin B, 14-hydroxycyperotandone, andrographiside, 2-((1R,5R,6R,8aS)-6-hydroxy-5-(hydroxymethyl)-5,8a-dimethyl-2-methylenedideca Hydronaphthalen-1-yl)ethylbenzoate, andrographolide, Sugetriol-3,9-diacetate, Baicalin, (1S,2R,4aS,5R,8aS)-1-formamide-1,4a-dimethyl-6 -methylene-5-((E)-2-(2-oxo-2,5-dihydrofuran-3-yl)ethenyl)decahydronaphthalen-2-yl-5-((R)-1,2-dithiolane -3-yl)pentanoate, l,7-dihydroxy-3-methoxyxanthone, l,2,6-trimethoxy-8-[(6-Ο-β-D-xylopyranosyl-β-D-glucopyranosyl)oxy]-9H -xanthene-9-one, and/or 1,8-dihydroxy-6-methoxy-2-[(6-Ο-β-D-xylopyranosyl-β-D-glucopyranosyl)oxy]-9H-xanthen-9-one , 8-(β-D-glucopyranosiloxy)-l,3,5-trihydroxy-9H-xanthen-9-one), diosmin, hesperidin, MK-3207, venetoclax, dihydroergocristine, borazine, R428 , diterqualinium, etoposide, teniposide, UK-432097, irinotecan, lumacaftor, velpatasvir, eluxadrine, ledipasvir, lopinavir/ritonavir in combination with ribavirin, alferon, and prednisone, dexamethasone, azithromycin, remdesivir, boceprevir, umifenovir, and favipiravir, alpha - Ketoamide compounds, RIG1 pathway activators, protease inhibitors, remdesivir, galidesivir, favilavir/avifavir, molnupiravir (MK-4482/EIDD 2801), AT-527, AT-301, BLD-2660, favipiravir, camosta, SLV213 emtricitabine /tenofivir, clevudine, dalcetrapib, boceprevir, ABX464, (3S)-3-({N-[(4-methoxy-1H-indol-2-yl)carbonyl]-L-leucyl}amino)-2-oxo-4 -[(3S)-2-oxopyrrolidin-3-yl]butyl dihydrogen phosphate, and its pharmaceutically acceptable salts, solvates, or hydrates (PF-07304814), (1R,2S ,5S)-N-{(1S)-1-cyano-2-[(3S)-2-oxopyrrolidin-3-yl]ethyl}-6,6-dimethyl-3-[(3-methyl-N- (Trifluoroacetyl)-L-valyl]-3-azabicyclo[3.1.0]hexane-2-carboxamide or its solvate or hydrate (PF-07321332), S-217622, glucocorticoid, recovery Phase plasma, recombinant human plasma, monoclonal antibody, lavulizumab, VIR-7831/VIR-7832, BRII-196/BRII-198, COVI-AMG/COVI DROPS (STI-2020), bamlanivimab (LY-CoV555), mavrilimab, leronlimab (PRO140), AZD7442, lenzilumab, infliximab, adalimumab, JS 016, STI-1499 (COVIGUARD), lanadelumab (Taczylo), canakinumab (Ilaris), gimcilumab, otilimab, antibody cocktail, recombinant fusion protein, anticoagulant, IL-6 Receptor agonists, PIKfyve inhibitors, RIPK1 inhibitors, VIP receptor agonists, SGLT2 inhibitors, TYK inhibitors, kinase inhibitors, bemcentinib, acalabrutinib, rosmapimod, baricitinib, tofacitinib, H2 inhibitors, anthelmintics, and furinib 29. The method of claim 28, wherein the method is selected from the group consisting of inhibitors. The therapeutic agent is (3S)-3-({N-[(4-methoxy-1H-indol-2-yl)carbonyl]-L-leucyl}amino)-2-oxo-4-[(3S)- 29. The method of claim 28, which is 2-oxopyrrolidin-3-yl]butyl dihydrogen phosphate, or a pharmaceutically acceptable salt, solvate, or hydrate thereof (PF-07304814). The therapeutic agent is (1R,2S,5S)-N-{(1S)-1-cyano-2-[(3S)-2-oxopyrrolidin-3-yl]ethyl}-6,6-dimethyl-3 -[(3-Methyl-N-(trifluoroacetyl)-L-valyl]-3-azabicyclo[3.1.0]hexane-2-carboxamide or its solvate or hydrate (PF-07321332) 39. The method of claim 38. Initiating management of the treatment procedure includes generating a graphical user interface element provided for display on user equipment, the graphical user interface element including at least the first set of phoneme characteristics. 28. The method of claim 27, wherein the recommendation of the treatment procedure is based on a comparison with a set of phoneme characteristics of two phoneme characteristics. 33. The method of claim 32, wherein the user equipment is separate from the acoustic sensor equipment. 33. The method of claim 32, further comprising applying the treatment procedure to the human subject based on the recommendation. 28. The method of claim 27, wherein the respiratory disease comprises coronavirus disease 2019 (COVID-19). A computerized method for tracking the efficacy of a therapeutic agent for treating a respiratory disease in a human subject, the method comprising: receiving a first set of phoneme features and a second set of phoneme features, each representing speech information of the human subject; the second set of phoneme characteristics is associated with a second date and time value that occurs after the first date and time value associated with the first set of phoneme characteristics, and the therapeutic agent is administered to the human subject. the first characteristic by performing a first comparison of the first set of phoneme characteristics and the second set of phoneme characteristics; A computerized method comprising: determining an aggregation distance; and determining whether there is a change in the respiratory disease of the human subject based on the first characteristic aggregation distance. 37. The computerized method of claim 36, wherein the respiratory disease is a respiratory infection and the therapeutic agent is an antimicrobial agent. 38. The computerized method of claim 37, wherein the therapeutic agent is an antibiotic. 38. The computerized method of claim 37, further comprising determining a change in efficacy of the antibiotic based on determining whether there is a change in the respiratory disease of at least the human subject. 37. The step of determining whether there is a change in the respiratory disease of the human subject comprises determining whether the respiratory disease has improved, worsened, or remained unchanged. computerized method. 37. The computerized method of claim 36, further comprising initiating an action to treat the human subject based on determining whether there is a change in the respiratory disease of the human subject. 42. The computerized method of claim 41, wherein the act of treating the human subject is initiated by a determination that the respiratory disease has worsened. 42. The computerized method of claim 41, wherein the act of treating the human subject is initiated by a determination that the respiratory disease has worsened or remains unchanged. 42. The computerized method of claim 41, wherein the act of treating the human subject comprises modifying a treatment procedure for the human subject. 45. The computerized method of claim 44, wherein altering the treatment procedure of the human subject comprises initiating a recommendation to adjust one or more of the therapeutic agent or dosage of the therapeutic agent. Method. 45. The computer of claim 44, wherein modifying the treatment procedure for the human subject includes sending a message to the human subject's care provider requesting a modification to the treatment procedure for the human subject. method. 42. The act of treating the human subject comprises electronically initiating a refill request for the therapeutic agent to a pharmacy determined from the human subject's electronic health record (EHR). The computerized method described.