DE112021003125T5 - Systems and methods for generating early health-based alerts from continuously collected data - Google Patents

Abstract

A speech-based health detection system used to monitor a user's voice changes to detect early signs of potential health problems. The system may include a voice-based health recognition system server communicatively coupled to sensors, such as wearable computing devices. The sensors can be used to collect signal data and monitor a user's vital signs. The server can be configured to receive the signal data from the sensors, identify features from the signal data, and extract features from the features. The extracted features may include user voice features such as pitch, speed, range, weight, and timbre. The server can be configured to detect whether the extracted features exceed a predetermined threshold. The system can therefore provide early health-based alerts for potential user health issues when the detected features exceed the predetermined threshold and fall outside of their regular control limits.

Description

priority

This application claims the benefit of and priority from U.S. Patent Application No. 17/337,814, filed June 3, 2021, which in turn claims priority from U.S. Provisional Application No. 63/124,306 , filed December 11, 2020, and U.S. Provisional Application No. 63/034,811 , filed June 4, 2020, all of which are incorporated herein in their entirety.

Area

The field of the present disclosure generally relates to artificial intelligence computing. More particularly, the field of the disclosure relates to processing continuously sensed data to generate early warning health alerts in response to sensed changes in one or more known characteristics of a user.

background

Early detection has been identified as a key factor in treating serious health problems. It is also known that there are many early signs of some serious health problems. Some of the early signs of stroke that have been recognized include drooping face, arm weakness, and difficulty speaking. Another example is the well-known early signs of Alzheimer's, which can include an increase in body temperature, heart rate, oxygen saturation (Sp02), and pitched voice, among others. Some of these early signs can be monitored by continuously monitoring basic vital signs such as blood pressure, heart rate, body temperature and SpO2.

Currently, many electronic watches and fitness trackers feature the ability to continuously monitor and measure various data, including some, if not all, of the basic vital signs mentioned above. And with each new generation of electronic watches and fitness trackers coming onto the market, the monitored data and the analysis of the monitored data by these devices can be used to generate an increasing amount of meaningful data and correlations. However, most vital signs, apart from body temperature and heart rate, typically need to be measured in a hospital setting, although many users would prefer to remain in the privacy and security of their own homes. In the meantime, the few vital signs that can be measured by wearable devices are still unused as these devices are not yet medically and administratively approved, which means that the accuracy of these devices may not yet meet minimum medical standards. However, as sensor technology and algorithms continue to evolve drastically, it is highly likely that next-generation devices will produce more accurate and reliable data.

character list

• 1 eine beispielhafte Darstellung eines sprachbasierten Gesundheitserkennungssystems gemäß einer Ausführungsform der vorliegenden Offenbarung ist;
• 2A eine abstrakte Darstellung einer sprachbasierten Gesundheitserkennungsvorrichtung gemäß einer Ausführungsform der vorliegenden Offenbarung ist;
• 2B eine abstrakte Darstellung bekannter Benutzerdaten gemäß einer Ausführungsform der vorliegenden Offenbarung ist;
• 3 eine detaillierte Blockdiagrammdarstellung eines sprachbasierten Gesundheitserkennungsservers ist, der in einem sprachbasierten Gesundheitserkennungssystem gemäß einer Ausführungsform der vorliegenden Offenbarung verwendet wird;
• 4 ein Flussdiagramm eines Prozesses zum Erzeugen bekannter Benutzerdaten gemäß einer Ausführungsform der vorliegenden Offenbarung ist;
• 5 ein Flussdiagramm eines sprachbasierten Gesundheitserkennungsprozesses gemäß einer Ausführungsform der vorliegenden Offenbarung ist;
• 6A ein Flussdiagramm eines immer eingeschalteten sprachbasierten Gesundheitserkennungsprozesses gemäß einer Ausführungsform der vorliegenden Offenbarung ist;
• 6B ein Flussdiagramm eines immer eingeschalteten sprachbasierten Gesundheitserkennungsprozesses unter Verwendung einer externen Rechenvorrichtung gemäß einer Ausführungsform der vorliegenden Offenbarung ist.

The foregoing and other aspects, features, and advantages of various embodiments of the present disclosure will become more apparent from the following description in conjunction with the following various drawing figures. The drawings relate to embodiments of the present disclosure in which:

• 1 Figure 12 is an exemplary illustration of a voice-based health recognition system according to an embodiment of the present disclosure;
• 2A Figure 12 is a high-level representation of a voice-based health recognition device according to an embodiment of the present disclosure;
• 2 B Figure 12 is an abstract representation of known user data in accordance with an embodiment of the present disclosure;
• 3 Figure 12 is a detailed block diagram representation of a voice health recognition server used in a voice health recognition system according to an embodiment of the present disclosure;
• 4 Figure 12 is a flowchart of a process for generating known user data according to an embodiment of the present disclosure;
• 5 Figure 12 is a flow diagram of a voice-based health recognition process according to an embodiment of the present disclosure;
• 6A Figure 12 is a flow diagram of an always-on speech-based health detection process according to an embodiment of the present disclosure;
• 6B a flowchart of an always-on speech-based health detection process using an external computing device according to one embodiment of the present disclosure.

Detailed description

In view of the problems described above, there is a need to monitor changes in a user in order to generate early health-related alerts for potential health problems based on continuously collected data. The embodiments described herein provide these generated early health-related alerts with systems and methods that address the

Detection and extraction of properties from audio data signals that can be used to provide early warning of potential serious health problems to the user. As described in more detail below, embodiments may allow the user to enter one or more attributes and vital signs, which may correspond to one or more spoken words and/or extracted attributes.

The embodiments may be configured to recognize continuously spoken words in a low-power, always-on (i.e., continuously recognized) mode and to extract one or more characteristics from the recognized spoken words to provide vital signs and threshold (or trend) data of the extracted characteristics over a determine a predetermined period. In the embodiments described herein, spoken words may include, but are not limited to, a variety of words, phrases, audio gestures, audio signals, etc. that may be associated with one or more users. For example, as audio and related sensor technologies advance, the embodiments described herein may be able to recognize one or more properties (or parameters, characteristics, etc.) from a user's voice and spoken words in general, such that the embodiments may Recognize, analyze or otherwise use keywords and/or any spoken words from that user's language. In other words, features of a user's voice can be recognized, parsed, or otherwise used without requiring a specific keyword or pre-programmed phrase to trigger a device or sensor for s/he to begin "listening."

In several embodiments, the extracted characteristics may include one or more voice characteristics extracted from the user's captured keywords. For example, as a non-limiting example, the extracted voice features may include pitch, voice speed, vocal range, voice weight, voice color tone, and so on. Meanwhile, in other embodiments, the extracted features may include any other health-based data extracted and/or collected with any type of sensors associated with any extracted voice features. For example, the health-based data extracted may correlate with one or more early signs of health changes, each of which may correlate with one or more potential serious health problems.

Additionally, embodiments may be configured to determine whether any of the extracted voice features have exceeded a predetermined threshold. In response to determining that one or more of the extracted voice features have exceeded their predetermined thresholds, the embodiments may be configured to generate and transmit alert data, including early sign health alert notifications, to a personal computing device of the user or the user's caregiver . For example, the early sign health alert signal can be used to provide the user with an early warning alert of a potential serious health issue and immediately seek further diagnosis of the potential issue.

Before the following embodiments are described in more detail, it should be understood that none of the embodiments described herein limit the scope of the concepts contained herein. Likewise, it should be understood that a particular embodiment described herein may have features that can be readily separated from that particular embodiment and optionally combined with or substituted for features of several other embodiments described herein.

As for the terms used herein, it is to be understood that these terms are for the purpose of describing particular embodiments and do not limit the scope of the concepts described herein and/or other embodiments. Ordinal numbers (e.g. first, second, third, etc.) are generally used to distinguish or identify different properties or steps in a group of properties or steps, and do not represent a serial or numeric delimitation. For example, must "First,""second," and "third" features or steps do not necessarily appear in that order, and the particular embodiments including such features or steps need not necessarily be limited to the three features or steps. Terms such as "left", "right", "front", "back", "top", "bottom" and the like are used for convenience and are not intended to imply a particular fixed position or orientation or direction, for example. Instead, such labels are used to represent, for example, relative position, orientation, or directions. Singular forms of "a,""an," and "the" include plural forms, unless the context clearly dictates otherwise.

For example, in certain situations, the term "logic" may represent hardware, firmware, and/or software configured to perform one or more functions. As hardware, the logic may include circuitry with data processing or storage functionality. Examples of such circuitry may include a microprocessor, one or more processor cores, a programmable gate array, a microcontroller, a controller, an application specific integrated circuit, a wireless receiver, transmitter and/or transceiver circuitry, semiconductor memory, or combinatorial logic but not limited to this.

Additionally, the term "property" herein may include any health-related data and any other sensor-related data received, transmitted, collected, processed, and processed by any type of sensors, any type of sensor-processing device (e.g., any type of wearable device), etc /or extracted, which data may be configured to detect and correlate with early signs of health changes that may correlate with one or more potential serious health problems. For example, any type of sensor may be communicatively coupled to sensor output detector logic of a voice-based sensing device, where the sensor output detector logic may be configured to detect one or more features (or properties, patterns, etc.) from the signal data received from such a sensor identified, and wherein the sensor output detector logic may be further configured to recognize one or more words from the received signal data, such that the identified features may include at least one or more of the recognized words. Similarly, as used herein, the term "voice characteristic" can include any vocal characteristics that can be extracted from the voice and words spoken by users in conjunction with other desired characteristics and detected with any type of sensors and sensor processing devices to do the same Identify and correlate with early signs of health changes, each of which may correlate with one or more potential serious health problems.

The term "machine learning" can encompass any computing circuitry that includes a digital implementation of a neural network. These circuits may include emulation of multiple neural structures and/or operations of a biologically based brain and/or nervous system. Some embodiments of machine learning and/or artificial intelligence circuits may include probabilistic calculations that may generate algorithmic approaches to deal with uncertainty, ambiguity, and contradiction in received input data. Machine learning circuits may consist of very large scale integration (VLSI) systems containing electronic analog circuits, digital circuits, mixed mode analog/digital VLSI, and/or software systems. The term "process" may include an instance of a computer program (e.g., a collection of instructions, also referred to herein as an application). In one embodiment, the process may consist of one or more threads executing concurrently (e.g., each thread may execute the same or a different instruction concurrently).

The term "processing" may include running a binary file or script, or launching an application in which an object is processed, where launching is understood to mean putting the application in an open state and, in some implementations, simulations of Actions are performed that are typical of human interactions with the application.

The term "object" generally refers to a collection of data, whether in transit (e.g., over a network) or at rest (e.g., stored), often having a logical structure or organization that requires a Categorization or typing enabled. The terms "binary file" and "binary" are used interchangeably herein.

The term "file" is used broadly to refer to a set or collection of data, information, or other content created with a computer program ver be turned. A file can be accessed, opened, stored, manipulated, or otherwise processed as a single entity, object, or unit. A file can contain other files and can contain related or unrelated content or no content at all. A file can also have a logical format and/or be part of a file system with a logical structure or organization of multiple files. Files can have a name, sometimes simply referred to as "filename", and often have attached properties or other metadata. There are many types of files, e.g. B. data files, text files, program files and directory files. A file may be created by a user of a computing device or may be created by the computing device. Access to and/or operations on a file may be mediated through one or more applications and/or the operating system of a computing device. A file system can organize the files of the computing device of a storage device. The file system can allow tracking of files and provide access to those files. A file system can also allow operations on a file. In some embodiments, operations on the file may include file creation, file modification, file opening, file reading, file writing, file closing, and file deleting.

Finally, as used herein, the terms "or" and "and/or" are to be construed as encompassing or meaning any one or any combination. Therefore, “A, B or C” or “A, B and/or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition occurs only when a combination of elements, functions, steps, or actions are in some way inherently mutually exclusive.

Referring now to 1 1, an exemplary illustration of a voice-based health recognition system 100 is shown, in accordance with embodiments of the disclosure. In many embodiments, the voice-based health recognition system 100 may include a plurality of personal sensing devices 101-109, a voice-based health recognition server 120, a caregiver server 130, and one or more data stores 140 and 142. Speech-based health recognition system 100 may utilize and/or otherwise communicate with personal computing devices 101-109, which may be configured to monitor various types of data, including but not limited to audio data, vital sign data, voice characteristics data, etc. from known users to identify early signs of potential health issues for known users.

As used herein, a known user can be a particular user, derived from a variety of sources, and identified by any of personal computing devices 101-109. For example, the voice-based health recognition system 100 may be configured to specifically identify whether a particular keyword is said by the known user. The known user may be derived from a variety of identified sources, which may be included in a predetermined list of authorized known users associated with the particular personal computing device. These identified and authorized known users can be associated with a variety of voice characteristics in their speech that are unique to that particular known user. These unique properties can be used to identify certain keywords spoken by a known user with other words spoken by an unidentified user who may not be associated with the particular device and thus not in the predetermined list of authorized known users can be found.

In many embodiments, the voice-based health recognition system 100 may use the personal computing devices 101-109, which may be configured to collect data related to the generation, recording, tracking, and processing of known user data, privacy data, threshold data, collected data such as vital signs of known Transmitting and receiving user and/or other signaling data, the voice-based health detection system 100 each providing a variety of alerts (or alert notifications) related to the transmitted and received data from the sensors in response to one or more vocal characteristics of the known users representing a predetermined Exceed threshold can generate. In such embodiments, the voice characteristics may include one or more voice characteristics extracted from the audio data (i.e., extracted voice characteristics) that are specific to the known user, such as, but not limited to, voice pitch, voice speed, vocal range, voice weight, voice color tone, and /or similar.

As in the in 1 illustrated embodiment, the voice-based Health detection server 120 may be communicatively coupled to one or more network(s) 110, such as the Internet. Voice-based health assessment server 120 may be implemented to transmit a variety of data over network 110 to any number of computing devices, such as but not limited to personal computing devices 101-109, caregiver server 130, and/or other computing devices. In additional embodiments, all voice-based health identification data may be mirrored in additional cloud-based servers of service providers, edge network systems, and/or the like. In other additional embodiments, voice-based health detection server 120 may be hosted as one or more virtual servers within a cloud-based service and/or application.

In some embodiments, transmission of data associated with voice-based health recognition system 100 may occur over network 110 through one or more wired and/or wireless connections. For example, one or more of personal computing devices 101-109 may be wirelessly coupled to network 110 via a wireless network access point and/or any wireless devices. As in 1 As illustrated, the personal computing devices 101-109 can be any type of computing device capable of capturing audio data and used by any of the known users, including but not limited to a pair of intelligent hearing aids 101 such as Earphones, headphones, etc., a head-mounted display 102 such as virtual reality displays, etc., a game console 103, a wearable device 104, a computer tablet 105, a wearable device 106 such as smart watches, fitness watches, etc., a smart Remote control 107 such as voice controlled television remote controls, voice controlled garage remote controls, voice controlled remote device/appliance controls, etc., a smart speaker 108 such as voice controlled intelligent personal assistants, voice controlled speakers, etc., and a smart home device 109 such as voice controlled thermostat controls, voice controlled security monitoring devices, smart home appliances, voice controlled lighting control devices, etc.

In other embodiments, personal computing devices 101-109 may be any type of speech-based computing device. For example, the voice-based computing devices may include any type of portable handheld device, such as a mobile device, cell phone, mobile or cellular pad, computer tablet, personal digital assistant (PDA), any type of handheld device, any other desired voice-based Device and/or one or more widely used running software and/or mobile operating systems. The voice controlled computing devices can be personal computers and/or laptops running various operating systems. The voice controlled computing devices can be workstation computers running a variety of commercially available operating systems. Alternatively, the voice-enabled computing devices may be any other electronic device, such as a thin client computer, an internet-enabled gaming system with a messaging input device, and/or a personal voice-enabled messaging device that enables communication over the network 110 . Although in 1 As nine personal computing devices 101-109 are illustrated, the voice-based health recognition system 100 may use any number of computing devices and any type of computing device without limitation. Likewise, it goes without saying that all types of wired and/or wireless connections between all components of the speech-based health recognition system 100 can be used without restrictions, based on any desired combination of devices, connections, etc.

In various embodiments, the voice-based health recognition system 100 may be implemented to continuously receive and monitor voice-based health recognition system data, such as known user data, privacy data, threshold data, sensed data, and/or any other signal data from the known users via any number of personal computing devices 101-109, personal computers, personal listening computing devices, and/or personal mobile computing devices. In many embodiments, the voice-based health recognition system can process a variety of data related to keywords, vital sign data, baseline measurements, and voice characteristics of known users; determining whether the processed voice characteristics exceed predetermined thresholds, such as dynamic and/or static predetermined thresholds; and generate and transmit alert notifications to the known users and/or the attendant alert server 130 in response to the extracted voice characteristics meeting the predetermined thresholds have exceeded value. Furthermore, in some embodiments, the alert notifications may be generated from a list of predetermined exposures within speech-based health detection server 120, caregiver server 130, and/or personal computing devices 101-109.

In other embodiments, voice health identification system data may also be separated from personal identification data, such as personal medical history data, and transmitted to voice health identification server 120, caregiver server 130, data stores 140, 142, and/or other cloud-based services for processing and/or storage . The processed and/or stored data can then be transmitted back to the personal computing devices 101-109 for delivery to the known users. For example, the separated, processed, and stored data may be transferred using one or more forms of data transfer, such as blockchain-based data transfer, hash-based data transfer, encrypted data transfer, and/or other similarly protected data transfer techniques. In various embodiments, the caregiver server 130 may be implemented to receive (and/or transmit) data relating to the known users' vital signs and any associated known user alert data, including alert messages intended as early warning alerts for potential serious Health problems of known users are received. Caregiver server 130 may be, for example, a server, device, and/or system associated with physicians, nurses, and/or other primary caregivers who have a medical-patient relationship with the known users and are qualified to provide further diagnostics to create potential serious health problems.

Additionally, in some embodiments, voice-based health recognition server 120 may be implemented to run one or more voice-based health recognition services or software applications provided by one or more components of voice-based health recognition system 100 . The voice-based health detection services or software applications may include non-virtual and virtual health monitoring/detection environments. In some embodiments, these services may be offered to known users of personal computing devices 101-109 as web-based or cloud-based services, or under a Software as a Service (SaaS) model. In turn, the known users of the personal computing devices 101-109 may use one or more client/user applications to interact with the voice-based health recognition server 120 (and/or the caregiver server 130) and utilize the services offered by these servers.

The speech-based health detection server 120 can be implemented as a personalized computer, specialized server (including, by way of non-limiting example, personal computer (PC) servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server cluster and/or any other suitable desired configuration can be configured. The voice-based health detection server 120 may include one or more virtual machines running virtual operating systems and/or other computer architectures that include virtualization. One or more flexible pools of logical storage devices can be virtualized to manage virtual devices for speech-based capture device server 120 . Virtual networks can be controlled by the voice-based health detection server 120 over a software-defined (or cloud-based/defined) network. In various embodiments, the voice-based health detection server 120 may be configured to execute one or more instructions, programs, services, and/or software applications described herein. For example, the speech-based health detection server 120 may be associated with a server implemented to perform any of the below 4 , 5 and/or perform the processes described in Figures 6A-6B. The speech-based health detection server 120 may implement one or more additional server applications and/or mid-tier applications, including but not limited to Hypertext Transport Protocol (HTTP) servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, database servers and/or the like.

As in 1 As shown, voice-based health recognition system 100 may also include one or more data stores 140 and 142 . The data stores 140 and 142 can be in different locations. As a non-limiting example, one or more of data stores 140 and 142 may reside on a non-transitory storage medium that is local to (and/or resides in) voice-based health awareness server 120 . Alternatively, the data stores 140 and 142 may be remote from the voice-based health detection server 120 and via anyone communicate desired connections/configurations with the speech-based health recognition server 120. In some embodiments, data stores 140 and 142 may be one or more external medical data stores used to store data related to patient information, private information, and/or medical history of any of the known users. For example, the external medical data stores may be stored remotely from the voice-based health recognition server 120 and any of the personal computing devices 101-109.

With reference to 2A A high-level representation of a voice-based health recognition device 200 according to an embodiment of the disclosure is shown. In many embodiments, voice-based health recognition device 200 may include processor 210, memory 215 with voice-based health recognition application 220, input/output 230, and data storage 240. In the 2A The voice-based health recognition device 200 depicted in FIG 1 be similar to the voice-based health detection server 120 depicted. For example, voice-based health recognition device 200 may be implemented by voice-based health recognition system 100 in conjunction with other additional devices, servers, and/or systems, such as, but not limited to, one or more of personal computing devices 101-109 and the 1 illustrated caregiver server 130. In some embodiments, the voice-based health recognition device 200 may be any device that implements a voice-based health recognition system process such as those described in FIG 1 illustrated speech-based health recognition system process 100 can implement. As previously mentioned, the computing devices may be any of the personal computing devices 101-109 of 1 and/or include any computing device sufficient to receive, transmit and respond to voice-based health identification events from known users.

In various embodiments, voice-based health recognition device 200 may be communicative with one of personal computing devices 101-109 1 be coupled that are configured to monitor voice characteristics and identify voice changes. Such voice changes can be used by the speech-based health recognition device 200 to identify and recognize early signs of possible health problems of known users. In many embodiments, the voice-based health recognition device 200 can detect these early health issues by implementing one or more logic within a voice-based health recognition application 220 to receive audio data from the sensors, identify keywords from the received audio data, and extract voice characteristics from the identified keywords.

As in 2A As illustrated, memory 215 may include speech-based health detection application 220, which may also include vital monitoring logic 221, sample preprocessing logic 222, sample processing logic 223, keyword detector logic 224 (and/or sensor output detector logic), voice characteristics logic 225, vital processing logic 226, an alarm logic 227, a data protection logic 228 and/or a heuristic logic 229. Data store 240 may include captured data 241, privacy data 242, threshold data 243, signal data 244, and known user data 250.

In a number of embodiments, vital monitoring logic 221 may be configured to exchange data between voice-based health recognition device 200 and external computing devices, such as personal computing devices 101-109 of FIG 1 , external sensor/monitoring services, etc., receives and/or facilitates the transmission thereof. For example, the data received by the vital monitoring logic 221 may be stored as the collected data 241 in the data store 240 , where the collected data 241 may include any type of data that the vital monitoring logic 221 collected and received. In some embodiments, the vital monitoring logic 221 may be connected over a network connection similar to that shown in FIG 1 illustrated network 110, establish communication channels with the external devices. Certain embodiments may utilize the network connection tools provided by the operating system of the speech-based health recognition device 200.

The vital monitoring logic 221 may be configured to receive signal inputs from appropriate signal sources, such as a microphone, an audio data source, and/or a sensor. The microphone may include audio microphones, digital microphones, or other waveform detection devices. The audio data source may comprise any other type of processing data source capable of various inputs to receive/capture/provide input signals. The sensor can include any type of sensors and/or sensor-enabled devices, such as, but not limited to, vital sign monitoring sensors (e.g., sensors used to monitor heart rate, blood pressure, body temperature, oxygen saturation (SpO2), voice characteristics, etc ), medical sensors, fitness tracking sensors, infrared sensors, pressure sensors, temperature sensors, proximity sensors, motion sensors, fingerprint scanners, photo eye sensors, wireless signal antennas, accelerometers, gyroscopes, magnetometers, tilt sensors, humidity sensors, barometers, light sensors (such as ambient light sensors), color sensors , touch sensors, flow sensors, liquid level sensors, ultrasonic sensors, smoke, alcohol and/or gas sensors (ie sensors capable of detecting smoke/alcohol/gas in the human respiratory tract) and so on. For example, the signal input data received by the vital monitoring logic 221 via the microphone, the audio data source and/or the sensors can be stored as signal data 244 in the captured data 241 of the data store 240, the signal data 244 being any type of signal input data such as audio data, audio signal streams, audio waveform may include samples, etc.

In many embodiments, the sample pre-processing logic 222 may be configured in conjunction with the sample processing logic 223 to use the signal data 244 received from the vital monitoring logic 221 to receive, process, and transmit any data associated with the acquired data 241 . The sample pre-processing logic 222 may be configured to use the voice characteristics extracted from the pre-processed sensor data, such as the acquired data 241, to arrive at one or more actionable decisions through a neural network or the like. In many embodiments, the sample pre-processing logic 222 may be configured as a filter bank or the like that receives, for example, the acquired signal data 244, where the received data may be filtered and pre-processed by the sample pre-processing logic 222 based on the desired actionable decisions before such data is provided to the sample processing logic 223 . That is, in some embodiments, the sample preprocessing logic 222 may be configured as an enhancement filter or the like, which may be configured to suppress unwanted noise in a signal by selectively attenuating or enhancing certain components of the signal on a time-varying basis, and/or by suppressing unwanted noise in a signal by selectively attenuating or amplifying certain components of the signal on a time-varying basis. For example, sample preprocessing logic 222 may be configured as pulse-density modulation (PDM) decimation logic configured to decimate PDM audio samples from any of the signal input sources described herein to a baseband audio sample rate for use in speech-based health recognition device 200.

Sample processing logic 223 may be configured to receive any type of signal data, such as frequency elements or signal spectrum information in the form of Fourier transforms or similar frequency decompositions, where the received signal data may be processed for audio signal processing tasks such as audio enhancement, noise reduction, and/or the like. In many embodiments, sample processing logic 223 may perform the audio signal processing tasks and cooperate with keyword detector logic 224, which is configured to receive audio input data and then perform word recognition tasks, such as enhancing features from the received input data and so on. For example, as described herein, the keyword detector logic 224 can be sensor output detector logic configured to identify features, keywords, etc. from the received signal data 244, and then the sample processing logic 223 can be configured to extract keyword data ( and/or feature data, sensor output data, etc.) related to the identified keywords and compares the generated keyword data to the known user data 250, as further described below. In some embodiments, sample processing logic 223 may be used in conjunction with keyword detector logic 224 to transmit the identified keywords and generated/processed keyword data to voice characteristics logic 225 based on the result(s) obtained from the word recognition tasks performed aggregated by both and/or one or more of sample processing logic 223 and keyword detector logic 224. Additionally, as described in more detail below, the keyword detector logic 224 may access one or more data types within the in 2 B illustrated known user data 250 comprising one or more lists of keywords stored in keyword data 267, voice characteristics stored in vital data 263, and/or particular voice identification data of the particular known users stored in voice data 261 and/or personal information data 264 stored may include.

In various embodiments, the voice characteristics logic 225 may be configured to extract one or more voice characteristics from the processed keyword data. For example, the voice characteristics logic 225 may extract all of the voice characteristics associated with the monitored characters of the known users, where the extracted voice characteristics are contained in the threshold data 243 and the in 2 B illustrated vital basis data 263 can be stored. The voice characteristics extracted by voice characteristics logic 225 may be extracted from all audio signals captured by vital monitoring logic 221, with each voice characteristic corresponding to one or more particularly monitored vital signs of the known user. The extracted voice characteristics may include voice characteristics of the known user, such as voice slope, speed, range, weight, and timbre. In some embodiments, the voice characteristics logic 225 may be configured to transmit the extracted voice characteristics and any processed vital basis data to the vital processing logic 226, which may be configured to detect whether the extracted voice characteristics exceed their respective predetermined thresholds.

In many embodiments, the vital processing logic 226 may be configured to compare one or more extracted voice characteristics to known user vitals, such as those described in 2 B illustrated vital baseline data 263 stored in known user data 250. Vital processing logic 226 may also be configured to determine whether the one or more voice characteristics being processed exceed one or more predetermined thresholds, such as dynamic and static predetermined thresholds, described in more detail below. In various embodiments, the vital processing logic may utilize external factors detected by the heuristic logic 229 to facilitate processing of the extracted voice characteristics and/or generation of alert data 266 2 B with alarm logic 227, as described in more detail below.

In many embodiments, the alarm logic 227 can be configured to use the in 2 B alert data 266 is generated in response to one or more extracted voice characteristics exceeding the predetermined thresholds stored in threshold data 243, which threshold data 243 may be used to detect any trends of the extracted voice characteristics with respect to the dynamic and/or static predetermined thresholds determine. The alarm logic 227 can also be configured and transmit the generated alarm data to one or more devices. In many embodiments, the alert logic 227 may be configured to generate and transmit the alert data 266 associated with one or more generated and transmitted known user alerts. For example, the alert logic 227 may be configured to generate and transmit known user alerts when the extracted properties are determined to have exceeded their respective predetermined thresholds.

In some embodiments, the stored alert data 266 can be generated with the alert logic 227, which can then trigger one or more predetermined actions defined in the 2 B illustrated predetermined action data 262 are stored. The predetermined actions generated by the alert logic 227 may include at least one or more known user alerts and attendant alerts based on the predetermined action data 262 triggered. The known user alerts may include an early warning alert, a warning alert, and an emergency alert. The caregiver alerts may include a caregiver early warning alert, a caregiver warning alert, and a caregiver emergency alert. In many cases, the user and caregiver alerts generated by alerting logic 227 may include any type of alert intended to provide early detection of potentially serious health issues associated with known users.

In some embodiments, the privacy logic 228 may be configured to receive and transmit any privacy data 242, which may also include any medical history data such as that described in 2 B medical history data 265 illustrated. The privacy logic 228 can be used to transmit any private data 242 related to medical information that is private and associated with one of the known users. Privacy logic 228 can be configured to strip off certain privacy data 242 that is not permitted to be transmitted and/or configured to hash all privacy data 242, such as medical history data 265, via blockchain-based data transmission -based data transmission, an encryption-based data transmission and/or another similarly protected data transmission.

For example, in some embodiments, heuristic logic 229 may be configured to use vital monitoring logic 221 and/or other device to detect a variety of external factors that may provide additional data that may be used to improve vital processing logic 226 determinations. In some embodiments, one or more external factors associated with the known user may be used to provide insight into the captured data 241 in conjunction with the captured speech data 261, the vital base data 263, and/or the in 2 B illustrated known user 250 data. External factors can indicate, for example, that a known user is undergoing an exercise program at a certain time of the week that can naturally cause changes in the user's voice. These particular acute changes can then be used to gain further insights (or at least generate additional data points) when changes and/or thresholds in their speech data 261 and others in the 2A-2B illustrated data are processed. The external factors can also include any additional data related to the event and the physical location where the data was collected. Some external factors captured by heuristic logic 229 may include the Global Positioning System (GPS) coordinates of the known user's home, the location where the vital signs measurements were taken (e.g., during an activity in the outdoors, on vacation, at work, etc.), the time they were taken (e.g. late at night or early in the morning, etc.), the quality of the recording, the duration of the recording, etc.

2 B Figure 2 shows an abstract representation of known user data 250 in accordance with embodiments of the disclosure. As above with reference to 2A described, known user data 250 may reside in data store 240 and be unique to each known user associated with device 200 . The known user data in the 2A-2B are illustrated as being divided into sections and stored on the basis of each data type associated with the known user. The types of data that may be included in known user data 250 are illustrated below. Known user data 250 may include voice data 261, predetermined action data 262, vital base data 263, personal information data 264 with keyword data 267, medical history data 265, and alarm data 266. Although in 2 B Six data types 261-266 are illustrated, any number of data types may be used, and any one or more of the data types illustrated may be omitted, combined, etc., without restrictions. Additionally, the known user data 250 can be used for all known users and/or can likewise be used to store any desired data types associated with only one known user, with each of the known users having their own known user data 250 with any number of data types and all types of data stored in known user data 250 without limitation.

In many embodiments, voice data 261 may include all voice data associated with each known user, which may include distinguishing certain voice characteristics of each known user. For example, the speech data 261 may include the speech data of a user who has a speech disability, while the second user has no problems and the speech data associated with that second user is different from that of the one user. Speech data 261 may include raw audio data recorded with a microphone or other audio recording device during the speech-based health recognition device. This speech data 261 may include waveform data and may be formatted into any desired audio format depending on the application and/or computer resources. For example, limited memory resources may result in increased use of compression algorithms to reduce size, while computational resources may limit the amount of compression that can be performed on the fly. Speech data 261 can be stored in lossy or lossless formats. In some embodiments, voice data 261 may be processed before being stored or used elsewhere within the voice-based health recognition system. The pre-processing can include noise reduction, frequency equalization, normalization and/or compression. Such pre-processing can increase the amount of ancillary data that can be generated from the speech data 261.

In additional embodiments, the predetermined action data 262 may consist of one or more actions that are triggered based on the extracted features exceeding their predetermined thresholds. For example, the alarm logic 227 of the 2A configured to take one or more predetermined actions in response to generated alert data NEN within the predetermined action data 262 to trigger. The triggered actions may include at least one or more of the known user alerts and attendant alerts based on the triggered predetermined action data 262 associated with the respective known user, where each of the known users may have the same and/or different predetermined action data associated with based on the preferences of each of the known users. For example, a first known user may have data in the predetermined action data 262 that allows the device 200 to generate and transmit all alert data 266 to the mother of the first known user, while all other known users may indicate that all predetermined action data 262 may only be transferred to yourself. Furthermore, the known user alerts triggered may include an early warning alert, a warning alert, and an emergency alert. Likewise, the triggered caregiver alerts may include a caregiver early warning alert, a caregiver warning alert, and a caregiver emergency alert.

In many embodiments, the vital baseline data 263 may be any data relating to one or more vital signs stored for each of the known users by the vital monitoring logic 221 of the 2A be monitored. The vital baseline data 263 may include one or more vital signs for each of the known users, including but not limited to body temperature, heart rate, oxygen saturation (Sp02), blood pressure, voice characteristics, and so on. Some of these vital signs can be monitored by continuously monitoring basic vital signs such as blood pressure, heart rate, body temperature and SpO2. Some of these vital signs can be monitored by continuously monitoring basic vital signs such as blood pressure, heart rate, body temperature and SpO2. In certain embodiments, the vital baseline data 263 may include any data obtained from one or more of the 1 illustrated personal computing devices 101-109, such as portable device 106 and/or in 1 illustrated mobile device 104, are monitored, generated and/or received. As described above, wearable device 106 may be, for example, a smartwatch, but may also be used to track vital signs such as heart rate, blood pressure, and/or other desired vital sign data available through the device's operating system.

As described above, the basic vitals data 263 may be generated by each of the known users logging into a speech-based health recognition system (or similar) and determining the known users' vital sign data for each of the vital signs tracked and selected by the respective known user . The vital base data 263 can be used in combination with other data types within the known user data 250 and with the threshold data 243 and the collected data 241 such that the combination of this data and the one or more logic of the speech-based health recognition application 220 then continuously monitor all predetermined vital signs and the known users can provide feedback or alert data 266 once a combination of parameters has exceeded their predetermined thresholds. For example, the vital baseline data 263 may also include any data related to any of the predetermined thresholds, such as the dynamic thresholds, the static thresholds, and so on. In addition, the vital baseline data 263 may include any data relating to the voice characteristics resulting from the analysis of the 2A-2B illustrated data 241, signal data 244, voice data 261, and keyword data 267 have been extracted (or derived).

For example, vital baseline data 263 may include all of the voice characteristics identified by voice characteristics logic 225 of FIG 2A extracted from audio signals generated by vital monitoring logic 221 of 2A where each voice characteristic stored in the vital baseline data 263 corresponds to one or more particular monitored vital signs of the known user. The extracted voice characteristics may include voice characteristics of the known user, such as voice slope, speed, range, weight, and timbre. In some embodiments, the device 200 of FIG 2A configured to detect whether the extracted voice characteristics exceed their respective predetermined thresholds, data of such voice characteristics and thresholds being storable in vital base data 263 .

In a number of embodiments, personal information data 264 may also include keyword data 267 . Personal information data 264 may include any additional personal data that may be generated and associated with each of the known users. In some embodiments, personal information data 263 may include relevant personal account and contact data such as names, addresses, phone numbers, age, external factors metadata, associated personal computing devices, and so forth. For example, some or any personal account data associated with the known user, which can be used to gain insight into the recorded speech data 261, vital base data 263, known user data 250 and/or any recorded data 241 in the data store 240 of the 2A to obtain. For example, the user data may indicate that a user's birthday is and then used to gain further insight (or at least have an additional data point) when processing their voice data 261 and other subsequent data. The external factor metadata may include any additional data related to the event and the physical location where the data was collected. Some examples of external factor metadata may be collected using heuristic logic 229 and/or the like, where some of the examples may include the Global Positioning System (GPS) coordinates of the known user's home, the location where the vital sign measurements were taken (e.g. during an outdoor activity, on vacation, at work, etc.), the time they were taken (e.g. late at night or early in the morning, etc.), the quality of the recording, the duration of the recording, etc.

In addition, the keyword data 267 stored in the personal information data 264 can be personalized for each of the known users. For example, keyword data 267 may include any data relating to words, phrases, conversations, and/or the like associated with a particular known user. The voice-based health recognition device 200 of FIG 2A can be configured as a keyword spotter, for example. The features extracted from the decimated audio samples are one or more signals in a time domain, a frequency domain, or both the time and frequency domain that are characteristic of keywords and voice characteristics can be trained to be identified by one or more neural Networks of the voice-based health detection device 200 to be recognized. The keyword data 267 may include any data related to user-defined keywords that can be identified from any type of signals that the particular user wishes to recognize. The user-specific keyword data can be, for example, spoken keywords, certain voice characteristics of the spoken keywords, non-verbal acoustic signals such as certain noises, signals, and so on. In such an example, the respective user may have created and stored the user-specific keyword data in the keyword data 267 so that the speech-based health recognition device 200, in addition to other standard keywords already in the in 2A signal data 244 illustrated and stored, can recognize personalized words, phrases, etc. such as "hello", "good morning", "on", "off", "hotter" and "colder".

In some embodiments, medical history data 265 may include any data related to medical information and detected medical data items that are private and associated with each of the respective known users. Medical history data 265 may include any personal and/or private information specific to the known user, such as: B. Previous medical events such as surgeries, speech disorders, etc., current medications the known user is taking, and so on. In some embodiments, one or more data points from medical history data 265 may be used during vital processing logic 226 to determine the extracted voice characteristics against their predetermined thresholds, e.g. e.g., whether the particular known user already has an existing speech disability that needs to be addressed, etc. Additionally, as described above, the medical history data 265 may be stored on the speech-based health recognition device 200, in contrast to the data shown in FIG 1 illustrated data memories 140, 142, in which the medical history data 265 can be separated from all special private data that may not be transmitted and/or via the in 2A shown data protection logic 228 can be transmitted. For example, the medical history data 265 may be transmitted using blockchain-based data transmission, hash-based data transmission, encrypted data transmission, and/or other similarly protected data transmission.

In many embodiments, alert data 266 may include any data associated with one or more generated and transmitted alerts for each of the known users. For example, alert data 266 may include any known user alerts that were generated and transmitted with respect to determining that the extracted properties exceeded their respective predetermined thresholds. In some embodiments, the stored alarm data 266 can be combined with the alarm logic 227 of FIG 2A configured to generate alert data that triggers at least one or more known user alerts and attendant alerts in response to the predetermined action data 262 being triggered. The known user alerts may include an early warning alert, a warning alert, and an emergency alert. The caregiver alerts may include a caregiver early warning alert, a caregiver warning alert, and a caregiver emergency alert. In many embodiments, the user alerts and caregiver alerts stored in alert data 266 may include any type of alert notification that can be used for early detection of a potential serious health issue associated with the particular user.

Those skilled in the art will understand that the references herein to the 2A-2B Known User Data 250 illustrated is only a single representation of possible known user data. For example, various embodiments may include merging known user data 250 such that all voice data 261 is stored together, all predetermined action data 262 for all known user entries are stored together, etc. Furthermore, other methods of storing known user data 250 may also be used without limitation so that the known user data 250 can be stored externally while other aspects are stored locally. For example, known user data 250 may store voice data 261 externally, while other data types 262-266 may be stored locally to avoid exposing private data such as medical history data 265 and personal information data 264.

3 FIG. 12 illustrates a detailed block diagram of a voice-based health recognition system 120 used in a voice-based health recognition system 300, in accordance with the disclosure. This in 3 The speech-based health recognition system 300 depicted in the 1 voice-based health recognition system 100 depicted. In addition, the in 3 illustrated speech-based health detection server 120 in 1 illustrated speech-based health detection server 120 may be substantially similar.

Speech-based health recognition system 300 illustrates an example system for speech recognition and voice characteristics recognition using speech-based health recognition server 120. As shown in FIG 3 As shown, in many embodiments, the speech-based health detection server 120 may be configured to provide audio input samples 322 to one or more neural networks 324 , each of which may process the provided audio input samples 322 to generate the signal output data 326 . The construction and use of neural networks in this way is discussed in the jointly pending application U.S. Patent Application 16/701,860 , filed December 3, 2019, which is assigned to common assignee, the disclosure of which is incorporated herein in its entirety.

In some embodiments, the voice-based health recognition system 300 may include a user 302, a mobile device 104, a network 110, the voice-based health recognition server 120, and a caregiver server 130. The mobile device 104, the network 110, the voice-based health recognition server 120 and the care server 130 in 3 can the in 1 illustrated mobile device 104, network 110, voice-based health awareness server 120, and caregiver server 130. In some embodiments, user 302 may be a known user using mobile device 104, where mobile device 104 may be any type of device described herein.

In many embodiments, voice-based health recognition system 300 may use voice-based health recognition server 120 to receive audio data 304 captured from mobile device 104 . The speech-based health recognition server 120 may be configured to process the audio data 304 to recognize (or identify, generate, etc.) one or more audio input samples 322 that are provided to a neural network 324, such as a digital neural network or the like become. In addition, the speech-based health detection server 120 can be configured to use signal output data 326 that can be generated by the neural network 324 . Likewise, the speech-based health detection server 120 may be configured to generate alert data 306 based on the generated signal output data 326 in response to the audio data 304 received, if necessary. In addition, the speech-based health detection server 120 can be used to transmit data related to the alert data 306 generated to the caregiver server 130 .

In some implementations, the voice-based health detection server 120 can receive a set of audio input samples 322 . The server may receive data indicative of a time-frequency representation based on a series of audio input samples 322 . Computer system 320 may provide the time-frequency representation based on a set of audio input/waveform samples as input to a neural network. Computer system 320 may identify one or more keywords spoken by user 302 and provide the identified keywords to neural network 324 as audio input samples 322 .

In the illustrated example, the user 302 of the mobile device 104 can speak words and the mobile device 104 can record multi-channel audio that includes the speech (i.e., the spoken words), respectively. The mobile device 104 can transmit the recorded audio data signal 312 to the speech-based health recognition server 120 via the network 110 . The voice-based health recognition server 120 may receive the audio data 304 to obtain one or more audio input samples 322 . For example, the speech-based health detection server 120 can identify a series of audio input samples (or waveform samples) 322 from the audio data 304 that can occur within a time window of the audio data signal 304 . The speech-based health detection server 120 can provide the audio waveform samples 322 to the neural network 324 .

The neural network 324 can be configured and trained to act as an acoustic model. For example, the neural network 324 may indicate one or more probabilities, which may be implemented as time-frequency property representations corresponding to different speech units, where the time-frequency property representations may be output based on the audio input samples 322. In some embodiments, the neural network 324 can be configured to identify keywords from the received audio data and to extract voice characteristics from the identified keywords. The extracted voice characteristics may include user voice characteristics such as pitch, speed, range, weight, and timbre. The neural network 324 may also be configured to detect whether the extracted voice characteristics exceed a predetermined threshold and provide the signal output data 326 with the extracted characteristics detected having exceeded their predetermined threshold. The speech-based health detection server 120 can therefore use the signal output data 326 provided by the neural network 324 to send early warning alert data 306 to the user's 302 mobile device 104 for potential health problems when the detected vocal characteristics exceed the predetermined threshold.

With reference to 4 An example flowchart of a voice-based health recognition process 400 for generating known user data, consistent with embodiments of the disclosure, is now shown. The process 400 can be illustrated as a flow chart for personalizing and updating a voice-based health recognition system by generating known user data that can be provided to a known user to perform desired user-specific functions. The process 400 may be implemented with one or more computing devices and/or systems, including but not limited to the 1 illustrated speech-based health recognition system 100, in 2A illustrated speech-based health recognition device 200, which is in 3 illustrated speech-based health recognition system 300 and / or in the 1 and 3 Additionally, as described above in various embodiments, the process 400 may be implemented using one or more web-based applications and/or any other suitable software application. In some embodiments, the application(s) may be implemented as a cloud-based application and/or distributed as a standalone software application, as desired and without limitation.

At block 410, the process 400 may begin entering a user's custom keyword data. The entered user-specific keyword data allows the user or a customer to enter any targeting signals into the application. The custom keyword data can be any type of signals (or target signals) that the user wants to recognize. The user-specific keyword data can be, for example, spoken keywords, non-verbal acoustic signals such as certain noises, types of images, etc., which are detected by one or more sensors such as those in 1 illustrated personal computing devices 101-109 and/or the like. In an exemplary embodiment, the user can enter the desired keywords and the sensors can recognize the personalized keywords, such as, by way of non-limiting example, "on", "off", "warmer" and "colder", in addition to any other standard keywords already included in a keyword data store.

At block 420, the process 400 may generate vital basis data for the keyword data provided by the user. The vital baseline data generated may include one or more voice characteristics and baseline data related to the keywords specified by the user. As described herein, after entering the desired keywords, the user may also specify the one or more voice characteristics and base data based on the entered keywords. For example, a speech-based health recognizer can be used to continuously monitor and detect changes over time in keywords spoken by the user. This allows the device to extract one or more predetermined voice characteristics of the spoken keywords, for example to identify and determine trends and threshold measurements of the extracted characteristics over time. The extracted voice characteristics may include one or more voice characteristics extracted from audio data (or audio signals, samples, etc.) specific to the user, including but not limited to, voice pitch, vocal speed, vocal range, voice weight, voice timbre, etc. As such any predetermined voice changes of the extracted characteristics based on the vital base data can be used to provide early health warnings to the user and/or a caregiver of the user. For example, the process 400 may allow the user to log into the aforementioned application and create the appropriate properties and vitals related to the entered keywords. In many embodiments, the extracted properties can be checked (or verified) against the generated base data. In many embodiments, the extracted voice characteristics can then be analyzed to determine whether any of the extracted voice characteristics exceeded one or more predetermined thresholds. As described above, the vital baseline data may include the data of all extracted vocal characteristics determined over time. This data is used to trend and threshold the extracted vocal characteristics to track one or more of the user's vital signs. In many embodiments, the user's vitals baseline data may include one or more thresholds associated with the extracted characteristics corresponding to the monitored vital signs. For example, a first predetermined threshold may include a range of minimum and maximum data values for a vital sign being monitored for a user, where a particular data value may be generated for an extracted voice characteristic of the user and typically falls within that threshold when the user is healthy .

Conversely, in other embodiments, if the one or more extracted voice characteristics have exceeded their respective predetermined thresholds, one or more early sign health alert data signals may be generated and transmitted to the user, and/or the extracted voice characteristics may be further processed to produce one or determine multiple subsequent exposures (e.g., generate and transmit the early sign health alert signal to a primary caregiver of the user). The early sign health alert data signal can give the user an early warning of a potential serious health problem and prompt them for further diagnosis of the potential serious health problem, where the potential serious health problem corresponds to one or more of the vital signs that are generated for the user based on the Basic vital data are monitored.

At block 430, the process 400 may retrieve waveform data from a data store associated with the user. The signal data can consist of standard keywords or the like that can be recognized. At block 440, the process 400 may build a modified data store based on the combination of user-specified keyword data and the signal data. In some embodiments, the user-specified target signals may be labeled with appropriate, corresponding labels, while all other signals may be identified using a generic label, such as "Other." Thereafter, the modified data store can then be used to train a neural network implementation. At block 450, the process 400 may train a neural network based on the modified data store to recognize both the combination of the user-specified keyword data and the signal data in the modified (or updated) data store. It is conceivable that the implementation of the neural network is a software model of a neural network, like the one shown in 3 illustrated neural network 324. At block 460, the process 400 may generate known user data to train the neural network implementation. For example, the generated known user data from a language-based health recognition device are used to recognize the user-specified keywords in the modified data store.

At block 470, process 400 may optionally translate the generated known user data into a file format suitable for storage in memory of a personal computing device or other device. For example, the memory can be similar to that in 2A data storage 240 illustrated. In some embodiments, a programming file comprising the generated known user data may be provided to an end user upon purchase of the device. In some embodiments, the file can be programmed into one or more integrated circuits and/or logic that can be purchased by the end user. When the end user installs the file comprising the generated known user data into the device, either by means of the aforementioned programming file, circuitry and/or logic, the device can recognize the entered user-specific keywords in the modified data store. As those skilled in the art know, the training or implementation of the neural network is performed external to the device and the resulting known user data is stored in memory, allowing the device to play the audio signals even when offline (i.e. in the absence of a cloud or other network connection). can monitor.

Referring now to 5 An exemplary flowchart of a voice-based health recognition process 500 is shown, in accordance with embodiments of the disclosure. The process 500 can be illustrated as a flowchart to generate alert data for early detection of known users' health problems. The process 500 may be implemented with one or more computing devices and/or systems, including but not limited to the 1 illustrated speech-based health recognition system 100, in 2A illustrated speech-based health recognition device 200, which is in 3 illustrated speech-based health recognition system 300 and / or in the 1 and 3 In addition, as described above in various embodiments, the process 500 may be implemented using one or more web-based applications and/or any other suitable software application. In some embodiments, the application(s) may be implemented as a cloud-based application and/or distributed as a standalone software application, as desired and without limitation.

At block 510, the process 500 may receive audio signal data. For example, the received audio signal data may be provided in the form of raw analog audio signals, digital signal data and patterns representing particular sounds or the like, and/or any other recognizable signal input obtained from one or more sensors such as those described in 1 illustrated personal computing devices 101-109. The received audio signal data may be captured by a voice-based health recognition device or captured remotely and transmitted to the voice-based health recognition device for processing. At block 520, the process 500 may identify (or recognize) one or more keywords in the received audio signal data.

For example, the speech-based health recognition device can recognize the predetermined keywords in the received audio signal data. In many embodiments, the identified keywords are received from sounds, voices, or the like, recorded in the vicinity of the device.

At block 530, the process 500 may extract voice characteristics from the identified keywords. The extracted voice characteristics are then processed and matched against the vital baseline data associated with the known user. In particular, the extracted vocal characteristics may match the identified keywords. The extracted vocal characteristics may include one or more vocal characteristics generated by the known user and extracted from the identified keywords, including but not limited to voice pitch, voice speed, vocal range, voice weight, voice timbre, and/or the like . At block 540, process 500 may process one or more changes to the extracted voice characteristics. For example, the processed voice characteristics can be compared to known user data associated with a particular known user. In addition, the processed voice characteristics can be compared to the vital basis data associated with the known user. By completing this evaluation, the processed voice characteristics of the known user may allow identification of one or more changes specific to the known user can be fish. Pre-existing characteristics, such as, but not limited to, a speech impediment, local dialects, and/or accents, may be taken into account in this assessment versus a baseline specific to the known user.

At block 550, the process 500 may determine whether the extracted property(s) have exceeded a predetermined threshold. If the extracted voice characteristics have not exceeded their respective predetermined thresholds, the process 500 may proceed back to block 510. The extracted characteristics are then analyzed to determine, for example, whether any of the extracted voice characteristics exceeded their respective predetermined thresholds, also assuming one or more external factors that may affect the predetermined thresholds and/or the processed voice characteristics (e.g B. can the in 2A illustrated heuristic logic 229 can be used when such external factors are taken into account). Additionally, as described above, the predetermined threshold may include at least one or more dynamic/static predetermined thresholds. The predetermined threshold may comprise, for example, a static predetermined threshold that may be generated based on the known user's vitals baseline data. In some embodiments, the static predetermined threshold may include a static range of minimum and maximum data values associated with the known user's vitals baseline data. That is, the static predetermined threshold may have a minimum data value and a maximum data value for a vital sign, where both the minimum and maximum data values for the one vital sign are fixed, cannot be changed, and/or do not take other external factors into account.

In other embodiments, the predetermined threshold may include a dynamic threshold. The dynamic threshold may be generated based on a variety of changing factors, including but not limited to the known user's vitals. In addition, the dynamic threshold may include a dynamic range of minimum and maximum data values associated with the known user's vitals data. For example, the dynamic threshold may be used in conjunction with vital processing logic (e.g., vital processing logic 226 in 2A) are generated, which can be configured to dynamically adjust the dynamic range of the minimum and maximum data values based on one or more external factors controlled by heuristic logic (e.g., heuristic logic 229 in 2A) are recorded. In some embodiments, the one or more sensed external factors may include at least one or more of geographic location, time, date, real-time activity, physical location, ambient temperature, and/or monitored temperature. For example, on a hot day, the dynamic predetermined threshold may dynamically adjust the respective minimum and maximum data values for the monitored vital signs associated with the known user's temperature. It is conceivable that a variety of dynamic thresholds could be configured based on any number of external factors.

Accordingly, at block 560, if the extracted voice characteristics have exceeded their respective thresholds, the process 500 may generate alarm data related to the extracted voice characteristics having exceeded their predetermined thresholds. The generated alarm data may be configured, for example, as a data command, a function call, a predetermined action associated therewith, a voltage change within the device, and/or the like. At block 570, the process 500 may transmit the generated alert data to one or more devices. Additionally, in response to the generated alert data, the transmitted alert data may trigger predetermined action data associated with the known user, which may trigger at least one or more known user alerts and/or attendant alerts. For example, the known user alerts may include an early warning alert, a warning alert, and an emergency alert, while the caregiver alerts may include a caregiver early warning alert, a caregiver warning alert, and a caregiver emergency alert. Finally, the process 500 may be configured to, in response to the generated and transmitted alert data, transmit one of the user alerts to a personal computing device of the known user and/or one of the caregiver alerts to a caregiver server associated with the known user, where both the user alerts and caregiver alerts include early detection alert notification of a potential serious health issue associated with the known user.

Referring now to 6A FIG. 6 is an exemplary flowchart of an always-on speech-based health detection process 600, in accordance with embodiments of FIG revelation shown. The process 600 can be illustrated as a flow chart used to monitor data on the known user's device and generate alert data for early detection of health problems of known users. The process 600 may be implemented with one or more computing devices and/or systems, including but not limited to the 1 illustrated speech-based health recognition system 100, in 2A illustrated speech-based health recognition device 200, which is in 3 illustrated speech-based health recognition system 300 and / or in the 1 and 3 Additionally, as described above in various embodiments, the process 600 may be implemented using one or more web-based applications and/or any other suitable software application. In some embodiments, the application(s) may be partially or fully implemented as a cloud-based application and/or distributed as a standalone software application, as desired and without limitation.

At block 610, the process 600 may transition to a low power listening mode with constant monitoring. In many embodiments, the process 600 can be implemented with a voice-based health recognition device or the like. The speech-based health recognition device can, for example, 2A illustrated speech-based health recognition device 200 may be similar. In the following embodiments, the device can be used to enter the listening mode, which can be used by a sensor (or the like) and is in a low-power, always-on (or consuming) state in the device so that the sensor can be any type of low-latency audio data signals.

At block 620, process 600 may receive audio data from one or more sensors and/or personal computing devices of a user. For example, the device may receive one or more audio signal inputs in the form of raw analog audio signals, digital signal data and patterns representing particular sounds or the like, and/or other recognizable signal inputs captured from one or more audio data sources, sensors, and/or the like. The received audio signals can be captured within the device or captured remotely and transmitted to the device for processing. At block 630, the process 600 may identify predetermined keywords in the received audio data. For example, the device may recognize one or more user-supplied keywords in the received audio data. At block 640, process 600 may process the recognized keywords against known user data associated with the user. For example, the transmitted keywords can be processed against the known user data, similar to the known user data 250 described in Figs 2A-2B are shown. For example, in some embodiments, the one or more identified keywords are then processed and compared to the known user data of one or more known users. The known user data may include data associated with the one or more known users who have been pre-authorized to use the device. In addition to the pre-authorized known users, the known user data may also include keywords, attributes and/or other desired data based on known users, such data may be associated with the one or more data types in the in 2 B known user data 250 shown. In some embodiments, the known users can be determined by processing the recognized keywords, since a separate recognition network can be used to determine the sources of the known users. For example, the checking of the recognized keywords against the known users can be performed sequentially or in parallel with step 630.

At block 650, the process 600 may match the extracted voice characteristics to the baseline data of the known users. The extracted properties can then be processed, for example, and verified against the basic vital data of the known user data. In particular, as described herein, the extracted voice characteristics and the user's known vitals may match the recognized keywords. At block 660, the process 600 may determine whether the processed and extracted voice characteristics have exceeded a predetermined threshold. This determination may be, for example, similar to the determination at block 550 made above in 5 is illustrated. At block 670, the process 600 may generate alarm data in response to the extracted voice features exceeding their predetermined thresholds. At block 680, process 600 may transmit the generated alert data to one or more personal computing devices. For example, the transmitted alarm data can be transmitted in a recognizable form that can be used by one of the personal computing devices 101-109 1 can be received. At block 690, the process 600 can optionally transmit the generated alert data to a caregiver device, server, and/or system. For example, the transmitted alert data may be transmitted in a recognizable form that the supervisor server 130 of 1 can be received. The transmitted alert data received by the caregiver can alert them to early health detection of a possible serious health problem associated with the user.

With reference to 6B In accordance with the disclosure, an example flowchart of an always-on speech-based health detection process 601 using an external computing device is shown. The process 601 can be illustrated as a flowchart used to receive off-device data from the known user and to generate alert data for early detection of health problems of known users. the inside 6B Process 601 illustrated may correspond to that in 6A Illustrated process 600, except that each (and/or most) of the illustrated steps of process 601 may be implemented in a cloud-based server and/or the like. Process 601 may be implemented with one or more computing devices and/or systems, including but not limited to the 1 illustrated speech-based health recognition system 100, in 2A illustrated speech-based health recognition device 200, which is in 3 illustrated speech-based health recognition system 300 and / or in the 1 and 3 Additionally, as described above in various embodiments, the process 601 may be implemented using one or more web-based applications and/or any other suitable software application. In some embodiments, the application(s) may be partially or fully implemented as a cloud-based application and/or distributed as a standalone software application, as desired and without limitation.

At block 611, the process 601 may transition to a low-power, always-monitoring listening mode. This entered listening mode may be similar, for example, to the entered listening mode in block 610, described above in 6A is illustrated. At block 621, process 601 may receive audio data transmitted from one or more sensors and/or personal computing devices that may be located on a user. At block 625, the process 601 may then determine whether predetermined keywords were recognized in the received audio data. If no predetermined keywords are recognized, the process 601 may return to block 611. Conversely, if predetermined keywords are identified, process 601 may proceed to block 631 and process the identified keywords in the received audio data. It will be understood that the previous blocks correspond to the corresponding blocks set out above in 6A are illustrated may be similar.

At block 641, process 601 may transmit the processed keywords to one or more external devices. As described above, the external computing devices may be implemented as a cloud-based device, server, system, and/or the like. For example, the cloud-based external computing devices may be configured to receive recognized keywords from the one or more sensors and personal computing devices. At block 651, process 601 may match the transmitted keywords to known user data. For example, the transmitted keywords can be processed against the known user data, similar to the known user data 250 described in Figs 2A-2B are shown. At block 661, process 601 may process the extracted properties against known user baseline data. At block 665, process 601 may determine whether extracted properties have exceeded a predetermined threshold. If the extracted voice characteristics have not exceeded their respective predetermined thresholds, the process 601 may proceed to end the process. Conversely, if the extracted voice characteristics have exceeded their respective predetermined thresholds, the process 601 may proceed to block 671 and generate alarm data related to the extracted characteristics being exceeded. It will be understood that the previous blocks correspond to the corresponding blocks set out above in 6A are illustrated may be similar.

At block 681, process 601 may transmit generated alert data to the personal computing device. For example, the cloud-based device (or service, application, etc.) may transmit the generated alert data to a personal computing device associated with the user. At block 685, process 601 may process predetermined actions based on transmitted alert data. For example, the predetermined actions processed may include generating alarm notifications gene and/or triggering other desired actions associated with the user. At block 691, process 601 may transmit the processed alert data to a caregiver. It should be understood that this block corresponds to the corresponding block(s) set out above in 6A illustrated may be similar.

The information shown and detailed herein is fully capable of achieving the above-described object(s) of the present disclosure and the presently preferred embodiment of the present disclosure, and thus constitutes subject matter which is intended to be considered in the broadest sense of the present disclosure is considered. The scope of the present disclosure also includes other embodiments that may be apparent to those skilled in the art and accordingly is not limited by anything other than the appended claims. Any reference to an element in the singular shall not mean "one and only one" unless expressly so stated, but rather "one or more". All structural and functional equivalents to the elements of the preferred embodiment described above and additional embodiments deemed equivalent by those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims.

Furthermore, a system or method need not address each and every problem sought to be solved by the present disclosure in order for solutions to such problems to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be made public, regardless of whether the element, component, or method step is expressly recited in the claims. Various changes and modifications in form, material, work and details of manufacture can be made without departing from the spirit and scope of the present disclosure as set forth in the appended claims, and are as well encompassed by the present disclosure as would be apparent to those skilled in the art.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US63/124306 [0001]
• US63/034811 [0001]
• US 16701860B [0056]

Claims (28)

System for generating data for the early detection of health problems, comprising: one or more sensors; a processor communicatively coupled to the one or more sensors; and a memory communicatively coupled to the processor, the memory comprising: vital monitoring logic to receive signal data from the one or more sensors; sensor output detector logic configured to identify features from the received signal data; feature logic configured to extract one or more features from the identified features; vital processing logic configured to determine whether the one or the plurality of extracted features exceed a predetermined threshold; and an alarm logic configured to, in response to the one or more extracted features exceeding the predetermined threshold, generate alarm data. system after claim 1 , wherein the one or more sensors are configured to monitor one or more vital signs of a known user. system after claim 1 , wherein the signal data comprises audio data collected from an always-on device. system after claim 2 , wherein the identified features include at least one or more features associated with the known user, wherein the sensor output detector logic is further configured to recognize words associated with the known user, and wherein the identified features include at least one or more of include recognized words. system after claim 4 wherein the memory further comprises: sample processing logic configured to: generate feature data based on the identified features from the sensor output detector logic; and processing the feature data against known user data associated with the known user. system after claim 5 , wherein the one or more extracted features are extracted from the processed feature data of the known user. system after claim 6 , wherein the one or more extracted features include at least one or more of a voice pitch, a voice speed, a vocal range, a voice weight, and a voice timbre. system after claim 2 wherein the one or more sensors include at least one or more of wearable devices, smart hearing aids, head-mounted displays, game consoles, mobile computing devices, computer tablets, smart remote controls, voice-based speakers, and smart home devices. system after claim 7 wherein determining the extracted features further includes determining whether the extracted features exceed the predetermined threshold based on vital baseline data of the known user. system after claim 1 , where the system operates in a constant power saving mode so that the system is continuously ready to receive the signal data. system after claim 2 wherein the alert logic is further configured to trigger one or more actions associated with predetermined action data related to the known user in response to the generated alert data. system after claim 11 wherein the alert logic is further configured to trigger at least one or more of known user alerts and caregiver alerts in response to the triggered predetermined known user action data. system after claim 12 , wherein the known user alerts include an early warning alert, a warning alert, and an emergency alert. system after Claim 13 wherein the caregiver alerts comprise at least one of a caregiver early warning alert, a caregiver warning alert, and a caregiver emergency alert. system after Claim 14 wherein, in response to the generated alert data, the alert logic is further configured to: transmit one of the known user alerts to a personal computing device of the known user; and transmitting one of the caregiver alerts to one A caregiver server associated with the known user, wherein both known user and caregiver alerts include an early health detection alert notification of a potential serious health issue associated with the known user. system after claim 15 , wherein the storage further comprises data protection logic configured to protect the medical history data associated with the known user, and wherein the data protection logic is further configured to transmit the known user's protected medical history data to the care server and data in response to receive the transmitted medical history data from the care server via one or more forms of data transmission. system after Claim 16 , wherein the one or more forms of data transmission comprises any of the following: blockchain-based data transmission, hash-based data transmission, and encryption-based data transmission. system after claim 9 wherein the predetermined threshold comprises a static predetermined threshold, wherein the static predetermined threshold is generated based on the known user's vitals baseline data, and wherein the static predetermined threshold comprises a static range of minimum and maximum data values associated with the known user's vitals baseline data. system after claim 9 wherein the predetermined threshold comprises a dynamic predetermined threshold, wherein the dynamic predetermined threshold is generated based on the known user's vitals baseline data, and wherein the dynamic predetermined threshold comprises a dynamic range of minimum and maximum data values associated with the known user's vitals baseline data . system after claim 19 , wherein the vital processing logic is further configured to dynamically adjust the dynamic range of the minimum and maximum data values of the dynamic predetermined threshold based on one or more external factors sensed by heuristic logic, and wherein the one or more sensed external factors at least include one or more of geographic location, time, date, real-time activities, physical location, ambient temperature, and monitored temperature. Method for detecting voice changes to provide data for early detection of serious health problems, comprising: receiving signal data from one or more sensors associated with a known user; identifying features from the received signal data; extracting one or more features from the identified features; processing the one or more extracted features against vital baseline data from the known user; determining whether the one or more extracted features exceed a predetermined threshold; generating alarm data based on the one or more extracted features that have exceeded the predetermined threshold; and transmitting the generated alert data to a personal computing device of the known user. procedure after Claim 21 , further comprising: generating feature data based on the identified features; and processing the feature data against known user data associated with the known user, wherein identifying features from the received signal data further includes detecting words from the received signal data and wherein the identified features include at least one or more of the detected words. procedure after Claim 22 wherein the one or more extracted features are extracted from the processed features of the known user, and wherein the one or more extracted features include at least one or more of a voice pitch, a voice speed, a vocal range, a voice weight, and a voice timbre. procedure after Claim 21 wherein the one or more sensors include at least one or more of wearable devices, smart hearing aids, head-mounted displays, game consoles, mobile computing devices, computer tablets, smart remote controls, voice-based speakers, and smart home devices. procedure after Claim 23 , wherein determining the extracted features further includes determining whether the extracted features exceed the predetermined threshold based on the known user's vital data. A system for remotely generating data for the early detection of health problems, comprising: a processor; and a memory communicatively coupled to the processor, the memory comprising: sample processing logic configured to: receiving characteristics of signal data collected from one or more computing devices associated with a known user; generating feature data based on the identified features; and processing the generated feature data against known user data; feature logic configured to extract one or more features from the processed feature data; vital processing logic configured to: processing the one or more extracted features against known user base data; and determining whether the one or more processed features exceed a predetermined threshold; and an alarm logic configured to: generating alarm data in response to the one or more extracted features exceeding the predetermined threshold; and transmitting the generated alert data to the one or more computing devices associated with the known user. system after Claim 26 , further comprising privacy logic configured to receive and transmit any privacy data. system after Claim 27 wherein the privacy data includes medical history data, and wherein the privacy logic is further configured to transmit the privacy data related to the medical history data that is private and associated with the known user.

Images