WO2015168274A1 - Système et procédé pour l'évaluation du langage expressif, du trouble du développement et des émotions - Google Patents

Système et procédé pour l'évaluation du langage expressif, du trouble du développement et des émotions Download PDF

Info

Publication number
WO2015168274A1
WO2015168274A1 PCT/US2015/028278 US2015028278W WO2015168274A1 WO 2015168274 A1 WO2015168274 A1 WO 2015168274A1 US 2015028278 W US2015028278 W US 2015028278W WO 2015168274 A1 WO2015168274 A1 WO 2015168274A1
Authority
WO
WIPO (PCT)
Prior art keywords
child
segments
key child
model
segment
Prior art date
Application number
PCT/US2015/028278
Other languages
English (en)
Inventor
Dongxin D. Xu
Terrance D. Paul
Original Assignee
LENA, Foundation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/265,188 external-priority patent/US9355651B2/en
Application filed by LENA, Foundation filed Critical LENA, Foundation
Publication of WO2015168274A1 publication Critical patent/WO2015168274A1/fr

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00—Speaker identification or verification techniques
    • G10L17/26—Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Definitions

  • the present invention relates generally to automated language assessment and, specifically, to assessing a key child's expressive language development by analyzing phones, phone-like sounds, and protophones used by the child, to analyzing recordings to assist in the detection of diseases and disorders such as Autism, and to detecting emotion.
  • a child's language and vocabulary ability at age three can indicate intelligence and test scores in academic subjects such as reading and math at later ages. Improving language ability typically results in a higher intelligent quotient (IQ) as well as improved literacy and academic skills.
  • IQ intelligent quotient
  • Exposure to a rich aural or listening language environment in which many words are spoken with a large number of interactive conversational turns between the child and adult and a relatively high number of affirmations versus prohibitions may promote an increase in the child's language ability and IQ.
  • the effect of a language environment surrounding a child of a young age on the child's language ability and IQ may be particularly pronounced.
  • Counting the number of words and determining content spoken may be particularly time and resource intensive, even for electronic analysis systems, since each word is identified along with its meaning. Accordingly, a need exists for methods and systems for obtaining and analyzing data associated with a child's language environment independent of content and reporting metrics based on the data in a timely manner. The analysis should also include an automatic assessment of the child's expressive language development. [0009] Beyond an automatic assessment of a child's expressive language development, a need exists for the development of specific metrics and methodologies for determining specific developmental disorders in a child. As expressed above, a test that is largely non-invasive, in terms of providing a human observer, and that is of low cost while at the same time generating a substantial amount of data is desirable.
  • autism Another factor contributing to language development may be emotion. When children are exposed to an emotionally stressed environment there learning and language development may suffer. Therefore, a system and method for detecting the emotional content of subject interactions may be desirable for assisting in language development.
  • Certain embodiments of the system and method for expressive language development provide methods and systems for providing metrics associated with a key child's language environment and development in a relatively quick and cost effective manner.
  • the metrics may be used to promote improvement of the language environment, key child's language development, and/or to track development of the child's language skills.
  • a method is provided for generating metrics associated with the key child's language environment.
  • An audio recording from the language environment can be captured.
  • the audio recordings may be segmented into a plurality of segments.
  • a segment ID can be identified for each of the plurality of segments.
  • the segment ID may identify a source for audio in the segment of the recording.
  • Key child segments can be identified from the segments.
  • Each of the key child segments may have the key child as the segment ID.
  • Key child segment characteristics can be estimated based in part on at least one of the key child segments.
  • the key child segment characteristics can be estimated independent of content of the key child segments.
  • At least one metric associated with the language environment and/or language development may be determined using the key child segment characteristics. Examples of metrics include the number of words or vocalizations spoken by the key child in a pre-set time period and the number of
  • the at least one metric can be output to an output device.
  • adult segments can be identified from the segments. Each of the adult segments may have the adult as the segment ID.
  • Adult segment characteristics can be estimated based in part on at least one of the adult segments. The adult segment characteristics can be estimated independent of content of the adult segments. At least one metric associated with the language environment may be determined using the adult segment characteristics.
  • a system for providing metrics associated with a key child's language environment may include a recorder and a processor-based device.
  • the recorder may be adapted to capture audio recordings from the language environment and provide the audio recordings to a processor-based device.
  • the processor-based device may include an application having an audio engine adapted to segment the audio recording into segments and identify a segment ID for each of the segments. At least one of the segments may be associated with a key child segment ID.
  • the audio engine may be further adapted to estimate key child segment characteristics based in part on the at least one of the segments, determine at least one metric associated with the language environment or language development using the key child segment characteristics, and output the at least one metric to an output device.
  • the audio engine may estimate the key child segment characteristics independent of content of the segments.
  • the key child's vocalizations are analyzed to identify the number of occurrences of certain phones, phone-like sounds, and protophones and to calculate a frequency distribution or a duration distribution for the phones, phone-like sounds, and protophones.
  • the analysis may be performed independent of the content of the vocalizations.
  • a phone decoder designed for use with an automatic speech recognition system used to identify content from adult speech can be used to identify the phones, phone-like sounds, and protophones.
  • the key child's chronological age is used to select an age-based model which uses the distribution of the phones, phone-like sounds, and protophones, as well as age-based weights associated with each phone, phone-like sound, and protophone, to assess the key child's expressive language development.
  • the assessment can result in a standard score, an estimated developmental age, or an estimated mean length of utterance measure.
  • a method of assessing a key child's expressive language development includes processing an audio recording taken in the key child's language environment to identify segments of the recording that correspond to the key child's vocalizations. The method further includes applying an adult automatic speech recognition phone decoder to the segments to identify each occurrence of each of a plurality of bi-phone categories. Each of the bi-phone categories corresponds to a pre-defined speech sound sequence. The method also includes determining a distribution for the bi-phone categories and using the distribution in an age-based model to assess the key child's expressive language development.
  • a system for assessing a key child's language development includes a processor-based device comprising an application having an audio engine for processing an audio recording taken in the key child's language environment to identify segments of the recording that correspond to the key child's vocalizations.
  • the system also includes an adult automatic speech recognition phone decoder for processing the segments that correspond to the key child's vocalizations to identify each occurrence of each of a plurality of bi-phone categories. Each of the bi-phone categories corresponds to a predefined speech sound sequence.
  • the system further includes an expressive language assessment component for determining a distribution for the bi-phone categories and using the distributions in an age -based model to assess the key child's expressive language development.
  • the age-based model is selected based on the key child's chronological age, and the age-based model includes a weight associated with each of the bi-phone categories.
  • a method for detecting autism in a natural language environment includes using a microphone, sound recorder, and a computer programmed with software for the specialized purpose of processing recordings captured by the microphone and sound recorder
  • the computer is programmed to execute a method that includes segmenting an audio signal captured by the microphone and sound recorder combination using the computer programmed for the specialized purpose into a plurality of recording segments.
  • the method further includes determining which of the plurality of recording segments correspond to a key child.
  • the method also includes extracting acoustic parameters of the key child recordings and comparing the acoustic parameters of the key child recordings to known acoustic parameters for children.
  • the method returns a determination of a likelihood of autism.
  • a method for detecting autism includes transforming an audio recording to display an indication of autism on an output mechanism selected from the group consisting of a display, a printout, and an audio output, the transforming of the audio recording performed by comparing it to a model developed by analyzing the transparent parameters of a plurality of sound recordings captured in a natural language environment.
  • another embodiment includes a method for detecting a disorder in a natural language environment using a microphone, sound recorder, and a computer programmed with software for the specialized purpose of processing recordings captured by the microphone and sound recorder combination.
  • the computer is programmed to execute a method.
  • the method includes segmenting an audio signal captured by the microphone and sound recorder combination using the computer programmed for the specialized purpose into a plurality of recording segments; determining which of the plurality of recording segments correspond to a key subject; determining which of the plurality of recording segments that correspond to the key subject are classified as key subject recordings; extracting acoustic parameters of the key subject recordings; comparing the acoustic parameters of the key subject recordings to known acoustic parameters for subjects; and determining a likelihood of the disorder.
  • a method for detecting a disorder includes transforming an audio recording to display an indication of autism on an output mechanism selected from the group consisting of a display, a printout, and an audio output, the transforming of the audio recording performed by comparing it to a model developed by analyzing the transparent parameters of a plurality of sound recordings captured in a natural language environment.
  • the analyzing includes segmenting the sound recording into a plurality of recording segments, wherein the sound recording is captured by a microphone and sound recorder combination; determining which of the plurality of recording segments correspond to a key subject; determining which of the plurality of recording segments that correspond to the key subject are classified as key subject recordings; and extracting acoustic parameters of the key subject recordings.
  • a method of creating an automatic language characteristic recognition system includes receiving a plurality of audio recordings.
  • the audio recordings are segmented to create a plurality of audio segments for each audio recording.
  • the plurality of audio segments is clustered according to audio characteristics of each audio segment to form a plurality of audio segment clusters.
  • a method of decoding speech using an automatic language characteristic recognition system includes receiving a plurality of audio recordings and segmenting each of the plurality of audio recordings to create a first plurality of audio segments for each audio recording. The method further includes clustering each audio segment of the plurality of audio recordings according to audio characteristics of each audio segment to form a plurality of audio segment clusters. The method additionally includes receiving a new audio recording, segmenting the new audio recording to create a second plurality of audio segments for the new audio recording, and determining to which cluster of the plurality of audio segment clusters each segment of the second plurality of audio segments corresponds.
  • a method of determining the emotion of an utterance includes receiving the utterance at a processor-based device comprising an application having an audio engine. The method further includes extracting emotion-related acoustic features from the utterance. The method additionally includes comparing the emotion-related acoustic features to a plurality of models representative of emotions. Further included is selecting a model from the plurality of models based on the comparing and outputting the emotion corresponding to the selected model.
  • a method for detecting autism in a natural language environment using a microphone, sound recorder, and a computer programmed with software for the specialized purpose of processing recordings captured by the microphone and sound recorder combination includes segmenting an audio signal captured by the microphone and sound recorder combination using the computer programmed for the specialized purpose into a plurality recording segments.
  • the method further includes determining which of the plurality of recording segments correspond to a key child.
  • the method further includes determining which of the plurality of recording segments that correspond to the key child are classified as key child recordings.
  • the method includes extracting phone-based features of the key child recordings; comparing the phone-based features of the key child recordings to known phone-based features for children; and determining a likelihood of autism based on the comparing.
  • the comparing includes Logical Regression Analysis.
  • the comparing includes Linear Discriminate Analysis.
  • the method further includes transforming a display of a user to display the likelihood of autism.
  • the method further includes transforming an information storage device to store the likelihood of autism.
  • the phone-based features may be represented by a plurality of feature vectors.
  • the comparing may include that the plurality of feature vectors are compared to the known phone -based features for children to return a plurality of results, wherein there is a result of the plurality of results for each of the plurality of feature vectors, and the plurality of results are averaged for use in the determining. Additionally, the plurality of feature vectors may be averaged to a single feature vector for use in the comparing.
  • a method includes capturing an audio recording from a language environment of a key child and segmenting the audio recording into a plurality of segments.
  • the method further includes identifying a segment ID for each of the plurality of segments, the segment ID identifying a source for audio in the segment.
  • the identifying includes comparing the plurality of segments to a plurality of models, a model of the plurality of models includes a key child model and the identifying includes identifying a plurality of key child segments from the plurality of segments.
  • the method further includes estimating key child segment characteristics based in part on at least one of the plurality of key child segments.
  • the key child segment characteristics are estimated independent of content of the plurality of key child segments.
  • the content is the meaning of the plurality of key child segments.
  • the method further includes determining at least one metric associated with the language environment using the key child segment characteristics and outputting the at least one metric.
  • the plurality of models further include models for other children, male adults, female adults, noise, and TV noise.
  • the plurality of models further include an adult model that includes characteristics of sounds from an adult, an electronic device model that includes characteristics of sounds from an electronic device, a noise model that includes characteristics of sounds attributable to noise, an other child model that includes characteristics of sounds from a child other than the key child, a parentese model that includes complexity level speech criteria of adult sounds, an age-dependent key child model that includes characteristics of sounds of a key child of different ages, and a loudness/ clearness detection model that includes characteristics of sounds directed to a key child.
  • a maximum likelihood analysis is used to perform the segmenting and identifying.
  • a method includes capturing an audio recording from a language environment of a key child. The method further includes segmenting the audio recording and simultaneously and identifying a segment ID for each of a plurality of segments segmented from the audio recording, the segment ID identifying a source for audio in the segment, the identifying includes comparing the plurality of segments to a plurality of models. The method further includes determining at least one metric associated with the language environment based on the plurality of segments that have been identified and outputting the at least one metric. Optionally, a maximum likelihood analysis is used to perform the segmenting and identifying.
  • a model of the plurality of models includes a key child model and the identifying includes identifying a plurality of key child segments from the plurality of segments.
  • the determining includes using the key child segment characteristics to determine the at least one metric.
  • the method further includes estimating key child segment characteristics based in part on at least one of the plurality of key child segments, the key child segment characteristics are estimated independent of content of the plurality of key child segments, the content is the meaning of the plurality of key child segments.
  • MD-GMM Minimum Duration Gaussian Mixture Model
  • Figure 1 illustrates a key child's language environment according to one embodiment of the present invention
  • Figure 2a is a front view of a recorder in a pocket according to one embodiment of the present invention.
  • Figure 2b is a side view of the recorder and pocket of Figure 2a;
  • Figure 3 is a recording processing system according to one embodiment of the present invention.
  • Figure 4 is flow chart of a method for processing recordings according to one embodiment of the present invention
  • Figure 5 is a flow chart of a method for performing further recording processing according to one embodiment of the present invention
  • Figure 6 illustrates sound energy in a segment according to one embodiment of the present invention
  • Figures 7 - 12 are screen shots illustrating metrics provided to an output device according to one embodiment of the present invention.
  • Figure 13 illustrates the correlation between chronological age and certain phones
  • Figure 14 illustrates the non- linear relationship between some of the phones of Figure 13 and chronological age
  • Figures 15a and 15b collectively referred to herein as Figure 15, is a table illustrating the weights used for the expressive language index z-score according to one embodiment of the present invention
  • Figure 16 is a block diagram illustrating the system used to assess language development according to one embodiment of the present invention.
  • Figure 17 is a block diagram illustrating one embodiment of a method used to detect disorders or diseases in individuals
  • Figure 18 is a block diagram illustrating one embodiment of a method used to create trends for a population of normal individuals and individuals with the disorder or disease in question;
  • Figure 19 shows an acoustic signal captured and transformed into a graphical representation
  • Figure 20 shows a graphical representation of an empirical display for how the grouping of formant bandwidths can mark the articulation level
  • Figure 21 shows 12 acoustic parameters of vocal development
  • Figure 22 shows an example of non-acoustic parameters
  • Figure 23 shows a data set used in developing a model for the detection of autism
  • Figure 24 shows a trend chart for acoustic parameters in autistic and normally developing children
  • Figure 25 shows a trend chart for acoustic parameters in autistic, normally developing, and language delayed children
  • Figure 26 shows a trend chart for acoustic parameters in normally developing and language delayed children
  • Figure 27 shows non-acoustic parameters in normal and autistic children
  • Figure 28 shows a trend chart for acoustic parameters in autistic, normally developing, and language delayed children
  • Figure 29 shows a trend chart for acoustic parameters in normally developing and language delayed children
  • Figure 30 shows the results of the use of logistical regression analysis in determining normally developing and autistic individuals
  • Figure 31 shows tables showing the accuracy of a machine employing one embodiment of the system and methods of detecting autism
  • Figure 32 shows an illustration of K-means clusters
  • Figure 33 shows a methodology for determining emotion in an utterance
  • Figure 34 shows a graph of detection rate vs. posterior probability cutoff threshold for detection for combination of vectors before analysis
  • Figure 35 shows a graph of detection rate vs. posterior probability cutoff threshold for detection for analysis of vectors before combination.
  • Certain aspects and embodiments of the present invention are directed to systems and methods for monitoring and analyzing the language environment
  • a key child as used herein may be a child, an adult, such as an adult with developmental disabilities, or any individual whose language development is of interest.
  • a key child's language environment and language development can be monitored without placing artificial limitations on the key child's activities or requiring a third-party observer.
  • the language environment can be analyzed to identify words or other noises directed to or vocalized by the key child independent of content.
  • Content may include the meaning of vocalizations such as words and utterances.
  • the analysis can include the number of responses between the child and another, such as an adult (referred to herein as “conversational turns"), and the number of words spoken by the child and/or another, independent of content of the speech.
  • a language environment can include a natural language environment or other environments such as a clinical or research environment.
  • a natural language environment can include an area surrounding a key child during his or her normal daily activities and contain sources of sounds that may include the key child, other children, an adult, an electronic device, and background noise.
  • a clinical or research environment can include a controlled environment or location that contains pre-selected or natural sources of sounds.
  • the key child may wear an article of clothing that includes a recording device located in a pocket attached to or integrated with the article of clothing.
  • the recording device may be configured to record and store audio associated with the child's language environment for a predetermined amount of time.
  • the audio recordings can include noise, silence, the key child's spoken words or other sounds, words spoken by others, sounds from electronic devices such as televisions and radios, or any sound or words from any source.
  • the location of the recording device preferably allows it to record the key child's words and noises and conversational turns involving the key child without interfering in the key child's normal activities.
  • the audio recordings stored on the recording device can be analyzed independent of content to provide characteristics associated with the key child's language environment or language development.
  • the recordings may be analyzed to identify segments and assign a segment ID or a source for each audio segment using a Minimum Duration Gaussian Mixture Model (MD-GMM).
  • MD-GMM Minimum Duration Gaussian Mixture Model
  • Sources for each audio segment can include the key child, an adult, another child, an electronic device, or any person or object capable of producing sounds. Sources may also include general sources that are not associated with a particular person or device. Examples of such general sources include noise, silence, and overlapping sounds. In some embodiments, sources are identified by analyzing each audio segment using models of different types of sources. The models may include audio characteristics commonly associated with each source. In some embodiments, to detect the source type of audio signal, silence is detected. Any non-silent segment may still contain some short silence period, such as the pause involved in the explosive consonants like "p" and "t".
  • Such a short low energy region may not contain the information about the signal source type; thus, it will be removed from the likelihood calculation of a non-silence-segment.
  • Audio segments for which the key child or an adult is identified as the source may be further analyzed, such as by determining certain characteristics associated with the key child and/or adult, to provide metrics associated with the key child's language environment or language development.
  • the key child is a child between the ages of zero and four years old. Sounds generated by young children differ from adult speech in a number of respects.
  • the child may generate a meaningful sound that does not equate to a word; the formant transitions from a consonant to a vowel or visa-versa for child speech are less pronounced than the transitions for adult speech, and the child's speech changes over the age range of interest due to physical changes in the child's vocal tract. Differences between child and adult speech may be recognized and used to analyze child speech and to distinguish child speech from adult speech, such as in identifying the source for certain audio segments.
  • Certain embodiments of the present invention use a system that analyzes speech independent of content rather than a system that uses speech recognition to determine content. These embodiments greatly reduce the processing time of an audio file and require a system that is significantly less expensive than if a full speech recognition system were used.
  • speech recognition processing may be used to generate metrics of the key child's language environment and language development by analyzing vocalizations independent of content.
  • the recommended recording time is twelve hours with a minimum time of ten hours.
  • certain embodiments of the present invention are adapted to process a recording at or under half of real time. For example, the twelve-hour recording may be processed in less than six hours.
  • the recordings may be processed overnight so that results are available the next morning.
  • Other periods of recording time may be sufficient for generating metrics associated with the key child's language environment and/or language development depending upon the metrics of interest and/or the language environment.
  • a one- to two-hour recording time may be sufficient in some circumstances such as in a clinical or research environment. Processing for such recording times may be less than one hour.
  • a recording device may be used to capture, record, and store audio associated with the key child's language environment and language development.
  • the recording device may be any type of device adapted to capture and store audio and to be located in or around a child's language environment.
  • the recording device includes one or more microphones connected to a storage device and located in one or more rooms that the key child often occupies.
  • the recording device is located in an article of clothing worn by the child.
  • Figure 1 illustrates a key child, such as child 100, in a language environment 102 wearing an article of clothing 104 that includes a pocket 106.
  • the pocket 106 may include a recording device (not shown) that is adapted to record audio from the language environment 102.
  • the language environment 102 may be an area surrounding the child 100 that includes sources for audio (not shown), including one or more adults, other children, and/or electronic devices such as a television, a radio, a toy, background noise, or any other source that produces sounds. Examples of language environment 102 include a natural language environment and a clinical or research language environment.
  • the article of clothing 104 may be a vest over the child's 100 normal clothing, the child's 100 normal clothing, or any article of clothing commonly worn by the key child.
  • the recorder is placed at or near the center of the key child's chest.
  • the recording device in pocket 106 may be any device capable of recording audio associated with the child's language environment.
  • One example of a recording device is a digital recorder of the LENA system.
  • the digital recorder may be relatively small and lightweight and can be placed in pocket 106.
  • the pocket 106 can hold the recorder in place in an unobtrusive manner so that the recorder does not distract the key child, other children, and adults that interact with the key child.
  • Figures 2a and 2b illustrate one embodiment of a pocket 106 including a recorder 108.
  • the pocket 106 may be designed to keep the recorder 108 in place and to minimize acoustic interference.
  • the pocket 106 can include an inner area 110 formed by a main body 112 and an overlay 114 connected to the main body 112 via stitches 116 or another connecting mechanism.
  • the main body 112 can be part of the clothing or attached to the article of clothing 104 using stitches or otherwise.
  • a stretch layer 118 may be located in the inner area 110 and attached to the main body 112 and overlay 114 via stitches 116 or other connecting mechanism.
  • the recorder 108 can be located between the main body 112 and the stretch layer 118.
  • the stretch layer 118 may be made of a fabric adapted to stretch but provide a force against the recorder 108 to retain the recorder 108 in its position.
  • the stretch layer may be made from a blend of nylon and spandex, such as 85% nylon, 15% spandex, which helps to keep the recorder in place.
  • the overlay 114 may cover the stretch layer 118 and may include at least one opening where the microphone of recorder 108 is located.
  • the opening can be covered with a material that provides certain desired acoustic properties. In one embodiment, the material is 100% cotton.
  • the pocket 106 may also include snap connectors 120 by which the overlay 114 is opened and closed to install or remove the recorder 108.
  • snap connectors 120 by which the overlay 114 is opened and closed to install or remove the recorder 108.
  • At least one of the stitches 116 can be replaced with a zipper to provide access to the recorder 108 in addition or alternative to using snap connectors 120.
  • the pocket 106 may include multiple openings that correspond to the placement of the microphones recorder 108.
  • the particular dimensions of the pocket 106 may change as the design of the recorder 108 changes or as the number or type of microphones change.
  • the pocket 106 positions the microphone relative to the key child's mouth to provide certain acoustical properties and secure the microphone (and optionally the recorder 108) in a manner that does not result in friction noises.
  • the recorder 108 can be turned on and thereafter record audio, including speech by the key child, other children, and adults, as well as other types of sounds that the child encounters, including television, toys, environmental noises, etc.
  • the audio may be stored in the recorder 108.
  • the recorder can be periodically removed from pocket 106 and the stored audio can be analyzed.
  • Methods for analyzing audio recordings from a recorder may be implemented on a variety of different systems.
  • An example of one such system is illustrated in Figure 3.
  • the system includes the recorder 108 connected to a processor-based device 200 that includes a processor 202 and a computer-readable medium, such as memory 204.
  • the recorder 108 may be connected to the processor-based device 200 via wireline or wirelessly. In some embodiments, the recorder 108 is connected to the device 200 via a USB cable.
  • the device 200 may be any type of processor-based device, examples of which include a computer and a server.
  • Memory 204 may be adapted to store computer-executable code and data.
  • Computer- executable code may include an application 206, such as a data analysis application that can be used to view, generate, and output data analysis.
  • the application 206 may include an audio engine 208 that, as described in more detail below, may be adapted to perform methods according to various embodiments of the present invention to analyze audio recordings and generate metrics associated therewith.
  • the audio engine 208 may be a separate application that is executable separate from, and optionally concurrent with, application 206.
  • Memory 204 may also include a data storage 210 that is adapted to store data generated by the application 206 or audio engine 208, or input by a user.
  • data storage 210 may be separate from device 200 but connected to the device 200 via wireline or wireless connection.
  • the device 200 may be in communication with an input device 212 and an output device 214.
  • the input device 212 may be adapted to receive user input and communicate the user input to the device 200. Examples of input device 212 include a keyboard, mouse, scanner, and network connection. User inputs can include commands that cause the processor 202 to execute various functions associated with the application 206 or the audio engine 208.
  • the output device 214 may be adapted to provide data or visual output from the application 206 or the audio engine 208. In some embodiments, the output device 214 can display a graphical user interface (GUI) that includes one or more selectable buttons that are associated with various functions provided by the application 206 or the audio engine 208. Examples of output device 214 include a monitor, network connection, and printer.
  • the input device 212 may be used to setup or otherwise configure audio engine 208. For example, the age of the key child and other information associated with the key child's learning environment may be provided to the audio engine 208 and stored in local storage 210 during a setup or configuration.
  • the audio file stored on the recorder 108 may be uploaded to the device 200 and stored in local storage 210.
  • the audio file is uploaded in a proprietary format which prevents the playback of the speech from the device 200 or access to content of the speech, thereby promoting identity protection of the speakers.
  • the audio file is uploaded without being encoded to allow for storage in local storage 210 and playback of the file or portions of the file.
  • the processor-based device 200 is a web server, and the input device 212 and output device 214 are combined to form a computer system that sends data to and receives data from the device 200 via a network connection.
  • the input device 212 and output device 214 may be used to access the application 206 and audio engine 208 remotely and cause it to perform various functions according to various embodiments of the present invention.
  • the recorder 108 may be connected to the input device 212 and output device 214, and the audio files stored on the recorder 108 may be uploaded to the device 200 over a network such as an internet or intranet where the audio files are processed and metrics are provided to the output device 214.
  • the audio files received from a remote input device 212 and output device 214 may be stored in local storage 210 and subsequently accessed for research purposes such as on a child's learning environment or otherwise.
  • the audio file may be compressed.
  • a DVI-4 ADPCM compression scheme is used. If a compression scheme is used, then the file is decompressed after it is uploaded to the device 200 to a normal linear PCM audio format.
  • Figure 4 illustrates one embodiment of a method for analyzing and providing metrics based on the audio recordings from a key child's language environment. For purposes of illustration only, the elements of this method are described with reference to the system depicted in Figure 3. Other system implementations of the method are possible.
  • the audio engine 208 divides the recording into one or more audio segments and identifies a segment ID or source for each of the audio segments from the recording received from the recorder 108.
  • This process is referred to herein as "segmentation" and "segment ID”.
  • An audio segment may be a portion of the recording having a certain duration and including acoustic features associated with the child's language environment during the duration.
  • the recording may include a number of audio segments, each associated with a segment ID or source.
  • Sources may be an individual or device that produces the sounds within the audio segment.
  • an audio segment may include the sounds produced by the key child, who is identified as the source for that audio segment.
  • Sources also can include other children, adults, electronic devices, noise, overlapped sounds, and silence.
  • Electronic devices may include televisions, radios, telephones, toys, and any device that provides recorded or simulated sounds such as human speech.
  • Sources associated with each of the audio segments may be identified to assist in further classifying and analyzing the recording.
  • Some metrics provided by some embodiments of the present invention include data regarding certain sources and disregard data from other sources. For example, audio segments associated with live speech directed to the key child can be distinguished from audio segments associated with electronic devices, since live speech has been shown to be a better indicator and better promoter of a child's language development than exposure to speech from electronic devices.
  • a number of models may be used that correspond to the key child, other children, male adult, female adult, noise, TV noise, silence, and overlap.
  • Alternative embodiments may use more, fewer, or different models to perform segmentation and identify a corresponding segment ID.
  • One such technique performs segmentation and segment ID separately.
  • Another technique performs segmentation and identifies a segment ID for each segment concurrently.
  • HMM Hidden Markov Model
  • a number of HMM models may be provided, each corresponding to one source.
  • the result of the model may be a sequence of sources with a likelihood score associated with each source. The optimal sequence may be searched using a Viterbi algorithm or dynamic programming and the "best" source identified for each segment based on the score.
  • this approach may be complex for some segments in part because it uses transition probabilities from one segment to another - i.e., the transition between each segment. Transition probabilities are related to duration modeling of each source. HMM duration models may have discrete geometric distribution or continuous exponential distribution, which may not be appropriate for the sound sources of concern. Most recordings may include segments of having a high degree of variation in their duration. Although the HMM model may be used in some embodiments of the present invention, alternative techniques may be used to perform segmentation and segment ID.
  • An alternative technique used in some embodiments of the present invention to perform segmentation and segment ID is a Minimum Duration Gaussian Mixture Model (MD-GMM).
  • MD-GMM Minimum Duration Gaussian Mixture Model
  • Each model of the MD-GMM may include criteria or characteristics associated with sounds from different sources.
  • models of the MD-GMM include a key child model that includes characteristics of sounds from a key child, an adult model that includes characteristics of sounds from an adult, an electronic device model that includes characteristics of sounds from an electronic device, a noise model that includes characteristics of sounds attributable to noise, an other child model that includes
  • the adult model may include an adult male model that includes characteristics of sounds of an adult male and an adult female model that includes characteristics of sounds of an adult female.
  • the models may be used to determine the source of sound in each segment by comparing the sound in each segment to criteria of each model and determining if a match of a pre-set accuracy exists for one or more of the models.
  • the MD-GMM technique begins when a recording is converted to a sequence of frames or segments.
  • Segments having a duration of 2*D, where D is a minimum duration constraint are identified using a maximum log-likelihood algorithm for each type of source. The maximum score for each segment is identified. The source associated with the maximum score is correlated to the segment for each identified segment.
  • the audio engine 208 may process recordings using the maximum likelihood MD-GMM to perform segmentation and segment ID.
  • the audio engine 208 may search all possible segment sequences under a minimum duration constraint to identify the segment sequence with maximum likelihood.
  • One possible advantage of MD-GMM is that any segment longer than twice the minimum duration (2*D) could be equivalently broken down into several segments with a duration between the minimum duration (D) and two times the minimum duration (2*D), such that the maximum likelihood search process ignores all segments longer than 2*D. This can reduce the search space and processing time.
  • the following is an explanation of one implementation of using maximum likelihood MD-GMM. Other implementations are also possible: 1.
  • Acoustic Feature Extraction - the audio stream is converted to a stream of feature vectors ⁇ Xj, X 2 ....X T ⁇ Xi ⁇ R" ⁇ using a feature extraction algorithm, such as the MFCC (mel-frequency cepstrum coefficients).
  • MFCC mel-frequency cepstrum coefficients
  • Lcs log(f c (Xi)) where f c (X is the likelihood of frame being in class c.
  • searching variable S(c,b,n) represents the maximum log-likelihood for the segment sequence up to the frame b-1 plus the log- likelihood of the segment from frame b to frame n being in class c.
  • S(c,b,n) S(c,b,n-1) + log(f c (X n ), V b,c,n-b ⁇ 2 D c , i.e., the current score at frame n could be derived from the previous score at frame n-1.
  • the searching variable for segments less than twice the minimum duration is retained.
  • step 4 Iterate step 4 to step 6 until the last frame T.
  • Additional processing may be performed to further refine identification of segments associated with the key child or an adult as sources.
  • the language environment can include a variety of sources that may be identified initially as the key child or an adult when the source is actually a different person or device. For example, sounds from a child other than the key child may be initially identified as sounds from the key child. Sounds from an electronic device may be confused with live speech from an adult. Furthermore, some adult sounds may be detected that are directed to another person other than the key child. Certain embodiments of the present invention may implement methods for further processing and refining the segmentation and segment ID to decrease or eliminate inaccurate source identifications and to identify adult speech directed to the key child.
  • Figure 5 illustrates one embodiment of an adaptation method for further processing the recording by modifying models associated with the MD-GMM subsequent to an initial MD-GMM.
  • the audio engine 208 processes the recording using a first MD-GMM.
  • the recording is processed in accordance with the MD-GMM described above to perform an initial segmentation and segment ID.
  • the audio engine 208 modifies at least one model of the MD-GMM.
  • the audio engine 208 may automatically select one or more models of the MD- GMM to modify based on pre-set steps. In some embodiments, if the audio engine 208 detects certain types of segments that may require further scrutiny, it selects the model of the MD-GMM that is most related to the types of segments detected to modify (or for
  • Any model associated with the MD-GMM may be modified.
  • models that may be modified include the key child model with an age-dependent key child model, an electronic device model, a loudness/clearness model that may further modify the key child model and/or the adult model, and a parentese model that may further modify the key child model and/or the adult model.
  • the audio engine 208 processes the recordings again using the modified models of the MD-GMM.
  • the second process may result in a different segmentation and/or segment ID based on the modified models, providing a more accurate identification of the source associated with each segment.
  • the audio engine 208 determines if additional model modification is needed. In some embodiments, the audio engine 208 analyzes the new segmentation and/or segment ID to determine if any segments or groups of segments require additional scrutiny. In some embodiments, the audio engine 208 accesses data associated with the language environment in data storage 210 and uses it to determine if additional model modification is necessary, such as a modification of the key child model based on the current age of the child. If additional model modification is needed, the process returns to block 404 for additional MD-GMM model modification. If no additional model modification is needed, the process proceeds to block 410 to analyze segment sound.
  • additional model modification is necessary, such as a modification of the key child model based on the current age of the child. If additional model modification is needed, the process returns to block 404 for additional MD-GMM model modification. If no additional model modification is needed, the process proceeds to block 410 to analyze segment sound.
  • the audio engine 208 may implement an age-dependent key child model concurrently with, or subsequent to, the initial MD-GMM to modify the key child model of the MD-GMM to more accurately identify segments in which other children are the source from segments in which the key child is the source.
  • the MD-GMM may be modified to implement an age- dependent key child model during the initial or a subsequent segmentation and segment ID.
  • the key child model can be age dependent since the audio characteristics of the vocalizations, including utterances and other sounds, of a key child change dramatically over the time that the recorder 108 may be used.
  • the use of two separate models within the MD-GMM one for the key child and one for other children, may identify the speech of the key child
  • the use of an age-dependent key child model further helps to reduce the confusion between speech of the key child and speech of the other children.
  • the age-dependent key child models are: 1) less than one-year old, 2) one-year old, 3) two-years old, and 4) three-years old.
  • Alternative embodiments may use other age groupings and/or may use groupings of different age groups. For example, other embodiments could use monthly age groups or a combination of monthly and yearly age groups.
  • Each of the models includes characteristics associated with sounds commonly identified with children of the age group.
  • the age of the key child is provided to device 200 via input device 212 during a set-up or configuration.
  • the audio engine 208 receives the age of the key child and selects one or more of the key child models based on the age of the key child. For example, if the key child is one year and ten months old, the audio engine 208 may select key child model 2 (one-year-old model) and key child model 3 (two-years-old model) or only key child model 2 based on the age of the key child.
  • the audio engine 208 may implement the selected key child model or models by modifying the MD-GMM models to perform the initial or a subsequent segmentation and segment ID.
  • any segments including sounds, such as words or speech, generated electronically by an electronic device can be identified as such, as opposed to an inaccurate identification as live speech produced by an adult.
  • Electronic devices can include a television, radio, telephone, audio system, toy, or any electronic device that produces recordings or simulated human speech.
  • the audio engine 208 may modify an electronic device model in the MD-GMM to more accurately identify segments from an electronic device source and separate them from segments from a live adult without the need to determine the content of the segments and without the need to limit the environment of the speaker (e.g., requiring the removal of or inactivation of the electronic devices from the language environment).
  • the audio engine 208 may be adapted to modify and use the modified electronic device model concurrently with, or subsequent to, the initial MD-GMM process.
  • the electronic device model can be implemented after a first MD- GMM process is performed and used to adapt the MD-GMM for additional determinations using the MD-GMM for the same recording.
  • the audio engine 208 can examine segments segmented using a first MD-GMM to further identify reliable electronic segments. Reliable electronic segments may be segments that are more likely associated with a source that is an electronic device and include certain criteria. For example, the audio engine 208 can determine if one or more segments include criteria commonly associated with sounds from electronic devices.
  • the criteria includes (1) a segment that is longer than a predetermined period or is louder than a predetermined threshold; or (2) a series of segments having a pre-set source pattern.
  • An example of one predetermined period is five seconds.
  • An example of one pre-set source pattern can include the following:
  • the reliable electronic device segments can be used to adapt the MD- GMM to include an adaptive electronic device model for further processing.
  • the audio engine 208 may use a regular K-means algorithm as an initial model and tune it with an expectation-maximization (EM) algorithm.
  • EM expectation-maximization
  • the number of Gaussians in the adaptive electronic device model may be proportional to the amount of feedback electronic device data and not exceed an upper limit. In one embodiment, the upper limit is 128.
  • the audio engine 208 may perform the MD-GMM again by applying the adaptive electronic device model to each frame of the sequence to determine a new adaptive electronic device log-likelihood score for frames associated with a source that is an electronic device.
  • the new score may be compared with previously stored log-likelihood scores for those frames.
  • the audio engine 208 may select the larger log-likelihood score based on the comparison.
  • the larger log-likelihood score may be used to determine the segment ID for those frames.
  • the MD-GMM modification using the adaptive electronic device model may be applied using a pre-set number of consecutive equal length adaptation windows moving over all frames.
  • the recording signal may be divided into overlapping frames having a pre-set length.
  • An example of frame length according to one embodiment of the present invention is 25.6 milliseconds with a 10 millisecond shift resulting in 15.6 milliseconds of frame overlap.
  • the adaptive electronic device model may use local data obtained using the pre-set number of adaptation windows.
  • An adaptation window size of 30 minutes may be used in some embodiments of the present invention.
  • An example of one pre-set number of consecutive equal length adaptation windows is three. In some embodiments, adaptation window movement does not overlap.
  • the frames within each adaptation window may be analyzed to extract a vector of features for later use in statistical analysis, modeling, and classification algorithms.
  • the adaptive electronic device model may be repeated to further modify the MD-GMM process. For example, the process may be repeated three times. Loudness/Clearness Detection Model
  • some embodiments of the present invention use frame level near/far detection or loudness/clearness detection model. Loudness/clearness detection models can be performed using a Likelihood Ratio Test (LRT) after an initial MD-GMM process is performed. At the frame level, the LRT is used to identify and discard frames that could confuse the LRT.
  • LRT Likelihood Ratio Test
  • the likelihood for each model is calculated.
  • the difference between the most probable model likelihood and the likelihood for silence is calculated and the difference is compared to a predetermined threshold. Based on the comparison, the frame is either dropped or used for segment ID. For example, if the difference meets or exceeds the predetermined threshold, then the frame is used; but if the difference is less than the predetermined threshold, then the frame is dropped.
  • frames are weighted according to the LRT.
  • the audio engine 208 can use the LRT to identify segments directed to the key child. For example, the audio engine 208 can determine whether adult speech is directed to the key child or to someone else by determining the loudness/clearness of the adult speech or sounds associated with the segments. Once segmentation and segment ID are performed, segment-level near/far detection is performed using the LRT in a manner similar to that used at the frame level. For each segment, the likelihood for each model is calculated. The difference between the most probable model likelihood and the likelihood for silence is calculated and the difference is compared to a predetermined threshold. Based on the comparison, the segment is either dropped or processed further.
  • the audio engine 208 may modify the key child model and/or adult model to identify segments including parentese and associate the segments with an adult source. For example, the models may be modified to allow the audio engine 208 to examine the complexity of the speech included in the segments to identify parentese. Since the complexity of adult speech is typically much higher than child speech, the source for segments including relatively complex speech may be identified as an adult.
  • Speech may be complex if the formant structures are well formed, the articulation levels are good, and the vocalizations are of sufficient duration - consistent with speech commonly provided by adults. Speech from a child may include formant structures that are less clear and developed and vocalizations that are typically of a lesser duration.
  • the audio engine 208 can analyze formant frequencies to identify segments including parentese. When an adult uses parentese, the formant frequencies of the segment typically do not change. Sources for segments including such identified parentese can be determined to be an adult.
  • the MD-GMM models may be further modified and the recording further processed for a pre-set number of iterations or until the audio engine 208 determines that the segment IDs have been determined with an acceptable level of confidence.
  • the identified segment can be further analyzed to extract characteristics associated with the language environment of the key child. Child Vocalization, Cry, Vegetative-sound/Fixed-Signal Detection (classification)
  • the audio engine 208 may classify key child audio segments into one or more categories.
  • the audio engine 208 analyzes each segment for which the key child is identified as the source and determines a category based on the sound in each segment.
  • the categories can include vocalizations, cries, vegetative-sound, and fixed-signal sounds.
  • Vocalizations can include words, phrases, marginal syllables, including rudimentary consonant-vowel sequences, utterances, phonemes, sequence phonemes, phoneme-like sounds, protophones, lip-trilling sounds commonly called raspberries, canonical syllables, repetitive babbles, pitch variations, or any meaningful sounds which contribute to the language development of the child, indicate at least an attempt by the child to communicate verbally, or explore the capability to create sounds.
  • Vegetative-sound includes non- vocal sounds related to respiration and digestion, such as coughing, sneezing, and burping. Fixed-signal sounds are related to voluntary reactions to the environment and include laughing, moaning, sighing, and lip smacking.
  • Cries are a type of fixed-signal sound, but are detected separately since cries can be a means of communication.
  • the audio engine 208 may classify key child audio segments using rule-based analysis and/or statistical processing.
  • Rule-based analysis can include analyzing each key child segment using one or more rules.
  • the audio engine 208 may analyze energy levels or energy level transitions of segments.
  • An example of a rule based on a pre-set duration is segments including a burst of energy at or above the pre-set duration are identified as a cry or scream and not a vocalization, but segments including bursts of energy less than the pre-set duration are classified as a vocalization.
  • An example of one preset duration is three seconds based on characteristics commonly associated with vocalizations and cries.
  • Figure 6 illustrates energy levels of sound in a segment associated with the key child and showing a series of consonant (/b/) and vowel (/a/) sequences. Using a pre-set duration of three seconds, the bursts of energy indicate a vocalization since they are less than three seconds.
  • a second rule may be classifying segments as vocalizations that include formant transitions from consonant to vowel or vice versa.
  • Figure 6 illustrates formant transitions from consonant lb I to vowel /a/ and then back to consonant lb I, indicative of canonical syllables and, thus, vocalizations. Segments that do not include such transitions may be further processed to determine a classification.
  • a third rule may be classifying segments as vocalizations if the formant bandwidth is narrower than a pre-set bandwidth.
  • the pre-set bandwidth is 1000 Hz based on common bandwidths associated with vocalizations.
  • a fourth rule may be classifying segments that include a burst of energy having a first spectral peak above a pre-set threshold as a cry.
  • the pre-set threshold is 1500 Hz based on characteristics common in cries.
  • a fifth rule may be determining a slope of a spectral tilt and comparing it to pre-set thresholds.
  • vocalizations include more energy in lower frequencies, such as 300 to 3000 Hz, than higher frequencies, such as 6000 to 8000 Hz.
  • a 30 dB drop is expected from the first part of the spectrum to the end of the spectrum, indicating a spectral tilt with a negative slope and a vocalization when compared to pre-set slope thresholds.
  • Segments having a slope that is relatively flat may be classified as a cry since the spectral tilt may not exist for cries.
  • Segments having a positive slope may be classified as vegetative- sound.
  • a sixth rule may be comparing the entropy of the segment to entropy thresholds. Segments including relatively low entropy levels may be classified as vocalizations. Segments having high entropy levels may be classified as cries or vegetative- sound due to randomness of the energy.
  • a seventh rule may be comparing segment pitch to thresholds.
  • Segments having a pitch between 250 to 600 Hz may be classified as a vocalization.
  • Segments having a pitch of more than 600 Hz may be classified as a cry or squeal, and a pitch of less than 250 Hz may be classified as a growl.
  • An eighth rule may be determining pitch contours. Segments having a rising pitch may be classified as a happy sound. Segments having a falling pitch may be classified as an angry sound.
  • a ninth rule may be determining the presence of consonants and vowels. Segments having a mix of consonants and vowels may be classified as vocalizations. Segments having all or mostly consonants may be classified as a vegetative-sound or fixed- signal sound.
  • a rule according to various embodiments of the present invention may be implemented separately or concurrently with other rules.
  • the audio engine 208 implements one rule only while in other embodiments the audio engine 208 implements two or more rules.
  • Statistical processing may be performed in addition to or alternatively to the rule-based analysis.
  • Statistical processing may include processing segments with a MD- GMM using 2000 or more Gaussians in which models are created using Mel-scale Frequency Cepstral Coefficients (MFCC) and Subband Spectral Centroids (SSC). MFCCs can be extracted using a number of filter banks with coefficients. In one embodiment, 40 filter banks are used with 36 coefficients. SSCs may be created using filter banks to capture formant peaks. The number of filter banks used to capture formant peaks may be 7 filter banks in the range of 300 to 7500 Hz. Other statistical processing may include using statistics associated with one or more of the following segment characteristics:
  • an age-dependent model may be used in addition or alternatively to the techniques described above to classify key child segments. For example, vocalization, cry, and fixed-signal/vegetative-sound models may be created for each age group.
  • 12 different models are used with Group 1 corresponding to 1 to 2 months old, Group 2 corresponding to 3 to 4 months old, Group 3 corresponding to 5 to 6 months old, Group 4 corresponding to 7 to 8 months old, Group 5 corresponding to 9 to 10 months old, Group 6 corresponding to 11 to 12 months old, Group 7 corresponding to 13 to 14 months old, Group 8 corresponding to 15 to 18 months old, Group 9 corresponding to 19 to 22 months old, Group 10 corresponding to 23 to 26 months old, Group 11 corresponding to 27 to 30 months old, and Group 12 corresponding to 31 to 48 months old.
  • vocalization, cry, and fixed-signal/vegetative-sound models may be created for each month of age from 1 month to 48 months. This model will include 144 models, 48 models for each category. Alternative embodiments may use a different number of groups or associate different age ranges with the groups.
  • the audio engine 208 may also identify segments for which an adult is the source.
  • the segments associated with an adult source can include sounds indicative of conversational turns or can provide data for metrics indicating an estimate of the amount or number of words directed to the key child from the adult.
  • the audio engine 208 also identifies the occurrence of adult source segments to key child source segments to identify conversational turns.
  • the audio engine 208 estimates key child segment characteristics from at least some of the segments for which the key child is the source, independent of content. For example, the characteristics may be determined without determining or analyzing content of the sound in the key child segments.
  • Key child segment characteristics can include any type of characteristic associated with one or more of the key child segment categories. Examples of characteristics include duration of cries, number of squeals and growls, presence and number of canonical syllables, presence and number of repetitive babbles, presence and number of phonemes, protophones, phoneme-like sounds, word or vocalization count, or any identifiable vocalization or sound element.
  • the length of cry can be estimated by analyzing segments classified in the cry category.
  • the length of cry typically decreases as the child ages or matures and can be an indicator of the relative progression of the child's development.
  • the number of squeals and growls can be estimated based on pitch, spectral intensity, and dysphonation by analyzing segments classified as vocalizations.
  • a child's ability to produce squeals and growls can indicate the progression of the child's language ability as it indicates the key child's ability to control the pitch and intensity of sound.
  • the presence and number of canonical syllables, such as consonant and vowel sequences, can be estimated by analyzing segments in the vocalization category for relatively sharp formant transitions based on formant contours.
  • the presence and number of repetitive babbles may be estimated by analyzing segments classified in the vocalization category and applying rules related to formant transitions, durations, and voicing.
  • Babbling may include certain consonant/vowel combinations, including three voiced stops and two nasal stops.
  • the presence and number of canonical babbling may also be determined.
  • Canonical babbling may occur when 15% of syllables produced are canonical, regardless of repetition.
  • the presence, duration, and number of phoneme, protophones, or phoneme-like sounds may be determined. As the key child's language develops, the frequency and duration of phonemes increases or decreases or otherwise exhibits patterns associated with adult speech.
  • the number of words or other vocalizations made by the key child may be estimated by analyzing segments classified in the vocalization category.
  • the number of vowels and number of consonants are estimated using a phone decoder and combined with other segment parameters such as energy level and MD-GMM log-likelihood differences.
  • a least-square method may be applied to the combination to estimate the number of words spoken by the child.
  • the audio engine 208 estimates the number of vowels and consonants in each of the segments classified in the vocalization category and compares it to characteristics associated with the native language of the key child to estimate the number of words spoken by the key child.
  • an average number of consonants and vowels per word for the native language can be compared to the number of consonants and vowels to estimate the number of words.
  • Other metrics/characteristics can also be used, including phoneme, protophones, and phoneme-like sounds.
  • the audio engine 208 estimates characteristics associated with identified segments for which an adult is the source, independent of content. Examples of characteristics include a number of words spoken by the adult, duration of adult speech, and a number of parentese. The number of words spoken by the adult can be estimated using similar methods as described above with respect to the number of words spoken by the key child. One example of a method to detect adult word count is based on human annotated word-count, using Least-Squared Linear Regression to train. The model may also be guided or trained by human annotated word-count. The duration of adult speech can be estimated by analyzing the amount of energy in the adult source segments.
  • the audio engine 208 can determine one or more metrics associated with the language environment using the key child segment characteristics and/or the adult segment characteristics. For example, the audio engine 208 can determine a number of conversational turns or "turn-taking" by analyzing the characteristics and time periods associated with each segment. In some embodiments, the audio engine 208 can be configured to automatically determine the one or more metrics. In other embodiments, the audio engine 208 receives a command from input device 212 to determine a certain metric.
  • Metrics can include any quantifiable measurement of the key child's language environment based on the characteristics.
  • the metrics may also be comparisons of the characteristics to statistical averages of the same type of characteristics for other persons having similar attributes, such as age, to the key child.
  • Examples of metrics include average vocalizations per day expressed by the key child, average vocalizations for all days measured, the number of vocalizations per month, the number of vocalizations per hour of the day, the number of words directed to the child from an adult during a selected time period, and the number of conversational turns.
  • metrics may relate to the key child's developmental age. In the alternative or in addition to identifying delays and idiosyncrasies in the child's development as compared to an expected level, metrics may be developed that may estimate causes of such idiosyncratic and developmental delays. Examples of causes include developmental medical conditions such as autism or hearing problems.
  • the audio engine 208 outputs at least one metric to output device 114.
  • the audio engine 208 may, in response to a command received from input device 212, output a metric associated with a number of words spoken by the child per day to the output device 214, where it is displayed to the user.
  • Figures 7 - 12 are screen shots showing examples of metrics displayed on output device 214.
  • Figure 7 illustrates a graphical vocalization report showing the number of vocalizations per day attributable to the key child.
  • Figure 8 illustrates a graphical vocalization timeline showing the number of vocalizations in a day per hour.
  • Figure 9 illustrates a graphical adult words report showing a number of adult words directed to the key child during selected months.
  • Figure 10 illustrates a graphical words timeline showing the number of words per hour in a day attributable to the key child.
  • Figure 11 illustrates a graphical representation of a turn- takings report showing the number of conversational turns experienced by the key child on selected days per month.
  • Figure 12 illustrates a graphical representation of a key child's language progression over a selected amount of time and for particular characteristics.
  • a series of questions are presented to the user to elicit information about the key child's language skills.
  • the questions are based on well- known milestones that children achieve as they learn to speak. Examples of questions include whether the child currently expresses certain vocalizations such as babbling, words, phrases, and sentences.
  • no new questions are presented and the user is presented with a developmental snapshot of the speaker based on the responses to the questions.
  • the system stops and determines the developmental snapshot.
  • the questioning may be repeated periodically and the snapshot developed based on the answers and, in some embodiments, data from recording processing.
  • An example of a snapshot may include the language development chart shown in Figure 12.
  • the series of questions is answered automatically by analyzing the recorded speech and using the information obtained to automatically answer the questions.
  • Certain embodiments of the present invention do not require that the key child or other speakers train the system, as is required by many voice recognition systems.
  • Recording systems according to some embodiments of the present invention may be initially benchmarked by comparing certain determinations made by the system with determinations made by reviewing a transcript. To benchmark the performance of the segmenter, the identification of 1) key child v. non-key child and 2) adult v. non-adult were compared, as well as the accuracy of the identification of the speaker/source associated with the segments.
  • the recorded speech file could be processed to identify a particular word or sequence of words or the speech could be saved or shared. For example, a child's first utterance of "mama” or “dada” could be saved much as a photo of the child is saved or shared via e-mail with a family member.
  • Each language has a unique set of sounds that are meaningfully contrastive, referred to as a phonemic inventory.
  • English has 42 phonemes, 24 consonant phonemes and 18 vowel phonemes.
  • a phoneme is the smallest phonetic unit in a language that is capable of conveying a distinction in meaning.
  • a sound is considered to be a phoneme if its presence in a minimal word pair is associated with a difference in meaning. For example, we know that Itl and /p/ are phonemes of English because their presence in the same environment results in a meaning change (e.g., "cat" and "cap” have different meanings). Following linguistic conventions, phonemes are represented between slashes, such as Irl.
  • ASR automatic speech recognition
  • Sphinx ASR Sphinx ASR system provided by Carnegie Mellon University.
  • the phone decoder recognizes a set of phones or speech sounds, including consonant-like phones, such as "t” and “r” and vowel-like phones such as "er” and “ey”.
  • ASR phones are approximates of phonemes; they are acoustically similar to true phonemes, but they may not always sound like what a native speaker would categorize as phonemic.
  • phones or phone categories” and are represented using quotation marks. For example, "r” represents phone or phoneme-like sounds.
  • a very young child cannot produce the phoneme /r/, so not many "r" phones appear. However, over time more and more "r" phones appear (at least for an English-speaking child) until the child really does produce the Ixl phoneme.
  • a very young child may not attribute meaning to a protophone or phone. A child begins to produce true phonemes about the time that they start to talk (usually around 12 months of age), but even then the phonemes may only be recognized by those who know the child well.
  • the child's vocalizations can be used to assess the child's language development.
  • an adult ASR model does not work well with child speech
  • one embodiment of the present invention uses a phone decoder of an ASR system designed for adult speech, since the objective is to assess the language development of a child independent of the content of the child's speech. Even though a child does not produce a true phoneme, the phone decoder is forced to pick the phone category that best matches each phone produced by the child. By selecting the appropriate phone categories for
  • the adult ASR phone decoder can be used to assess child vocalizations or speech.
  • Fig. 13 describes the correlation between selected phones and chronological age. As shown in Fig. 13, there is a positive correlation between age and the "r" phone and a negative correlation between age and the "b" phone. As shown in Fig. 14, the correlation can be nonlinear over the age range of interest. For example, the correlation for the "1" phone is positive for ages 0 to 6 months, 7 to 13 months, and 14 to 20 months, but then becomes negative for ages 21 to 30 months and 31+ months.
  • one embodiment uses one or more recordings taken in the child's language environment. Each recording is processed to identify segments within the recording that correspond to the child with a high degree of confidence. Typically, the recording will be around 12 hours in duration in which the child produces a minimum of 3000 phones.
  • multiple models can be used to identify the key child segments, including, but not limited to, an age- based key child model, an other-child model, a male adult model, a female adult model, an electronic device model, a silence model, and a loudness/clearness model. The use of these models allows the recording to be taken in the child's language environment rather than requiring that the recording be taken in a controlled or clinical environment.
  • the phone decoder processes the high confidence key child segments
  • a phone parameter PCn for a particular phone category n represents the frequency count for that phone category divided by the total number of phones in all phone categories.
  • One particular embodiment uses 46 phone categories where 39 of the phone categories correspond to a speech sound (see Fig. 13) and 7 of the phone categories correspond to non- speech sounds or noise (filler categories), such as sounds that correspond to a breath, a cough, a laugh, a smack, "uh", "uhum,” “um” or silence.
  • phone decoders other than the Sphinx decoder. Since different phone decoders may identify different phone categories and/or different non-phone categories, the particular phone and non-phone categories used may vary from that shown in Figs. 12 and 13.
  • ELZ(key child) the phone parameters PCn are used in the following equation:
  • the expressive language index includes a weight bn(age) associated with each phone category n at the age (AGE) of the key child.
  • bl(12) corresponds to the weight associated with phone category 1 at an age of 12 months
  • b2(18) corresponds to the weight associated with phone category 2 at an age of 18 months.
  • the weights bn(age) in the expressive language index equation may differ for different ages, so there is a different equation for each monthly age from 2 months to 48 months.
  • the equation for a 12-month-old child uses the weights shown in the "12 months" column in Figure 15. The derivation of the values for the weights bn(age) is discussed below.
  • SLP-administered language assessment tools typically estimate developmental age from counts of observed behaviors. Using a large sample of children in the age range of interest, developmental age is defined as the median age for which a given raw count is attained. In one embodiment of the system, the phone probability distribution does not generate raw counts of observed behaviors, and development age is generated in an alternative approach as an adjustment upward or downward to a child's chronological age. In this embodiment, the magnitude of the adjustment is proportional both to the expressive language standard score (ELSS) and to the variability in ELSS observed for the child's chronological age.
  • ELSS expressive language standard score
  • Boundary conditions are applied to prevent nonsensical developmental age estimates.
  • the boundary conditions set any estimates that are greater than 2.33 standard deviations from the mean (approximately equal to the 1st and 99th percentiles) to either the 1st or 99th percentiles.
  • An age-based smoothed estimate of variability is shown below in equation (3). The determination of the values shown in equation (3) other than age is discussed below.
  • ELD A To determine the child's expressive language developmental age, ELD A, the child's chronological age is adjusted as shown below in equation (4). The determination of the constant value shown in equation (4) is discussed below.
  • the expressive language developmental age is calculated using a chronological age of 12 and a constant of 7.81 as shown below:
  • the system can output the child's EL standard score, ELSS, and the child's EL developmental age, ELDA.
  • the system can compare the child's chronological age to the calculated developmental age and based on the comparison output a flag or other indicator when the difference between the two exceeds a threshold. For example, if the ELSS is more than 1.5 standard deviations lower than normal, then a message might be output suggesting that language development may be delayed or indicating that further assessment is needed.
  • the validity of the EL model was tested by comparing EL standard scores and EL developmental ages to results derived from the assessments administered by the SLPs.
  • the EL standard score is an accurate predictor of potential expressive language delay. Using a threshold score of 77.5 (1.5 standard deviations below the mean), the EL standard score correctly identified 68% of the children in one study who fell below that threshold based on an SLP assessment. Thirty-two percent of the children identified as having possible delays had below average EL scores, but did not meet the 77.5 threshold score. Only 2% of the non-delayed children were identified as having possible delay based on their EL score.
  • One way of increasing the accuracy of the EL assessment is to average the EL scores derived from three or more recording sessions.
  • the LENA Developmental Snapshot questionnaire is one example of a questionnaire that uses a series of questions to the parent to elicit information about important milestones in a child's language development, such as identifying when the child begins to babble, uses certain words, or constructs sentences.
  • the foregoing example detects single phones and uses the frequency distribution of the single phones to estimate a standard score and developmental age
  • bi-phone sequences may be used instead of single phones and in another embodiment, tri-phone sequences may be used.
  • combinations of phones and bi-phones or phones, bi-phones, and tri-phones may be used. The invention is not limited in use to phones, bi-phones, or tri-phones.
  • Bi-phone (or the usage of more than one phone) allows for the incorporation of sequence information.
  • phones tend to occur in a logical sequence; therefore, additional resolution is gained by analyzing not just the phones but the sequence of the phones.
  • Bi-phones are defined as each pair of adjacent phones in a decoded sequence.
  • the decoded phone sequence "P A T” contains the phone pairs "P-A' and "A-T".
  • a tri-phone sequence in this case would be "P A T.”
  • uni-phones are included as a single phone paired with an utterance start or stop marker.
  • bi-phone frequencies then are used as the input to the same type of linear regression models described above for the uni-phone case.
  • the introduction of bi- phone or tri-phone also introduces a challenging technical issue, i.e., the dimension of bi- phone (total number of bi-phone) is significantly larger than uni-phone (n-squared versus n), and the dimension of tri-phone (n-raised-power-to-3) is even much bigger than that of both bi-phone and uni-phone.
  • PCA Principal Component Analysis
  • PCA principle component analysis
  • the over 2300 bi-phone combinations were reduced to 50 principal components to use as predictors in multiple linear regression models predicting SLP -based scores, exactly as described above in the uni-phone case.
  • the first m components could be chosen to construct linear transform, where m ⁇ n .
  • Another alternative embodiment uses phone duration rather than phone frequency.
  • the phone decoder determines the length of time or duration for each phone category.
  • a phone duration parameter PCn for a particular phone category n represents the duration for that phone category divided by the total duration of phones in all phone categories.
  • the phone duration parameters are used in an equation that is similar to equation (1), but that uses different weights. The weights may be calculated in a matter similar to that used to calculate weights for frequency distribution.
  • MLU mean length of utterance
  • This measurement originally formalized by Brown, assumes that since the length of child utterances increases with age, one can derive a reasonable estimate of a child's expressive language development by knowing the average length of the child's utterances or sentences. See Brown, R., A First Language: The Early Stages, Cambridge, Mass., Harvard University Press (1973). Brown and others have associated utterance length with developmental milestones (e.g., productive use of inflectional morphology), reporting consistent stages of language development associated with MLU. Utterance length is considered to be a reliable indicator of child language complexity up to an MLU of 4 to 5 morphemes.
  • transcribers computed the MLU for 55 children 15 to 48 months of age (approximately two children for each age month).
  • the transcribers followed transcription and morpheme-counting guidelines described in Miller and Chapman, which were in turn based on Brown's original rules. See Miller, J.F. & Chapman, R.S., "The Relation between Age and Mean Length of Utterance in Morphemes", Journal of Speech and Hearing Research, Vol. 24, pp. 154-161 (1981). They identified 50 key child utterances in each file and counted the number of morphemes in each utterance. The MLU was calculated by dividing the total number of morphemes in each transcribed file by 50.
  • the system produces an Estimated Mean Length of Utterance (EMLU).
  • EMLU may be generated by predicting human-derived MLU values directly from phone frequency or phone duration distributions, similar to the estimate of the expressive language estimate ELZ.
  • the EMLU may be generated based on simple linear regression using developmental age estimates to predict human-derived MLU values. For example,
  • the observations and assessments for each child were averaged together and transformed to a standard z-score to produce an expressive language index value for each child for a particular age.
  • the phone category information output from the Sphinx phone decoder was used along with multiple linear regression to determine the appropriate weights for the expressive language index for each age.
  • data from 4-, 5-, 6-, 7-, and 8-month olds were used to create a different set of weights for the expressive language index for a 6-month old, etc. This process was repeated for all age months and across increasingly broad age ranges.
  • a dynamic programming approach was used to select the optimal age range and weights for each monthly age group.
  • the age band is from age 6 months to age 18 months and the weights are shown in the table in Figure 15.
  • Figure 15 also illustrates the weights for another example for a key child aged 6 months with an age band from 3 months to 9 months, and the weight for a key child aged 18 months with an age band from 11 months to 25 months.
  • the age ranges in these examples are symmetric, the age ranges do not have to be symmetric and typically are not symmetric for ages at the ends of the age range of interest.
  • Figure 16 illustrates a block diagram for an exemplary system that computes an EL score and developmental age as described above.
  • the illustrated system includes a digital recorder 1602 for recording audio associated with the child's language environment.
  • the recorded audio is processed by the feature extraction component 1604 and segmentation and segment ID component 1606 to extract high confidence key child segments.
  • a phone decoder 1608 based on a model used to recognize content from adult speech processes the high confidence key child segments 1607.
  • the phone decoder provides information on the frequency distribution of certain phones to the EL component 1610.
  • the EL component uses the information to calculate the EL score, estimate the developmental age, and/or estimate the mean length of utterances as described above.
  • the Reports and Display component 1612 outputs the EL information as appropriate.
  • Figure 16 illustrates that a recording is processed using a system that processes recordings taken in the child's language environment, such as the LENA system
  • the EL assessment can operate with key child segments generated in any manner, including recordings taken in a clinical or research environment, or segments generated using a combination of automatic and manual processing.
  • a system and method for detecting autism uses the automatic language processing system and methodologies described above. Recordings captured in a natural language environment are processed, and a model of the language development of those known subjects is created. By using a large enough sample, trends in language development can be determined. This is referred to as normative trends. Generally, if there is a particular developmental disorder that is desired to be studied, then the language of individuals having the disorder and normal individuals is studied and trends are developed.
  • the methodology described herein is an example of how a particular developmental disorder, autism, may be detected using language analysis. The method and system, however, may be applied to a variety of disorders and diseases, for example autism and Alzheimer's disease. All diseases and disorders that may be detected through the analysis of language may be detected through this embodiment.
  • autism In the case of autism, aberrations in the voice of individuals have been noted in the descriptions of Autism Spectrum Disorders (ASD). It has been shown in numerous studies that autism is indeed associated with abnormalities of vocal quality, prosody, and other features of speech. See R. Paul, A. Augustyn, A. Klin, F. R. Volkmar, Journal of Autism and Developmental Disorders 35, 205 (2005); W. Pronovost, M. P.
  • autism may be affirmatively detected based on positive markers based on the characteristics of speech that could not previously be performed.
  • autism is detected by using "negative markers", such as a deficit in joint attention. See, for example: S. Baron-Cohen, J. J Allen, C. Gillberg, The British Journal of Psychiatry 161, 839 (1992); K. A. Loveland, S. H. Landry, Journal of Autism and
  • Transparent Parameters are those parameters that may be extracted from the sound signal and are independent of the actual content of the sound signal in terms of meaning of the language or sounds produced. Transparent parameters are discussed further below.
  • CSATP includes a number of steps: segmentation; VOC, CRY, and VEGFIX Classification and vocalization count; acoustic analysis; extraction of transparent parameters; and data set classification. Using this methodology and a sample of sufficient size of children having normal speech development, delayed speech development, and autism, trends in language may be developed for these groups. See the above discussion of VOC, CRY, and VEGFIX classification in relation to audio engine 208 that may classify key child audio segments into one or more categories.
  • Figures 17 and 18 show a flow chart for a method of detecting autism and a method of creating trends for use in the method of detecting autism, respectively.
  • the segmentation of block 1810 and 1835 is performed as described above in reference to Figure 4 and block 304.
  • the segmentation is performed on data for an individual key child, and in block 1835 the segmentation is performed on a plurality of recordings of normal, delayed, and autistic children.
  • the speaker is identified for a particular piece of a recording. After the speaker has been identified, the language from the speaker of interest, in this case the key child, is analyzed further.
  • Figure 19 shows the segmentation process in the top graph and the further break down of key child segments into VOC, CRY, and VEGFIX segments.
  • VOC vocalizations
  • CRC cries
  • VEGFIX vegetative-sound and fixed-signal sounds
  • vocalizations may include canonical syllables and repetitive babbles which are well constructed consonant and vowel sequences. At this stage, a child may explore with variation of pitch creating high pitched squeals and low pitched and dysphonated growls. Around a year, a child starts saying isolated words, but keeps babbling too until 18 months or so. By two years, a child will have a fairly large vocabulary of spoken words. In short, vocalizations include all meaningful sounds which contribute to the language development of the child.
  • Vegetative-sound includes all non- vocal sounds related to respiration and digestion, e.g., coughing, sneezing, and burping.
  • Fixed-signals are sounds which are related to the voluntary reactions to the environment, e.g., laughing, moaning, sighing, and lip smacking.
  • Vegetative-sound and fixed-signal sounds are detected collectively. These types of sounds are eliminated since they do not provide information about linguistic
  • cries are also a type of fixed-signal. Unlike other fixed-signals, cries are very frequent (depending on the age) and convey various emotional feelings and physical needs. Although not performed in this specific method, the analysis of cries according to the described techniques may be used to detect disorders or diseases, since crying is also another means of communication in a baby's life.
  • Child speech classification is performed by statistical processing using Mel-scale Frequency Cepstral Coefficients (MFCC) and Subband Spectral Centroids (SSC). Other statistical processing techniques may be used.
  • MFCC Mel-scale Frequency Cepstral Coefficients
  • SSC Subband Spectral Centroids
  • MFCC MFCC
  • SSC SSC
  • the power spectrum in a given subband is smoothed out, so that only the weighted amplitude of the power spectrum is kept, while in SSC the centroid frequency of each subband is extracted.
  • SSC's can track the peak frequency in each subband for the speech section, while for the non-speech section it stays at the center of the subband.
  • MFCC is a better feature than SSC by itself, but the combination of MFCC and SSC demonstrates better performance for the automatic speech recognition of adult speech. SSC has been applied for various applications - some of them are listed below:
  • FB-GMM Fixed Boundary Gaussian Mixture Model
  • MFCC Fixed Boundary Gaussian Mixture Model
  • SSC SSC's
  • the purpose is to classify three types of speech - vocalizations, cries, and fixed-signal/vegetative-sound sounds.
  • acoustic analysis is performed on the VOC islands (recordings corresponding to periods of very high energy bounded by periods of very low energy).
  • the islands within the child segments then are further analyzed using acoustic features.
  • the following acoustic features are extracted from the VOC islands:
  • Fig. 6 shows an example of a vocalization, which is a series of consonant vowel sequences, (hi-ba-ba-bab-bab). Only the vowels are the high energy parts while the consonants have low energy. The duration of the high energy parts are measured for validation of vocalization.
  • Canonical syllable identification Formant transitions (mainly for Fl and F2) can be noticed in CV, VC, CVC or VCV sequences.
  • Articulation analysis Formant bandwidths mark the clarity of
  • Figure 20 shows an empirical display for how the grouping of Fl and F2 bandwidths can mark the articulation level. A score is assigned to each articulation group based of the "goodness" of each articulation level.
  • Emotional intensity analysis High intensity speech sounds (e.g., a cry with a full lung of air) are observed to have the first spectral peak above 1500 Hz. Normal vocalizations will have more energy in the lower frequencies (ranging from 300 to 3000 Hz) than higher frequencies (6000 to 8000 Hz). Thus, there will be a 30 dB drop expected from the first part of the spectrum to the end of the spectrum, which is referred to as spectral tilt with a negative slope. For cries, the spectral tilt may not exist, where the spectrum is rather flat. A spectral tilt with a positive slope (low energy in lower frequencies and high energy in higher frequencies) indicates non-vocal sound (e.g., breathing, lip smacking).
  • Dysphonation analysis It is assumed that normal vocalizations which are mostly composed of vowels makes the spectrum periodic. On the other hand, dysphonated sounds have rather random spectrums with subharmonics in the spectrum. The randomness of the spectrum can be measured by the entropy of the spectrum. The higher the entropy, the more random is the spectrum and the higher the dysphonation.
  • Pitch analysis Pitch is used to detect squeals and growls. Normal pitch for a child is in the range of 250 to 600 Hz. A vocalization is marked as a squeal if the pitch is more than 600 Hz (it could go up to 3000 Hz). Similarly, growls are vocalizations which have pitch lower than 250 Hz.
  • Intonation analysis Intonation has a major role in determining the emotion of the child. Squeals and growls are vocalizations only when they are playful and happy. Angry versions of those high- or low pitched and dysphonated sounds are cries. Pitch contours help determine whether the speech is angry or happy. Typically, a rising pitch is an indicator of happy sounds, while a falling pitch is a sad sound. 8. voicing analysis: It is assumed that vocalizations are mostly composed of vowels, which are voiced speech, with interlaced consonants (unvoiced speech). If an entire speech section is unvoiced, then it is assumed to be some sort of a vegetative-sound/fixed- signal sound (e.g., cough, throat clearing, etc.).
  • formant peaks and formant bandwidths are detected using Linear Predictive (LP) analysis, while pitch is calculated based on autocorrelations.
  • formant and pitch contours are extracted by applying a smoothing filter - median filter.
  • Other spectrum analyses are performed using a 1024 point FFT.
  • the transparent parameters are extracted. These parameters are used to determine whether a subject is normative or autistic.
  • Figure 21 shows acoustic parameters pertinent to the determination of autism.
  • Figures 21 and 22 show additional acoustic and non-acoustic parameters that may be extracted from recordings.
  • the acoustic parameters depicted in Figures 21 and 22 are used for the detection of autism.
  • the non-acoustic parameters depicted in Figure 22 may be used for the detection of autism.
  • these acoustic and non-acoustic parameters are referred to as transparent parameters.
  • the acoustic parameters relate to those vocalizations created by the key child
  • non-acoustic parameters are those relating to the interactions, specifically those interactions between the key child and adults, and the environment that the child experiences.
  • the nine non-acoustic parameters are shown in Figure 22.
  • the adult vocalization length in seconds refers to the length of adult vocalization on the recording.
  • the adult vocalization count refers to the number of vocalizations made by an adult.
  • the number of child-initiated conversations refers to the number of times a child makes a vocalization and an adult replies.
  • the number of conversational turns refers to the number of times a child responds to an adult vocalization.
  • the number of conversational turns in child-initiated conversations refers to when a child initiates a conversation and then responds to an adult vocalization thereafter.
  • the child vocalization length in seconds in conversational turns refers to the length of time child vocalizations last in conversational turns.
  • the child vocalization counts in conversational turns refer to the number of vocalizations a child makes in a conversational turn (which may indicate the complexity of an answer).
  • the child vocalization length in conversations with an adult is the average vocalization length of a child over a conversation with an adult.
  • the child vocalization counts in conversations with and adult is the number of vocalizations made by a child over a delineated conversation with an adult.
  • the twelve acoustic parameters shown in Figure 21 are both theoretically (based on models from 30 years of research in vocal development) and statistically (as indicated by principal components analysis, PCA) clustered into four groupings pertaining to the infrastructure for speech. Each of the twelve parameters are classified as a plus or minus. To adjust for differences in rate of vocalization (volubility) across individual children and recordings as well as differences in lengths of recordings, for each parameter the ratio of the number of vocalizations labeled plus to the number of utterances is taken. This yields a set of 12 numbers (one for each parameter) per recording. This 12-dimensional vector is used to predict vocal development and to classify recordings as belonging to typically developing or autistic children in the analyses.
  • FIG. 24 shows a trend chart for acoustic parameters in autistic and normally developing children.
  • the gray dots represent the vocal development scores for normally developing children.
  • the gray line is a trend line for normally developing children.
  • the asterisks represent vocal development scores for autistic children.
  • the diamonds represent the average (based on multiple recordings for a single child) vocal development scores for autistic children.
  • the black trend line is for autistic children.
  • Figure 25 shows a trend chart for acoustic parameters in autistic, normally developing, and language delayed children.
  • the gray stars represent the average (based on multiple recordings for a single child) vocal development scores for language delayed children.
  • the black diamonds represent the average (based on multiple recordings for a single child) vocal development scores for autistic children.
  • the gray trend line is for language delayed children.
  • the black trend line is for autistic children.
  • the broken trend line is for normally developing children.
  • Figure 26 shows a trend chart for acoustic parameters in normally developing and language delayed children.
  • the gray dots represent the vocal development scores for normally developing children.
  • the asterisks represent vocal development scores for language delayed children.
  • the black stars represent the average (based on multiple recordings for a single child) vocal development scores for language delayed children.
  • the black trend line is for language delayed children.
  • the gray trend line is for normally developing children.
  • Figure 27 shows non-acoustic parameters in normally developing and autistic children.
  • the gray dots represent the vocal development scores for normally developing children.
  • the gray line is a trend line for normally developing children.
  • the asterisks represent vocal development scores for autistic children.
  • the diamonds represent the average (based on multiple recordings for a single child) vocal development scores for autistic children.
  • the black trend line is for autistic children.
  • Figure 28 shows a trend chart for acoustic parameters in autistic, normally developing, and language delayed children.
  • the gray stars represent the average (based on multiple recordings for a single child) vocal development scores for language delayed children.
  • the black diamonds represent the average (based on multiple recordings for a single child) vocal development scores for autistic children.
  • the gray trend line is for language delayed children.
  • the black trend line is for autistic children.
  • the broken trend line is for normally developing children.
  • Figure 29 shows a trend chart for acoustic parameters in normally developing and language delayed children.
  • the gray dots represent the vocal development scores for normally developing children.
  • the asterisks represent vocal development scores for language delayed children.
  • the black stars represent the average (based on multiple recordings for a single child) vocal development scores for language delayed children.
  • the black trend line is for language delayed children.
  • the gray trend line is for normally developing children.
  • the predicted vocal development score by employing acoustic or non- acoustic parameters for the population studied can be projected versus the age in months of the child.
  • the creation of a predicted vocal development score is based on analysis of transparent parameters (including acoustic or non-acoustic). For example, in a case of acoustic parameters, multiple linear regression (MLR) analysis can be conducted to obtain perspective on both development and group differentiation.
  • MLR multiple linear regression
  • FIG. 21 the 12 acoustic parameter ratios of speech-related vocal islands (SVIs, previously referred to as VOC islands) to speech-related child utterances (SCUs) were regressed within each recording against age for the typically developing sample, yielding a normative model of development with respect to acoustic organization of vocalizations.
  • the data set related to the key child in question is compared to the trend lines of known subjects in order to make a determination as to whether the individual is autistic, delayed, or normal.
  • Logistic Regression Analysis was used to model optimum classification of children as autistic or non- autistic based on the 12 acoustic parameters. In the case of normally developing children, a high percentage of normal children were identified as normal.
  • FIG. 31 a number of tables are shown showing the accuracy of various methodologies of determining the likelihood of autism.
  • EER Logistic Regression and an equal error rate
  • the method had a high degree of success while only delivering a small number of false positives. For instance, in the case where a probability of .98 was used, the system and method determined that 93% of those subjects were considered normal, and only had a small error rate in determining that some normal individuals were autistic. At the same time, only 12% of individuals were determined to be normal when they were really autistic, and 88% of autistic individuals were correctly identified as autistic.
  • the bottom row of tables shows the alternative Linear Discriminant Analysis, and shows similar results.
  • the above system and method is described for application in detecting autism, it may be used in for a number of different diseases and disorders related to speech.
  • the model/trend creation functions according to the same principles described in Figure 18.
  • a trend or model can be created in block 1855 that may be used to compare new subjects in order to determine if they have the disease or disorder. New subjects are processed according to Figure 17 in a similar fashion and ultimately compared to the trend determined in block 1830. Furthermore, although the above description focuses on vocalization data, as the database of child recordings in a natural language environment grows for children of very young (less than a year) ages, data concerning the cries of children may reveal trends that can allow for the detection of autism.
  • autism and other diseases may be detected using either solely the above-described phone analysis in relation to child language development or the above-described phone analysis in conjunction with transparent feature analysis .
  • PCA principal component analysis
  • human SLP assessment scores can be predicted by an embodiment of the above-described system and method.
  • a phone-based feature used for AVA could be used for autism detection with the rest of the system unchanged, including LDA (linear discriminant analysis), logistic regression, etc.
  • LDA linear discriminant analysis
  • the addition of phone-based feature analysis to acoustic transparent feature analysis could provide additional resolution in respect to autism detection.
  • much of the analysis is focused on vocalizations as the database of child recordings in a natural language environment grows for children of very young (less than a year) ages, data concerning the cries of children may reveal trends.
  • phone-based features are used to detect autism.
  • Alternative methods of incorporating multiple recordings for analysis of the language of a single child are also included.
  • These methods are specifically targeted towards autism in this example; however, they may be utilized for the detection of other disorders and the analysis of speech according to any method described herein.
  • phone-based features yielded better results than the above explained transparent features. This was especially true for distinguishing autism from language delay.
  • the method of incorporating multiple recordings from a single child may vary; and in the present example, it was determined that using posterior probabilities was most effective considering the data used. Previously, age-normalized features from different recordings are averaged together to form a single feature vector for a child.
  • each individual recording and its feature vector may be used to obtain posterior probability.
  • the incorporation of multiple recordings for a child may be done in the posterior probability space.
  • the posterior probabilities from multiple recordings may be averaged together to obtain a single averaged posterior probability for a child.
  • the average may be "geometric" or "arithmetic".
  • the data used in the present example is the same data described above and depicted in Fig. 23.
  • This data includes three sets of children: 1) typically developing or normative children (represented by “N” or “n” in Table 1 below); 2) language-delayed children (represented by “D” or “d” in Table 1 below); and 3) autistic children (represented by "A” or “a” in Table 1 below).
  • a summary of the data is:
  • Set-N typical children; 328 children; 2678 recordings
  • the three basic tasks are based on each pair of Set-N, D, A to see the classification of each pair of them: 1) classification of autism from delay; 2) classification of delay from normal; and 3) classification of autism from normal.
  • autism detection the detection of autism from normative set and delayed set is the actual focus. Additional resolution, even for autism versus non-autism (delayed + typical), is achievable in relation to the detail of separating autism from delay and separating autism from typical set. Following is the summary of six cases investigated (and reflected in Table 1):
  • a-d of Set-A from Set-D, trained and tested with LOOCV on Set- A, D
  • d-n detection of Set-D from Set-N, trained and tested with LOOCV on Set- D, N.
  • a-dn_a-d training is the same as “a-dn”, but just check the performance of "a-d”
  • a-dn_a-n training is the same as “a-dn”, but just check the performance of "a-n” B. Performance Measure
  • LOOCV leave-one-out-cross-validation
  • LOOCV may be used for the validation of the detection of other disorders or classifications other than autism, such as the many disorders and classifications discussed elsewhere in this disclosure.
  • class-c the classification of the child being validated
  • non-c the classification of the child being validated
  • one child is left out of the model each time, regardless of whether the child is associated with one feature vector, which is some kind of combination from multiple recordings, or if the child is associated with several feature vectors, which are from each corresponding recording for that child.
  • a ROC curve (Receiver Operating Characteristic curve, which is a plot of the true positive rate against the false positive rate for the different possible cutpoints of the test) could be drawn based on posterior probabilities of all children. Equal-error-rate could be calculated at the same time. Specifically, the procedure to draw ROC and to calculate equal-error-rate is as follows:
  • array_p unique posterior probabilities sorted in ascending order
  • threshold_array [array_p(l ..n-l) + array_p(2..n)] / 2, i.e., middle-points between neighboring unique posterior probabilities
  • detection decision could be made: any child is detected as class-c if its posterior probability is above the threshold; otherwise, the child is detected as class non-c
  • detection error rate number-of-class-c-children-mis-detected-as-non-c
  • the equal-error-rate-point is the intersection of two curves mentioned in step 5.
  • the calculation of the intersection point is trivial since the two curves are either monotonically increasing or decreasing.
  • Figure 34 shows ROC for a case "a-d” in Baseline with LDA method.
  • Figure 35 shows ROC of a case "a-d” with biph-50 feature and geometric-posterior- probability average to combine multiple recordings of a key child.
  • a feature vector is converted into a posterior probability; although, explained in the context of autism detection, this technique may be applied to other analyses of speech to determine characteristics or disorders of an individual.
  • Two modeling methods are used to perform the conversion: logistic regression and LDA (linear discriminant analysis).
  • Logistic Regression uses the following function to convert a feature vector into posterior probability:
  • posterior_probability 1 / (1 + exp( A * feature_vector + b ) ) where A is a linear model vector, * is inner product, and b is offset parameter. Both A and b could be estimated using Maximum Likelihood method with Newton-Raphson optimization algorithm.
  • LDA itself could not directly provide posterior probability.
  • the purpose of LDA is to find a linear transform so that the Fisher-Ratio is optimized in the output space of the linear transform or discrimination is optimized in the output space.
  • x) is the posterior probability of being as class-c given observation x
  • P(c) is a priori probability of class-c
  • c) is the data distribution of class-c.
  • the "ft-12" is done basically as described above in relation to the determination of autism according to transparent features. 3.
  • the single recording performance is worse than that of child-level. In other words, the performance of child-level could be improved by using multiple recordings for a child.
  • the geometric average of posterior probability for multirecordings of a child is usually better than arithmetic average.
  • biph-50 is significantly better than ft-12, especially for delayed versus autistic. The superior of biph-50 over ft-12 is consistent through all cases.
  • the posterior probability may be combined into the above-described analysis techniques for determining the developmental age of a key child; or it can be used in the detection of other disorders, diseases, or characteristics from the analysis of speech.
  • a party interested in a detecting autism in a child may request a test system be sent to them.
  • a test system may be sent to them by mail or other delivery means, or may be given to them by a doctor or medical professional.
  • the system includes the recording unit, instructions, and clothing for the subject (the key child) to wear that is adapted to hold the recording unit.
  • the child is then recorded for the specified period and the system is returned by mail or physically returned to a central processing receiver.
  • the central processing receiver retrieves the data from the system and processes the data. Reports are returned to the necessary parties which may include the parents of the key child, the physician, other professionals, etc.
  • This method may be implemented in a low cost fashion since the key child or key child's guardian/parent is in effect "renting" the unit for a one time use. After usage the same unit may be reused for another subject who will pay the "rental” fee, collect the needed data, return the unit, and receive the needed test results.
  • ASR automatic speech recognition
  • ASR ASR designed for adults
  • the adult ASR is based on adult speech.
  • the data analyzed is child speech. Therefore, the data from which the model was created may have limitations or inaccuracies when compared to disparate data, e.g., child speech.
  • a model created from the analysis of child speech may be used.
  • unsupervised clustering methods could be used for child speech modeling. This method, based on the statistical characteristics of data, clusters similar child speech data together. This methodology may reduce the need for human classification of child speech. Since the above methods are based on statistically comparing the development of a subject to a model for development of known subjects, the actual phones spoken may be excluded from the analysis. Instead, clusters of speech segments that may or may not represent actual phones are developed, and the speech of a subject is compared to these clusters.
  • K-means algorithm is trying to find k representative points ⁇ c ;
  • i 1,..., k ⁇ , where k is smaller (or much smaller) than n.
  • C i are cluster centroids or cluster means. This is why it is called K-means.
  • step 3 Iterate step 3 and step 4 until some convergence criterion is met (theoretically, the iteration is guaranteed to converge to at least the local minimum of smallest overall data "distortion").
  • the above table shows essentially the same performance of unsupervised method as the one using an adult phone model. This is a verification of previous analysis using an adult phone model. At the same time, this also shows the promise and potential of unsupervised method because it may be more flexible in terms of number of clusters to choose, etc. Although particular numbers of clusters are shown, the optimal number of clusters for a given data set may depend on the size of the data set and various numbers of clusters may be used.
  • cluster-based feature analysis can be used for autism detection or the detection of other disorders/diseases.
  • the combination of cluster- based feature, adult-phone-model-based feature, acoustic-transparent feature could be done towards autism detection.
  • transparent features are used in the analysis.
  • clusters are developed in relation to the characteristics of speech and sound. These characteristics may include the pitch of the sound, the duration of the sound, the rhythm of the sound, the organization of the sound, etc.
  • the acoustic parameters shown have definitions for short, medium, long, and extra-long duration islands. Instead, these definitions may be established by clustering of actual sound recordings and will create a cluster model representative of the data collected.
  • the model developed may be finely tuned according to specific age and any other characteristics that are known about the population representing the recording data upon which the model is based.
  • the characteristics of speech primarily consist of the pitch of the speech, the duration of the speech, and organization of the speech. Clustering can be done according to any and all of these characteristics alone and in combination. Additional speech characteristics may include speech flow, loudness, intonation, and intensity of overtones. Speech flow includes the production speed of utterances and the length of breaks in speaking. Loudness is the amount of energy associated with the speech. Intonation relates to rise and fall in pitch with respect to the speaker's mean vocal pitch. Overtones include higher tones which accompany fundamental tones and are generally fainter than the fundamental tone. All of these characteristics and more can be used to form clusters.
  • Clustering allows for analysis in the absence of preconceived notions about the characteristics of speech and may reveal patterns previously unrecognized. As long as the sample collected is large enough (statistically speaking), the patterns revealed through clustering will hold true for a population and may be applied to any type of speech analysis in terms of development, detection of disease and disorder (such as autism), and other characteristics of speech, such as emotion, the speaker's underlying motivations, veracity, for example. Emotion/Stress Detection
  • the utterance is scored upon the feature in respect to a plurality of models representing emotions.
  • the model having the maximum score is selected, and the emotion associated with that model is identified as the emotion status of the utterance.
  • a Gaussian Mixture Model may be used for the scoring, which is described above for segmentation and segment-ID task.
  • GMM Gaussian Mixture Model
  • the detection of a stressed or non- stressed condition may be simpler than specific emotion detection and, thus, may be more accurate. This methodology may be performed using the speech detection and analysis system described herein.
  • Table 4 MFCC feature size for emotion detection on German database
  • Table 5 Different Feature for Emotion Detection on German database
  • the dimension of feature needs to be increased to include more relevant characteristics. This may be done by using higher orders of MFCC or PMVDR and including more context (or neighboring) feature frames to cover dynamics of speech which may be associated with emotion.
  • increasing the feature dimension may not necessarily improve the detection rate. The reason is that the increased feature dimension may result in the model size increase and thus intensify the conflict between model size and limited amount of training data.
  • increasing feature size may incorporate more useful information, increasing the feature size could also introduce some irrelevant features or noise. This could make the modeling process even harder to converge to relevant characteristics of input features.
  • LDA Linear Discriminant Analysis
  • other forms of analysis that can reduce the
  • Table 6 Simple Test of LDA for emotion detection on German database [0232]
  • the output dimension of standard LDA may be confined by the total number of classes involved (actually the maximum number of output feature for standard LDA is J - 1 if there are J classes).
  • the standard LDA can only have one output feature, which may not be good enough.
  • subclass LDA was proposed. For each class, different sub-classes (or clusters) could be obtained using, e.g., K-means algorithm which is described earlier. Since this is basically an unsupervised method, each class can have as many sub-classes as needed.
  • each class has 5 sub-classes
  • the German database is acted emotion data.
  • Infoture LENA-Emotion- Data-1 comes from a real natural home environment in an unobtrusive way.
  • To test ideas and methods for emotion detection on Infoture LENA-Emotion-Data-1 may be interesting since the Infoture LENA-Emotion-Data-1 was collected in a natural language environment.
  • the model trained with the German database was applied on LENA-Emotion-Data-1 for stress/non-stress detection.
  • the detection rate is 51%, similar to random guessing. This is probably due to the mismatch between the LENA-Emotion-Data-1 and the model trained from the German database.
  • models trained on LENA-Emotion-Data-1 are directly tested on LENA data.
  • Table 8 Leave-one-recording-out-cross-validation on LENA-Emotion-Data-1
  • a number of analysis techniques are mentioned herein to address the detection of developmental age, autism, emotion, etc. Although the analysis techniques expressed are believed to be the best available techniques for the determination of such characteristics, they are based at least in part on the quality and quantity of data collected on which the analysis is based. Therefore, the individual techniques utilized at various stages of the analysis may be interchanged. For instance, LDA and Logistical Regression Analysis may be interchanged depending on their performance characteristics, as may methodologies of incorporating multiple recordings for a subject and choice of recording features used (transparent features vs. phone -based features). [0236] In all cases of the above-described embodiments, the results of any of the transformations of data described may be realized by outputting the results by transforming any physical or electronic medium available into another state or thing.
  • Such output includes, but is not limited to, producing hardcopy (paper), sounds, visual display (as in the case of monitors, projectors, etc.), tactile display, changes in electronic medium, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Machine Translation (AREA)

Abstract

Dans un mode de réalisation selon l'invention, un procédé de détection de l'autisme dans un environnement de langage naturel à l'aide d'un microphone, d'un enregistreur de sons et d'un ordinateur programmé avec un logiciel dans le but précis de traiter des enregistrements capturés par le microphone associé audit enregistreur, l'ordinateur étant programmé pour exécuter le procédé, consiste à segmenter en une pluralité de segments d'enregistrement un signal audio capturé par le microphone associé audit enregistreur à l'aide de l'ordinateur programmé dans ce but précis. Le procédé consiste en outre à déterminer les segments de la pluralité de segments d'enregistrement qui correspondent à un enfant clé. Le procédé consiste en outre à déterminer les segments de la pluralité de segments d'enregistrement correspondant à l'enfant clé qui sont classés en tant qu'enregistrements de l'enfant clé. Par ailleurs, le procédé consiste à extraire des caractéristiques reposant sur des phones des enregistrements de l'enfant clé; à comparer les caractéristiques reposant sur des phones des enregistrements de l'enfant clé avec des caractéristiques reposant sur des phones connues pour les enfants; et à déterminer une probabilité d'autisme sur la base de la comparaison.
PCT/US2015/028278 2014-04-29 2015-04-29 Système et procédé pour l'évaluation du langage expressif, du trouble du développement et des émotions WO2015168274A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/265,188 2014-04-29
US14/265,188 US9355651B2 (en) 2004-09-16 2014-04-29 System and method for expressive language, developmental disorder, and emotion assessment

Publications (1)

Publication Number Publication Date
WO2015168274A1 true WO2015168274A1 (fr) 2015-11-05

Family

ID=53373537

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/028278 WO2015168274A1 (fr) 2014-04-29 2015-04-29 Système et procédé pour l'évaluation du langage expressif, du trouble du développement et des émotions

Country Status (1)

Country Link
WO (1) WO2015168274A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107452401A (zh) * 2017-05-27 2017-12-08 北京字节跳动网络技术有限公司 一种广告语音识别方法及装置
CN110473519A (zh) * 2018-05-11 2019-11-19 北京国双科技有限公司 一种语音处理方法及装置
US11164319B2 (en) 2018-12-20 2021-11-02 Smith & Nephew, Inc. Machine learning feature vector generator using depth image foreground attributes
CN113900617A (zh) * 2021-08-03 2022-01-07 钰太芯微电子科技（上海）有限公司 具有声线接口的麦克风阵列系统及电子设备

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080235016A1 (en) * 2007-01-23 2008-09-25 Infoture, Inc. System and method for detection and analysis of speech
US20090208913A1 (en) * 2007-01-23 2009-08-20 Infoture, Inc. System and method for expressive language, developmental disorder, and emotion assessment

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080235016A1 (en) * 2007-01-23 2008-09-25 Infoture, Inc. System and method for detection and analysis of speech
US20090208913A1 (en) * 2007-01-23 2009-08-20 Infoture, Inc. System and method for expressive language, developmental disorder, and emotion assessment

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
K. A. LOVELAND; S. H. LANDRY, JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDERS, vol. 16, 1986, pages 335
MILLER, J.F.; CHAPMAN, R.S.: "The Relation between Age and Mean Length of Utterance in Morphemes", JOURNAL OF SPEECH AND HEARING RESEARCH, vol. 24, 1981, pages 154 - 161
P. MUNDY; C. KASARI; M. SIGMAN, INFANT BEHAVIOR AND DEVELOPMENT, vol. 15, 1992, pages 377
R. PAUL; A. AUGUSTYN; A. KLIN; F. R. VOLKMAR, JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDERS, vol. 35, 2005, pages 205
S. BARON-COHEN; J. J ALLEN; C. GILLBERG, THE BRITISH JOURNAL OF PSYCHIATRY, vol. 161, 1992, pages 839
S. J. SHEINKOPF; P. MUNDY; D. K. OLLER; M. STEFFENS, JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDERS, vol. 30, 2000, pages 345
W. PRONOVOST; M. P. WAKSTEIN; D. J. WAKSTEIN, EXCEPTIONAL CHILDREN, vol. 33, 1966, pages 19

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107452401A (zh) * 2017-05-27 2017-12-08 北京字节跳动网络技术有限公司 一种广告语音识别方法及装置
CN110473519A (zh) * 2018-05-11 2019-11-19 北京国双科技有限公司 一种语音处理方法及装置
US11164319B2 (en) 2018-12-20 2021-11-02 Smith & Nephew, Inc. Machine learning feature vector generator using depth image foreground attributes
US11688075B2 (en) 2018-12-20 2023-06-27 Smith & Nephew, Inc. Machine learning feature vector generator using depth image foreground attributes
CN113900617A (zh) * 2021-08-03 2022-01-07 钰太芯微电子科技（上海）有限公司 具有声线接口的麦克风阵列系统及电子设备
CN113900617B (zh) * 2021-08-03 2023-12-01 钰太芯微电子科技（上海）有限公司 具有声线接口的麦克风阵列系统及电子设备

Similar Documents

Publication Publication Date Title
US10573336B2 (en) System and method for assessing expressive language development of a key child
US8938390B2 (en) System and method for expressive language and developmental disorder assessment
US9355651B2 (en) System and method for expressive language, developmental disorder, and emotion assessment
US10223934B2 (en) Systems and methods for expressive language, developmental disorder, and emotion assessment, and contextual feedback
CA2712447C (fr) Systeme et procede d'evaluation d'un langage expressif
US8744847B2 (en) System and method for expressive language assessment
Alaie et al. Cry-based infant pathology classification using GMMs
Ramakrishnan Recognition of emotion from speech: A review
Friedland et al. Prosodic and other long-term features for speaker diarization
Busso et al. Iterative feature normalization scheme for automatic emotion detection from speech
An et al. Automatic recognition of unified parkinson's disease rating from speech with acoustic, i-vector and phonotactic features.
US11328738B2 (en) Systems and methods for automatic determination of infant cry and discrimination of cry from fussiness
Sztahó et al. Automatic estimation of severity of Parkinson's disease based on speech rhythm related features
WO2015168274A1 (fr) Système et procédé pour l'évaluation du langage expressif, du trouble du développement et des émotions
Chee et al. Overview of automatic stuttering recognition system
Fritsch Novel Methods For Detection And Analysis Of Atypical Aspects In Speech
Bharathi et al. Hybrid Approach for Analyzing Acute Spots of Clinical Speech Data Using Fuzzy Inference System
Weiand Unsupervised Speaker Visualization for Forensic Speaker Comparison
Jacob et al. An approach for Female Speech Emotion Recognition in the Indian context

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15728227

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15728227

Country of ref document: EP

Kind code of ref document: A1