KR102539191B1 - Cognitive state information providing method and electronic device for the same - Google Patents

Abstract

Disclosed are a cognitive status information providing method and an electronic apparatus therefor. According to the present disclosure, the cognitive status information providing method comprises the following steps of: providing a first page containing examination-related information; obtaining user voice data for the examination-related information while the first page is displayed; and providing a second page including first evaluation information obtained using a learning model set based on the user voice data. An objective of the present invention is to provide the cognitive status information providing method, which allows a user to conduct a cognitive status test and check the evaluation information therefor, and the electronic apparatus therefor.

Description

Cognitive state information providing method and electronic device therefor

The present disclosure relates to a method for providing information related to a cognitive state and an electronic device therefor.

Dementia refers to the loss of a person's mental abilities and ability to perform social activities, and when it is severe enough to cause impairment in a person's daily life, we call it dementia. It is known that the incidence of dementia also increases as people age, and accordingly, in today's society where life expectancy increases and rapid aging progresses, interest in methods for treating and preventing dementia is increasing.

Dementia is a pathological symptom caused by various diseases, and the symptoms of dementia vary greatly depending on the type and severity of the underlying disease. One of the representative symptoms of dementia is a symptom of cognitive decline, and includes memory decline, language ability decline, time and space understanding ability decline, judgment and daily life performance decline, and the like. One method for diagnosing dementia is a voice test to check language ability. The use of voice to screen for cognitive disorders has been verified and tried in many studies. In the case of dementia, Parkinson's disease, etc., linguistic and phonetic degradation from speech and conversation are often used as early detection signals. In this way, voice, especially free speech, which can find not only acoustic features but also linguistic and semantic features, is widely used in screening for cognitive disorders.

Meanwhile, the speaker's voice data may include speaker identification information related to the speaker's personal characteristics (age, gender, timbre, etc.) and content identification information related to characteristics directly related to voice (rhythm, silence rate, etc.). If only content identification information with speaker identification information removed from voice data is extracted and used in the diagnosis of dementia, the accuracy of cognitive disorder screening can be improved by focusing on more important factors such as voice rhythm, degree of interruption, silencing rate, and specificity. is expected to increase Therefore, there is a need for a method of providing information related to a cognitive state by extracting content identification information from speaker's voice data.

The present disclosure has been proposed to solve the above-described problems, and an object of the present disclosure is to provide a cognitive state information providing method and an electronic device therefor, which allow a cognitive state examination to be performed and evaluation information thereof to be checked.

The technical problem to be achieved by the present disclosure is not limited to the above-described technical problem, and other technical problems may be inferred from the following embodiments.

A method for providing cognitive state information according to an embodiment disclosed herein includes providing a first page including test-related information, acquiring user voice data for the test-related information while the first page is displayed, and and providing a second page including first evaluation information obtained by using a learning model set based on user voice data.

The first page may include a first area including the test-related information and a second area including an icon indicating a state related to voice input.

The user voice data may be obtained according to an input corresponding to the icon while the first page is displayed.

The icon of the second area may be displayed as a first image corresponding to the confirmation of the voice input, and may be displayed as a second image corresponding to the absence of the confirmation of the voice input.

The icon of the second area may be displayed as a third image when a time period in which the voice input is not confirmed passes a reference time period.

The first image may be a dynamic image moving in response to confirmation of the voice input, and the second image and the third image may be static images.

The inspection-related information includes first inspection information including image information and first text information requesting an explanation of the image information, second text information on a specific subject, and requesting an explanation of the second text information. It may include at least one of second test information including third text information.

The user voice data includes at least one of first voice data obtained with respect to the first test information and second voice data obtained with respect to the second test information, and the first voice data and the second voice data are obtained. Data can be analyzed by different learning models.

An upper limit of the length of the first voice data may be different from an upper limit of the length of the second voice data.

When the obtained user voice data satisfies the set condition, information on completion of recording may be displayed, and when the obtained user voice data does not satisfy the set condition, information on a re-recording request may be displayed.

The second page may further include second evaluation information obtained in a different way from the first evaluation information.

Each of the first evaluation information and the second evaluation information may be provided as a numerical score.

The second page may further include recommendation information for guiding at least one of retest, additional test, cognitive training, and transfer to a hospital when a score obtained by digitizing the first evaluation information is equal to or greater than a reference value.

Acquiring check list information before testing, providing a third page including at least one piece of test information, obtaining test voice data corresponding to the test information while the third page is displayed; and The method may further include verifying voice input based on the test voice data.

The pre-examination checklist information may include information on at least one of whether the user is awake, surrounding environment, examination period, and re-examination.

An electronic device for a method for providing cognitive state information according to an embodiment disclosed herein provides a memory for storing at least one command, and a first page including test-related information based on the at least one command, the A processor that obtains user voice data for the test-related information while a first page is displayed, and provides a second page including first evaluation information obtained by using a learning model set based on the user voice data can do.

Details of other embodiments are included in the detailed description and drawings.

According to the present disclosure, through a page provided by a user terminal and text information and image information of the page, a user can easily and quickly perform a cognitive state test, and can easily and quickly check evaluation information for the cognitive state, through which any follow-up measures can be taken. You can check if you need it.

Effects of the invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a conceptual diagram illustrating a device in which a method of providing cognitive state information in an electronic device according to embodiments of the present disclosure is implemented.
2 is a schematic block diagram illustrating an electronic device and a user terminal according to embodiments of the present disclosure.
3A and 3B are flowcharts illustrating a method for providing cognitive state information according to embodiments of the present disclosure.
4A to 4G are diagrams for describing a page including first inspection information provided to a user terminal according to embodiments of the present disclosure.
5A to 5F are diagrams for describing a page including second examination information provided to a user terminal according to embodiments of the present disclosure.
6A to 6E are diagrams for describing a page including evaluation information provided to a user terminal according to embodiments of the present disclosure.
7A to 7C are diagrams for describing pages for obtaining pre-test check list information and test voice data according to embodiments of the present disclosure.

The terms used in the embodiments have been selected as general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

In the entire specification, when a part is said to "include" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated. In addition, terms such as "...unit" and "...module" described in the specification mean a unit that processes at least one function or operation, which is implemented as hardware or software, or a combination of hardware and software. It can be.

The expression of “at least one of a, b, and c” described throughout the specification means 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c' ', or 'all of a, b, and c'.

A “terminal” referred to below may be implemented as a computer or portable terminal capable of accessing a server or other terminals through a network. Here, the computer includes, for example, a laptop, desktop, laptop, etc. equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution), etc. It may include a handheld-based wireless communication device.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure will be omitted. This is to more clearly convey the gist of the present disclosure without obscuring it by omitting unnecessary description.

For the same reason, in the accompanying drawings, some components are exaggerated, omitted, or schematically illustrated. Also, the size of each component does not entirely reflect the actual size. In each figure, the same reference number is assigned to the same or corresponding component.

Advantages and features of the present disclosure, and methods for achieving them, will become clear with reference to embodiments described below in detail in conjunction with the accompanying drawings. However, the present disclosure is not limited to the following embodiments and may be implemented in various different forms, but the present disclosure is intended to make the present disclosure complete, and to those skilled in the art to which the present disclosure belongs. It is provided to fully inform the scope of, and this disclosure is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

At this time, it will be appreciated that each block of the process flow diagrams and combinations of the flow diagrams can be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, so that the instructions executed by the processor of the computer or other programmable data processing equipment are described in the flowchart block(s). It creates means to perform functions. These computer program instructions may also be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular way, such that the computer usable or computer readable memory It is also possible for the instructions stored in to produce an article of manufacture containing instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to generate computer or other programmable data processing equipment. Instructions for performing processing equipment may also provide steps for performing the functions described in the flowchart block(s).

Further, each block may represent a module, segment or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative implementations it is possible for the functions mentioned in the blocks to occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on their function.

1 is a conceptual diagram illustrating a device in which a method of providing cognitive state information in an electronic device according to embodiments of the present disclosure is implemented.

Referring to FIG. 1 , a method for providing cognitive state information according to embodiments of the present disclosure may be implemented in various types of devices. For example, a method for providing cognitive state information according to embodiments of the present disclosure may be implemented through the electronic device 100 and the user terminal 110 interoperating with the electronic device 100 . In other words, the electronic device 100 and the user terminal 110 interoperating with the electronic device 100 may perform operations according to embodiments of the present disclosure based on information providing methods implemented in each. Meanwhile, the method for providing cognitive state information according to embodiments of the present disclosure is not limited to that shown in FIG. 1 and may be implemented in various electronic devices or servers.

The electronic device 100 according to various embodiments may be a device that performs wired or wireless communication with the user terminal 110 interworking with the electronic device 100 and includes a memory having a large storage capacity. For example, the electronic device 100 may be a cloud device connected to the user terminal 110 . In other words, the method for providing cognitive state information according to embodiments of the present disclosure may be implemented on a logical device of a cloud device.

The user terminal 110 that works with the electronic device 100 according to various embodiments may be a device that can be used by a service user, such as a desktop PC, a tablet PC, or a mobile terminal. In addition, other electronic devices performing similar functions may be used as the user terminal 110 interworking with the electronic device 100 .

Various modules for operations according to embodiments of the present disclosure may be provided. The modules may be computer codes implemented so that a physical device (eg, the electronic device 100) in which an information providing method is implemented (or included in the physical device) may perform a designated operation or one or more instructions. In other words, the physical device in which the information providing method is implemented stores a plurality of modules in a memory in the form of computer code, and when the plurality of modules stored in the memory are executed, the plurality of modules are designated by the physical device corresponding to the plurality of modules. actions can be performed.

2 is a schematic block diagram illustrating an electronic device and a user terminal according to embodiments of the present disclosure.

Referring to FIG. 2 , each of an electronic device 100 and a user terminal 110 interworking with the electronic device 100 may include a communication unit 210 , a memory 220 and a processor 230 .

According to embodiments, each of the electronic device 100 and the user terminal 110 interworking with the electronic device 100 may further include an input unit and an output unit. Each of the input unit and the output unit may be various interfaces or connection ports that receive user input or output information to the user.

The input unit may include an input module, and the input module receives a user input from a user. User input may be made in various forms including key input, touch input, and voice input. Examples of input modules capable of receiving such user input include a traditional keypad, keyboard, and mouse, as well as a touch sensor that detects a user's touch, a microphone that receives a voice signal, a camera that recognizes gestures through image recognition, A proximity sensor including at least one of an illuminance sensor and an infrared sensor for detecting user approach, a motion sensor for recognizing a user's motion through an acceleration sensor or a gyro sensor, and various other types of sensors that detect or receive user input. There is an input means, and the input module according to an embodiment of the present disclosure may include at least one of the devices listed above. Here, the touch sensor may be implemented as a piezoelectric or capacitive touch sensor that detects a touch through a touch panel attached to a display panel or a touch film, an optical touch sensor that detects a touch by an optical method, and the like. In addition, the input module may be implemented in the form of an input interface (USB port, PS/2 port, etc.) connecting an external input device that receives a user input instead of a device that detects a user input by itself.

Meanwhile, the output unit may include an output module, and the output module may output various types of information. The output module may include at least one of a display for outputting an image, a speaker for outputting sound, a haptic device for generating vibration, and other various types of output means. In addition, the output module may be implemented in the form of a port type output interface connecting the above-described individual output means. For example, the output module in the form of a display can display text, still images, and moving images. The display includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flat panel display (FPD), and a transparent display. display), curved display, flexible display, 3D display, holographic display, projector, and other various types of devices capable of performing image output functions. may contain at least one. Such a display may be in the form of a touch display integrally formed with the touch sensor of the input module.

The communication unit 210 may communicate with other devices. Accordingly, each of the electronic device 100 and the user terminal 110 interworking with the electronic device 100 may transmit and receive information with other devices through the communication unit 210 . For example, the electronic device 100 and each of the user terminals 110 interworking with the electronic device 100 may communicate with each other or with other devices using the communication unit 210 .

Here, communication, that is, transmission and reception of data may be performed by wire or wireless. To this end, the communication unit 210 includes a wired communication module that accesses the Internet through a LAN (Local Area Network), a mobile communication module that accesses a mobile communication network via a mobile communication base station to transmit and receive data, and a WLAN such as Wi-Fi. (Wireless Local Area Network)-based communication method, short-distance communication module using WPAN (Wireless Personal Area Network)-based communication method such as Bluetooth or Zigbee, GNSS (Global Positioning System) such as GPS It can be composed of a satellite communication module using a Navigation Satellite System) or a combination thereof.

The memory 220 may store various kinds of information. The memory 220 may temporarily or semi-permanently store data. For example, the memory 220 of the electronic device 100 includes an Operating System (OS) for driving the electronic device 100, a program or application for generating data or Braille characters for hosting a website (e.g. For example, data related to a web application) may be stored. Also, the memory 220 may store the modules in the form of computer codes as described above.

Examples of the memory 220 include a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a read-only memory (ROM), a random access memory (RAM), and the like. This can be. This memory 220 may be provided in a built-in type or a detachable type.

The processor 230 controls the overall operation of each of the electronic device 100 and the user terminal 110 interworking with the electronic device 100 . To this end, the processor 230 may perform calculations and processing of various types of information and control operations of components of the electronic device 100 . For example, the processor 230 may execute a program or application for providing delivery-related information. The processor 230 may be implemented as a computer or a similar device according to hardware, software, or a combination thereof. In terms of hardware, the processor 230 may be implemented in the form of an electronic circuit that performs a control function by processing electrical signals, and in terms of software, it may be implemented in the form of a program that drives the processor 230 in hardware. Meanwhile, in the following description, unless otherwise noted, the operation of the electronic device 100 and the user terminal 110 interoperating with the electronic device 100 may be interpreted as being performed under the control of the processor 230 . That is, when the modules implemented in the delivery-related information providing method according to the embodiments of the present disclosure are executed, the modules include the processor 230, the electronic device 100, and the user terminal 110 interworking with the electronic device 100. It can be interpreted as controlling to perform the steps of the delivery related information providing method.

In summary, various embodiments may be implemented through various means. For example, various embodiments may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to various embodiments includes one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs ( field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc.

In the case of implementation by firmware or software, the method according to various embodiments may be implemented in the form of a module, procedure, or function that performs functions or operations described below. For example, software codes can be stored in memory and run by a processor. The memory may be located inside or outside the processor, and may exchange data with the processor by various means known in the art.

Hereinafter, various embodiments will be described in more detail based on the above-described technical idea. The above-described contents may be applied to various embodiments described below. For example, operations, functions, and terms that are not defined in various embodiments described below may be performed and described based on the contents described above.

Hereinafter, various embodiments will be described on the premise that the user terminal 110 performs steps of a method for providing cognitive state information. According to various embodiments, the user terminal 110 performs steps and steps of a method for providing cognitive state information. Related information may be transmitted and received with the electronic device 100 .

3A is a flowchart illustrating a method for providing cognitive state information according to embodiments of the present disclosure.

Referring to FIG. 3A , in step S310, the user terminal may provide a first page including test-related information. The first page may further include an icon representing a state related to voice input. For example, the icon may be displayed as a different image depending on whether a voice input is confirmed.

The test-related information may be information related to a test that induces free speech, which is an analysis target of a learning model to be described later. The test-related information may include a plurality of pieces of test information. The test-related information may include, for example, first test information including image information and first text information requesting an explanation of the image information and a specific subject (eg, happy, sad, or yesterday's events). It may include at least one of second text information about the second text information and second inspection information including third text information requesting an explanation of the second text information. For example, the first test information may be information related to a 'Picture Description Test (PDT)', and the second test information may be information related to a 'Storytelling Test'. The type of inspection is only exemplary and the present disclosure is not limited thereto.

In step S320, the user terminal may acquire user voice data for examination-related information in a state in which the first page is displayed. User voice data may be obtained according to an input corresponding to an icon while the first page is displayed. The user voice data may include, for example, at least one of first voice data obtained for the first test information and second voice data obtained for the second test information. In this case, the first voice data and the second voice data may be analyzed by different learning models. However, this is only an example and the present disclosure is not limited thereto, and the first voice data and the second voice data may be analyzed by the same learning model.

For example, the length of the first voice data and the length of the second voice data may be different. Also, each of the first voice data and the second voice data may have at least one of a lower limit and an upper limit in length. In this case, the upper limit of the length of the first voice data may be different from the upper limit of the length of the second voice data. For example, the upper limit of the length of the first voice data may be greater than the upper limit of the length of the second voice data. According to embodiments, when the length of the user voice data has an upper limit, step S320 may automatically end when the length of the user voice data becomes equal to the upper limit.

According to embodiments, the user terminal may display information about completion of recording when the user voice data obtained in step S320 satisfies a set condition. Also, according to embodiments, the user terminal may display information on a re-recording request when the user voice data obtained in step S320 does not satisfy a set condition. Each of information on completion of recording and information on request for re-recording may be provided on the first page or on a separate page different from the first page. At this time, the set condition for determining which information to display is whether or not voice input is confirmed, whether the length of the user voice data satisfies the lower and upper limits, and the sum of the lengths of the sections in which the voice input is confirmed among the user voice data is the reference value. It may include at least one of whether or not there is an anomaly (ie, whether the user voice data includes a utterance amount that can be substantially analyzed).

In step S330, the user terminal may provide a second page including first evaluation information obtained using a learning model set based on user voice data. According to embodiments, an electronic device that works with a user terminal may analyze user voice data with a set learning model. The second page may further include second evaluation information for at least one indicator related to user voice data.

The second evaluation information may be obtained in a different way from the first evaluation information. According to embodiments, an index of the second evaluation information may be input by an evaluator based on at least a part of user voice data for deriving the first evaluation information. In this case, the method for providing cognitive state information according to embodiments of the present disclosure may further include, between steps S320 and S330, the user terminal receiving information on an indicator of the second evaluation information input by the evaluator. there is.

In addition, it may be displayed on a second page in a form different from that of the first evaluation information or on a separate page connected through link information of the second page. For example, the first evaluation information may be displayed as an image capable of intuitively grasping the user's cognitive state, and the second evaluation information may be at least one indicator related to user voice data and a description of the indicator. Can be displayed as text.

The user voice data may include, for example, sound domain information related to the amount and rate of speech itself (eg, speech-silence ratio, whether there is an outlier during speech), linguistic meaning of speech content (eg, speech-silence ratio). information in linguistic domains related to the meaning of words, connection or complexity of sentences, number of unique words), and semantics related to whether utterances can be connected in a coherent way (for example, how coherently they describe a given topic). It can contain information about the area. The indicators related to the user voice data may include indicators related to each of the acoustic region, the language region, and the semantic region.

Indices related to the acoustic region include, for example, the total utterance volume (ie, the sum of the lengths of the intervals in which voice input is confirmed), the utterance rate (ie, the ratio of the intervals in which voice input is confirmed among the entire voice data), the silence rate ( That is, it may include the ratio of sections in which voice input is not confirmed among all voice data) and speech specificity (ie, the length or ratio of unusually short or long sections in all voice data). Speech specificity may be determined based on the average length of a plurality of sections in which speech input is continuous. For example, a section having a length of less than half of the average length may be determined as an unusually short section, and a section having a length of more than twice the average length may be determined as an unusually long section.

Indicators related to the language domain include, for example, noun ratio (i.e., the ratio of nouns to total words), sentence complexity (i.e., the degree to which a sentence is composed in a complex way), and linking word ratio (i.e., linking words and conjunctions in total words). ratio) and the number of unique words (ie, the number of unique words that are newly spoken without overlapping). Sentence complexity may be determined based on at least one of the number of words forming a sentence and the number of different parts of speech forming a sentence.

Indicators related to the semantic domain may include overall consistency (ie, consistency between the presented topic and the entire content of the speech data) and sentence consistency (ie, consistency between sentences of the speech data).

Each of the first evaluation information and the second evaluation information may be provided as a numerical score. According to embodiments, the digitized score may be provided after additional normalization. The first evaluation information may be provided as, for example, a numerical score between 0 and 1 or a numerical score between 0% and 100%. Among the second evaluation information, at least one indicator related to user voice data may be provided as a numerical score between 0 and 100 points, for example. However, this is merely an example, and the unit and range of the digitized score are not limited thereto.

According to embodiments, the second page is a retest using similar test-related information when the score obtained by digitizing the first evaluation information is equal to or higher than the reference value, an additional test through video call and consultation with a professional counselor, cognitive training with difficulty selection, and Recommendation information for guiding at least one follow-up action during transfer to the hospital may be further included.

According to the present disclosure, a user can easily and quickly perform a cognitive state test through a first page and text information and image information of the first page provided by a user terminal, and can easily and quickly check evaluation information for this on a second page. and, through this, it is possible to determine what follow-up measures are required.

In step S330, in more detail, obtaining a speaker identification embedding based on user voice data using a first encoder, and obtaining content embedding based on the user voice data and speaker identification embedding using a second encoder. The method may include acquiring, identifying first evaluation information based on content embedding using a target model, and providing a second page including the first evaluation information.

Acquiring the speaker identification embedding includes identifying a partial speech data set including at least one partial speech data based on the user speech data, and obtaining the speaker identification embedding based on the partial speech data set using a first encoder. steps may be included. Speaker identification embedding may include timbre information included in user voice data.

Obtaining the content embedding may include concatenating the user voice data and the speaker identification embedding and inputting the concatenated data to a second encoder. Content embedding may include rhythm information, pitch information, and content information included in user voice data.

The second encoder may be learned based on a comparison of user voice data and output voice data generated by decoding the speaker identification embedding using the second decoder corresponding to the content embedding and the second encoder. The second encoder and the second decoder may be trained to minimize a difference between output voice data and user voice data.

The first encoder may include at least one of a Long Short Term Memory (LSTM) and a Rectified Linear Unit (ReLU), and the second encoder may include at least one of a convolutional layer, a bidirectional LSTM, and a downsampling unit. may contain one.

The target model may include a fully connected layer. The fully connected layer may be learned based on a plurality of content embeddings and a label corresponding to each of the plurality of content embeddings. The first evaluation information may be provided as, for example, a score associated with a mini-mental state examination (MMSE).

According to the present disclosure, a user's cognitive state can be automatically determined from user voice data. More specifically, information related to a user's cognitive state can be quickly and accurately determined based on content embedding obtained by separating speaker identification embedding from user voice data, and the user terminal can provide the information related to the cognitive state to the user.

3B is a flowchart illustrating a method for providing cognitive state information according to embodiments of the present disclosure.

Referring to FIG. 3B , in step S340, the user terminal may obtain checklist information before examination. The pre-examination checklist may include information on at least one of whether the user is awake, surrounding environment, examination period, and re-examination. The accuracy of the user voice data input to the user terminal and the reliability of the judgment based on the user voice data can be improved by checking several items before the test. According to embodiments, the user terminal may provide a pop-up guide related to the voice recording authority for the test to proceed after step S340.

In step S350, the user terminal may provide a third page including at least one piece of test information. The third page may further include an icon representing a state related to voice input. For example, the icon may be displayed as a different image depending on whether a voice input is confirmed. The test inspection information may be, for example, information related to a test inducing free speech that is an analysis target of a learning model. For example, test information may be test information including text information on a specific subject (eg, animal name).

In step S360, the user terminal may acquire test voice data corresponding to the test examination information while the third page is displayed. Test voice data may be obtained according to an input corresponding to an icon while the third page is displayed. The upper limit of the length of the test voice data may be smaller than the lower limit (eg, about 2 minutes) of the first voice data described with reference to FIG. 3A . The upper limit of the length of the test voice data may be, for example, about 10 seconds or less.

In step S370, the user terminal can check the voice input based on the test voice data. The test voice data is not an analysis target of the learning model, and the user terminal can check whether the voice is properly input based on the test voice data. After step S370, step S310 described with reference to FIG. 3A may be performed.

4A to 4G are diagrams for describing a page including first inspection information provided to a user terminal according to embodiments of the present disclosure. In this case, the first inspection information is inspection information including image information and first text information requesting an explanation of the image information among the inspection-related information described with reference to FIG. 3A .

Referring to FIG. 4A , a first page 410 guiding first inspection information is shown. The first test information is information related to the first test (eg, Picture Description Test (PDT)). Through the first page 410, the user can check how to proceed with the first examination, including a method of inputting a voice for the first examination information. In particular, through the first page 410, the user can check the lower limit (eg, 2 minutes) and upper limit (eg, 10 minutes) of the first voice data obtained for the first test information. The first page 410 may include a 'test start' icon 411 at the bottom.

Referring to FIGS. 4B, 4C, 4D, and 4E, a second page 420 provided to the user terminal according to an input corresponding to the 'test start' icon 411 described with reference to FIG. 4A is illustrated. The second page 420 may correspond to the first page including test-related information described with reference to FIG. 3A .

The second page 420 may include a first area 421 including first inspection information and a second area 422 including a first icon 4221 . The first inspection information may include image information 4211 and first text information 4212 requesting an explanation of the image information 4211 . The image information 4211 of the first inspection information includes, for example, at least one object (eg, 'candy on the shelf', 'window with curtains installed', 'sink overflowing with water'). At least one person performing at least one action (e.g., 'taking out the candy', 'pointing to the shelf', 'wiping the dishes') in the space (eg, 'moving next to the chair') 'Girl', 'Boy on a chair', 'Woman in front of a sink'). According to embodiments, when a plurality of image information 4211 of the first examination information is provided, a plurality of second pages 420 corresponding to the plurality of image information 4211 may be sequentially provided to the user terminal.

The first text information 4212 of the first inspection information may include, for example, a phrase such as 'Please feel free to talk about the picture for 2 to 10 minutes.' According to embodiments, a graphical user interface (GUI) in the form of a progress bar indicating a time when a voice input has been performed may be provided on the image information 4211 of the first area 421 .

Referring to FIG. 4B , the first icon 4221 of the second region 422 may have, for example, a red circular shape. A 'recording start' phrase may be provided below the first icon 4221 . The first voice data for the first test information may be obtained according to an input corresponding to at least one of the first icon 4221 and the phrase 'start recording' while the second page 420 is displayed.

Referring to FIG. 4C , the first icon 4221 of the second area 422 may be displayed as a first image IMa having a human face shape in response to confirmation of voice input. The first image IMa may be, for example, a dynamic image (ie, animation) that moves (eg, a person's face in a circular shape nods) in response to confirmation of a voice input. At this time, 'I'm listening!' A phrase may be provided. Also, a second icon 4222 may be additionally provided under the first icon 4221 . The second icon 4222 may have, for example, a shape of a rectangle within a circle rim. Under the second icon 4222, a phrase 'end recording' may be provided. The process of obtaining the first voice data for the first test information may be terminated according to an input corresponding to at least one of the second icon 4222 and the phrase 'end recording'.

Referring to FIG. 4D , the first icon 4221 of the second area 422 corresponds to the fact that the previously confirmed voice input is not confirmed, and instead of the first image IMa, which is a dynamic image described with reference to FIG. 4C , a smiling expression. may be displayed as a second image IMb having a human face shape of . At this time, 'I'm listening!' A phrase may be provided. Also, referring to FIG. 4E , the first icon 4221 of the second region 422 shows a surprised expression when the time period in which the voice input is not confirmed has elapsed by a reference time (eg, 5 seconds to 1 minute). It may be displayed as a third image IMc having a human face shape. At this time, 'I can't hear the sound! A 'Please tell me' phrase may be provided. Unlike the first image IMa, the second image IMb and the third image IMc may be static images that do not move.

Referring to FIGS. 4F and 4G , a third page 430 provided to the user terminal when first voice data is acquired according to an input corresponding to the first icon 4221 described with reference to FIGS. 4B to 4E , and A fourth page 440 is shown respectively.

Referring to FIG. 4F , the third page 430 may include information about completion of recording displayed when the first voice data satisfies a set condition. More specifically, the third page 430 is 'Describe the picture inspection complete!' Text information 431 including a phrase, image information 432 having a human face with a smiling expression, and 'Yes. OK' icon 433 may be included.

Referring to FIG. 4G , a fourth page 440 may include information on a re-recording request displayed when the first voice data does not satisfy a set condition. More specifically, the fourth page (440) is 'recording was not done' at the top. It may include text information 441 including a phrase, image information 442 having a human face with a worried expression, and a 'do it again' icon 443. According to an input corresponding to the 'play again' icon 443, the first page 410 described with reference to FIG. 4A or the second page 420 described with reference to FIG. 4B may be provided to the user terminal again.

5A to 5F are diagrams for describing a page including second examination information provided to a user terminal according to embodiments of the present disclosure. In this case, the second examination information is examination information including second text information on a specific subject among the examination-related information described with reference to FIG. 3A and third text information requesting an explanation of the second text information. .

Referring to FIG. 5A , a first page 510 guiding second inspection information is shown. The second test information is information related to a second test (eg, a storytelling test). Through the first page 510, the user can check how to proceed with the second examination, including a method of inputting a voice for the second examination information. In particular, through the first page 510, the user can check the upper limit (eg, 1 minute) of the second voice data acquired for the second examination information. The first page 510 may include a 'test start' icon 511 at the bottom.

According to embodiments, 'Yes. According to an input corresponding to the 'I understand' icon 433 , the first page 510 may be provided to the user terminal. This assumes that the second inspection is performed after the first inspection, but the present disclosure is not limited thereto, and the first inspection is performed after the second inspection, as described with reference to FIG. 4F 'Yes. According to an input corresponding to the 'I understand' icon 433, a page including first evaluation information obtained by using a learning model set based on user voice data may be provided to the user terminal.

Referring to FIGS. 5B, 5C, and 5D, second pages 520a, 520b, and 520c provided to the user terminal according to an input corresponding to the 'test start' icon 511 described with reference to FIG. 5A are illustrated. . The second pages 520a, 520b, and 520c may correspond to the first page including test-related information described with reference to FIG. 3A.

The second pages 520a, 520b, and 520c may include a first area 521 including second examination information and a second area 422 including a first icon 5221 . The second examination information may include second text information 5211 on a specific subject and third text information 5212 requesting an explanation of the second text information 5211 . The second text information 5211 of the second examination information may include, for example, one of 'happy things', 'sad things', and 'what happened yesterday'. According to embodiments, when a plurality of specific subjects of the second examination information are provided, a plurality of second pages 520a, 520b, and 520c corresponding thereto may be sequentially provided to the user terminal.

The third text information 5212 of the second examination information may include, for example, a phrase such as 'Please feel free to tell me about a happy event (or a sad event, or an event that happened yesterday) for 1 minute.' According to embodiments, a GUI in the form of a progress bar indicating a time when a voice input progresses may be provided to a portion of the first region 521 .

The second area 522 of the second pages 520a, 520b, and 520c and the first icon 5221 of the second area 522 are the second pages described with reference to FIGS. 4B, 4C, 4D, and 4E. The second area 422 of 420 and the first icon 4221 of the second area 422 may be substantially the same. Second voice data for the second examination information may be obtained according to an input corresponding to at least one of the first icon 5221 and the phrase 'start recording' while the second pages 520a, 520b, and 520c are displayed. . In addition, the first icon 5221 of the second area 522 corresponds to each other when the voice input is confirmed, when the voice input is not confirmed, and when the time when the voice input is not confirmed passes the reference time. It may be displayed as a different image.

Referring to FIGS. 5E and 5F , a third page 530 provided to the user terminal when second voice data is obtained according to an input corresponding to the first icon 5221 described with reference to FIGS. 5B to 5D , and A fourth page 540 is shown respectively.

Referring to FIG. 5E , the third page 530 may include information about completion of recording displayed when the second voice data satisfies a set condition. More specifically, the third page (530) at the top of 'happy event (or sad event, yesterday's event) recording complete!' Text information 531 including a phrase, image information 532 having a human face with a smiling expression, and an 'I understand' icon 533 may be included.

Referring to FIG. 5F , a fourth page 540 may include information on a re-recording request displayed when the second voice data does not satisfy a set condition. More specifically, the 4th page (540) is 'not recorded' at the top. Text information 541 including a phrase, image information 542 having a human face with a worried expression, and a 'do it again' icon 543 may be included. According to an input corresponding to the 'play again' icon 543, the first page 510 described with reference to FIG. 5A or the second page 520a described with reference to FIG. 5B may be provided to the user terminal again.

6A to 6E are diagrams for describing a page including evaluation information provided to a user terminal according to embodiments of the present disclosure.

Referring to FIGS. 6A, 6B, and 6C, first pages 610a, 610b, and 610c including first evaluation information obtained using a learning model set based on user voice data are shown. The first pages 610a, 610b, and 610c may correspond to second pages including the first evaluation information described with reference to FIG. 3A.

The first pages 610a, 610b, and 610c include analysis result text information 611, analysis result image information having a human face shape 612, 'go to see more detailed results' icon 613, and 'confirm' icon 614 ) and a 'recheck' icon 615.

The first evaluation information may be provided as a numerical score between 0% and 100%, for example. For example, when the first evaluation information is 0% or more and 50% or less, the cognitive state may be determined to be 'normal'. Also, for example, when the first evaluation information is greater than 50% and less than or equal to 75%, the cognitive state may be determined as 'attention'. Also, for example, when the first evaluation information is greater than 75% and less than or equal to 100%, the cognitive state may be determined to be 'dangerous'.

If the cognitive status is 'normal', the analysis result text information (611) is 'Hong Gil-dong's cognitive status is normal.' phrase, and the analysis result image information 612 may have a human face shape with a smiling expression. At this time, the first page 610a, for example, provided under the analysis result image information 612, 'Hong Gil-dong's cognitive status is a healthy level compared to the same gender, age group, and educational level. It may further include recommendation information 616 including an explanatory phrase such as 'I hope you maintain a healthy cognitive state by steadily improving your current condition.'

If the cognitive state is 'attention', the text information 611 as a result of the analysis is 'Hong Gil-dong's cognitive status is caution.' phrase, and the analysis result image information 612 may have a human face shape with a worried expression. At this time, the first page 610b, for example, provided under the analysis result image information 612, 'Hong Gil-dong's cognitive status is at a slightly lower level compared to the same gender, age group, and educational level. Recommendation information 616 including an explanatory phrase such as 'I am concerned about mild cognitive impairment, which is the previous stage of dementia, so I recommend that you check more detailed results through additional tests.'

If the cognitive state is 'danger', the analysis result text information (611) is 'Hong Gil-dong's cognitive status is dangerous.' phrase, and as a result of the analysis, the image information 612 may have a human face shape with a frowning expression. At this time, the first page 610c, for example, provided under the analysis result image information 612, 'Hong Gil-dong's cognitive status is at a markedly lower level compared to the same gender, age group, and educational level. Recommendation information 616 including an explanatory phrase such as, 'Check for more detailed results through additional tests or recommend hospital treatment.'

Referring to FIG. 6D , a second page 620 including second evaluation information for at least one indicator related to user voice data is shown. The second page 620 may be provided to the user terminal according to an input corresponding to the 'Go to see more detailed results' icon 613 described with reference to FIGS. 6A, 6B, and 6C. According to embodiments, the second page 620 may be provided together on the same page as the first pages 610a, 610b, and 610c. That is, the second page 620 may correspond to a part of the second page including the first evaluation information described with reference to FIG. 3A.

The second page 620 includes a first area 621 including comprehensive analysis result information on information in the acoustic domain, information in the language domain, and information in the semantic domain, indicators related to the acoustic domain (eg, 'total A second area 622 including information on utterance amount, utterance rate, silence rate, and utterance specificity') and indicators related to language areas (eg, 'noun ratio, sentence complexity, linking word ratio, number of unique words') ') and a fourth area 624 including information on indicators related to the semantic domain (eg, 'overall consistency, sentence consistency'). there is.

Comprehensive analysis result information of the first area 621 is provided as, for example, a radial chart (ie, a radar chart) based on at least one of the analysis result for each area and the average analysis result of the same age group as the user. It can be.

Information on indicators related to each area may be provided as a numerical score between 0 and 100 points, for example. The digitized scores for each of the indices may be displayed in a GUI in the form of a bar graph, and at this time, an average score of the same age group as the user may be displayed on the bar graph.

Each of the second area 622, the third area 623, and the fourth area 624 is displayed in the 'What does each area mean?' An icon 625 may be further included. According to embodiments, the first to fourth areas 621, 622, 623, and 624 of the second page 620 are sequentially displayed on the user terminal according to an input for scrolling a portion of the second page 620. It can be. According to other embodiments, the first to fourth areas 621 , 622 , 623 , and 624 of the second page 620 may be provided on different pages.

Referring to FIG. 6E, 'What does each region mean?' described with reference to FIG. 6D. A third page 630 provided to the user terminal according to the input corresponding to the icon 625 is shown. The third page 630 includes text information 631 describing indicators related to the acoustic domain, text information 632 describing indicators related to the language domain, and text describing indicators related to the semantic domain. information 633. According to embodiments, the text information 631 , 632 , and 633 of the third page 630 may be sequentially displayed on the user terminal according to an input for scrolling a portion of the third page 630 . According to other embodiments, the text information 631, 632, and 633 of the third page 630 may be provided on different pages.

7A to 7C are diagrams for describing pages for obtaining pre-test check list information and test voice data according to embodiments of the present disclosure.

Referring to FIG. 7A , a first page 710 for guiding check list information before examination is shown. Through the first page 710, the user can check necessary situations and conditions before proceeding with the cognitive state test. The pre-examination checklist information may include information on at least one of whether the user is in an awake state, the surrounding environment, an examination period, and whether or not the examination is re-examination. The first page 710 may include a 'confirmed' icon 711 at the bottom.

Referring to FIG. 7B , a second page 720 for guiding test inspection information is shown. The second page 720 may be provided to the user terminal according to an input corresponding to the 'confirmed' icon 711 described with reference to FIG. 7A . Through the second page 720, the user can confirm a method of proceeding with the test including a method of inputting a voice for the test information. In particular, through the first page 720, the user can check the upper limit (eg, 10 seconds) of test voice data acquired for the test examination information. The second page 720 may include a 'practice test start' icon 721 at the bottom.

Referring to FIG. 7C , a third page 730 provided to the user terminal according to an input corresponding to the 'start practice test' icon 721 described with reference to FIG. 7B is shown. The third page 730 may correspond to the third page described with reference to FIG. 3B .

The third page 730 may include a first area 731 including test information and a second area 732 including a first icon 7321 . The test information includes text information 7311 (for example, 'animal name') for a specific subject and text information 7312 requesting an explanation (for example, 'animal name'). Please feel free to tell me for a second.'). According to embodiments, when a specific subject of test information is provided in plural, a plurality of third pages 730 corresponding to the specific subject may be sequentially provided to the user terminal. According to embodiments, a GUI in the form of a progress bar indicating a time when a voice input progresses may be provided to a portion of the first area 731 .

The second area 732 of the third page 730 and the first icon 7321 of the second area 732 are of the second page 420 described with reference to FIGS. 4B, 4C, 4D, and 4E. It may be substantially the same as the second area 422 and the first icon 4221 of the second area 422 . Test voice data for the test examination information may be obtained according to an input corresponding to at least one of the first icon 7321 and the phrase 'start recording' while the third page 730 is displayed. In addition, the first icon 7321 of the second area 732 corresponds to the case where the voice input is confirmed, the voice input is not confirmed, and the time when the voice input is not confirmed passes the reference time. It may be displayed as a different image. According to embodiments, when the test voice data is obtained according to the input corresponding to the first icon 7321, the third page 530 and the fourth page 540 described with reference to FIGS. 4F and 4G are provided to the user terminal. At least one of the substantially identical pages may be provided.

On the other hand, in the present specification and drawings, preferred embodiments of the present disclosure have been disclosed, and although specific terms have been used, they are only used in a general sense to easily describe the technical content of the present disclosure and help understanding of the present invention. It is not intended to limit the scope of the disclosure. In addition to the embodiments disclosed herein, it is obvious to those skilled in the art that other modified examples based on the technical spirit of the present disclosure may be implemented.

An electronic device or terminal according to the above-described embodiments includes a processor, a memory for storing and executing program data, a permanent storage unit such as a disk drive, a communication port for communicating with an external device, a touch panel, and a key. , user interface devices such as buttons, and the like. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on a processor. Here, the computer-readable recording medium includes magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM) ), and DVD (Digital Versatile Disc). A computer-readable recording medium may be distributed among computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

This embodiment can be presented as functional block structures and various processing steps. These functional blocks may be implemented with any number of hardware or/and software components that perform specific functions. For example, an embodiment is an integrated circuit configuration such as memory, processing, logic, look-up table, etc., which can execute various functions by control of one or more microprocessors or other control devices. can employ them. Similar to components that can be implemented as software programming or software elements, the present embodiments include data structures, processes, routines, or various algorithms implemented as combinations of other programming constructs, such as C, C++, Java ( It can be implemented in a programming or scripting language such as Java), assembler, Python, or the like. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, this embodiment may employ conventional techniques for electronic environment setting, signal processing, and/or data processing. Terms such as “mechanism”, “element”, “means” and “composition” may be used broadly and are not limited to mechanical and physical components. The above terms may include meanings of a series of software routines in connection with a processor or the like.

The foregoing embodiments are just examples, and other embodiments may be implemented within the scope of the following claims.

Claims (17)

A method for providing information related to a cognitive state by an electronic device,
providing a first page including test-related information;
obtaining user voice data for the examination-related information in a state in which the first page is displayed; and
Providing a second page including first evaluation information obtained using a learning model set based on the user voice data;
The first evaluation information is identified based on a content embedding obtained by separating a speaker identification embedding including timbre information from the user voice data;
The speaker identification embedding is obtained using a first encoder based on the user voice data;
The content embedding is obtained using a second encoder based on the user voice data and the speaker identification embedding;
The content embedding includes rhythm information and pitch information,
How to Provide Cognitive State Information.
According to claim 1,
The first page includes a first area including the test-related information and a second area including an icon indicating a state related to voice input.
How to Provide Cognitive State Information.
According to claim 2,
The user voice data is obtained according to an input corresponding to the icon while the first page is displayed.
How to Provide Cognitive State Information.
According to claim 2,
The icon of the second area is displayed as a first image corresponding to the confirmation of the voice input, and displayed as a second image corresponding to the confirmation of the voice input.
How to Provide Cognitive State Information.
According to claim 4,
The icon of the second area is displayed as a third image when the time when the voice input is not confirmed passes the reference time.
How to Provide Cognitive State Information.
According to claim 5,
The first image is a dynamic image that moves in response to the confirmation of the voice input,
The second image and the third image are static images,
How to Provide Cognitive State Information.
According to claim 1,
The inspection-related information includes first inspection information including image information and first text information requesting an explanation of the image information, second text information on a specific subject, and requesting an explanation of the second text information. Including at least one of the second inspection information including the third text information,
How to Provide Cognitive State Information.
According to claim 7,
The user voice data includes at least one of first voice data obtained for the first test information and second voice data obtained for the second test information;
The first voice data and the second voice data are analyzed by different learning models,
How to Provide Cognitive State Information.
According to claim 8,
The upper limit of the length of the first voice data is different from the upper limit of the length of the second voice data,
How to Provide Cognitive State Information.
According to claim 1,
When the obtained user voice data satisfies a set condition, information on completion of recording is displayed;
Displaying information on a re-recording request when the obtained user voice data does not satisfy the set condition;
How to Provide Cognitive State Information.
According to claim 1,
The second page further includes second evaluation information obtained in a different way from the first evaluation information.
How to Provide Cognitive State Information.
According to claim 11,
Each of the first evaluation information and the second evaluation information is provided as a digitized score,
How to Provide Cognitive State Information.
According to claim 1,
The second page further includes recommendation information for guiding at least one of retest, additional test, cognitive training, and hospital transfer when the score obtained by digitizing the first evaluation information is equal to or greater than a reference value.
How to Provide Cognitive State Information.
According to claim 1,
obtaining check list information before inspection;
providing a third page including at least one test check information;
obtaining test voice data corresponding to the test examination information while the third page is displayed; and
Further comprising the step of confirming voice input based on the test voice data,
How to Provide Cognitive State Information.
According to claim 14,
The pre-examination checklist information includes information on at least one of whether the user is awake, the surrounding environment, an examination period, and whether or not a re-examination is performed.
How to Provide Cognitive State Information.
A computer-readable non-transitory storage medium storing a program for executing the method of any one of claims 1 to 15 in a computer.
An electronic device providing information related to a cognitive state,
a memory storing at least one instruction; and
Based on the at least one command, providing a first page including test-related information, obtaining user voice data for the test-related information while the first page is displayed, and based on the user voice data A processor providing a second page including first evaluation information obtained using a set learning model;
The first evaluation information is identified based on a content embedding obtained by separating a speaker identification embedding including timbre information from the user voice data;
The speaker identification embedding is obtained using a first encoder based on the user voice data;
The content embedding is obtained using a second encoder based on the user voice data and the speaker identification embedding;
The content embedding includes rhythm information and pitch information,
electronic device.

Images