WO2026009654A1 - Terminal device and data processing system - Google Patents

Abstract

This terminal device comprises: a camera that captures the surroundings of a wearer; a sensor that detects biological data of the wearer; a microphone; and a control unit that executes control for configuring a storage mode of output data of each of the camera, the sensor, and the microphone collected in a certain period of time to a specific memory according to a value of a specific index calculated on the basis of each set of the output data, and recording each set of output data corresponding to the configured storage mode in the memory as a life log of the wearer.

Description

Terminal equipment and data processing system

The technology disclosed herein relates to a terminal device and a data processing system.

Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Japanese Patent Application Laid-Open No. 2022-180282

However, existing technology leaves room for improvement when it comes to collecting users' biometric data.

Furthermore, conventional technology has the problem that it may not be possible to properly record data depending on the strength of emotions.

A first aspect of the technology disclosed herein includes a terminal device. The terminal device includes a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, a microphone, and a control unit that sets a storage mode in a specific memory for each piece of output data collected over a certain period of time in accordance with a specific index value calculated based on the output data of the camera, the sensor, and the microphone, and executes control to record each piece of output data in accordance with the set storage mode in the memory as a life log of the wearer.

In a second aspect of the technology disclosed herein, the control unit of the first aspect analyzes at least one of the wearer's emotions, the content of the wearer's voice, and the wearer's biometric information based on each of the output data, and calculates the index value based on the analyzed information.

In a third aspect of the technology disclosed herein, the storage mode includes at least some of the compression rate of each piece of output data, deletion of all or part of each piece of output data, storage period for each piece of output data, and storage destination for each piece of output data.

In a fourth aspect of the technology disclosed herein, the control unit of the first aspect associates each piece of output data with the index and records it in the memory.

In a fifth aspect of the technology disclosed herein, the terminal device of the first aspect is a necklace-type terminal worn around the wearer's neck.

A sixth aspect of the technology disclosed herein includes a data processing system including the terminal device of the first aspect and a data processing device. The data processing device includes an input unit that accepts the wearer's speech picked up by the microphone, a processing unit that inputs a prompt including the speech into a data generation model and acquires a response to the speech using the output of the data generation model, and an output unit that plays the acquired response from a speaker of the terminal device.

A data processing system according to a seventh aspect of the technology disclosed herein includes a terminal device including at least a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, and a microphone; and a processing unit that estimates the intensity of the wearer's emotions based on at least a portion of the output data from the camera, the sensor, and the microphone, generates first-format data from each of the output data collected during a period in which a value indicating the estimated intensity of the emotion is less than a predetermined value and stores the data in a specific memory, and generates second-format data with a larger amount of information than the first format from each of the output data collected during a period in which a value indicating the estimated intensity of the emotion is equal to or greater than a predetermined value and stores the data in the specific memory.

In an eighth aspect, in the seventh aspect, the processing unit estimates the wearer's negative emotion as the wearer's emotion.

The ninth aspect is the eighth aspect, wherein the negative emotion is at least one of "anger," "sadness," and "anxiety."

A tenth aspect is the seventh aspect, wherein the processing unit estimates the emotion using a neural network based on at least a portion of each of the output data.

In an eleventh aspect, in the tenth aspect, the plurality of artificial neurons constituting the neural network include emotional artificial neurons, which are artificial neurons in which a current emotion is defined, and the processing unit estimates the intensity of the emotion based on the internal state of the emotional artificial neurons.

A twelfth aspect is the seventh aspect, in which the terminal device is a necklace-type terminal worn around the wearer's neck.

A thirteenth aspect is the seventh aspect, further including an input unit that receives the wearer's speech picked up by the microphone, an acquisition unit that inputs a prompt including the speech into a data generation model and acquires a response to the speech using the output of the data generation model, and an output unit that plays the acquired response from a speaker of the terminal device.

A data processing system according to a fourteenth aspect includes a terminal device including at least a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, and a microphone; and a processing unit that estimates the strength of the wearer's emotions and determines the importance of a situation related to the wearer based on at least a portion of the output data of the camera, the sensor, and the microphone; generates first-format data from each of the output data collected during a period in which the value indicating the estimated intensity of the emotion is less than a first predetermined value or the value indicating the determined importance is less than a second predetermined value, and stores the data in a specific memory; and generates second-format data with a larger amount of information than the first format from each of the output data collected during a period in which the value indicating the estimated intensity of the emotion is equal to or greater than the first predetermined value and the value indicating the determined importance is equal to or greater than the second predetermined value, and stores the data in the specific memory.

A fifteenth aspect is the fourteenth aspect, wherein the processing unit estimates the wearer's negative emotion as the wearer's emotion.

A sixteenth aspect is the fifteenth aspect, wherein the negative emotion is at least one of "anger," "sadness," and "anxiety."

A seventeenth aspect is the fourteenth aspect, wherein the processing unit estimates the emotion using a neural network based on at least a portion of each of the output data.

In an eighteenth aspect, in the tenth aspect, the plurality of artificial neurons constituting the neural network include emotional artificial neurons, which are artificial neurons in which a current emotion is defined, and the processing unit estimates the intensity of the emotion based on the internal state of the emotional artificial neurons.

In a nineteenth aspect, in the fourteenth aspect, the processing unit recognizes the content of speech made by the wearer or the other person when the wearer is conversing with the other person based on at least a portion of the output data, and determines the importance based on the recognized content of speech.

A twentieth aspect is the fourteenth aspect, in which the terminal device is a necklace-type terminal worn around the wearer's neck.

The 21st aspect is the 14th aspect, further including an input unit that receives the wearer's speech picked up by the microphone, an acquisition unit that inputs a prompt including the speech into a data generation model and acquires a response to the speech using the output of the data generation model, and an output unit that plays the acquired response from a speaker of the terminal device.

In a 22nd aspect of the technology disclosed herein, the control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, and the content of the medical professional's remarks collected by the microphone based on each of the output data, and calculates the index value related to the content of the voice related to the wearer's symptoms based on the analyzed information.

A 23rd aspect of the technology disclosed herein is one in which the storage mode includes at least some of the compression rate of each piece of output data, the storage format of the output data, deletion of all or part of each piece of output data, the storage period of each piece of output data, and the storage destination of each piece of output data.

A 24th aspect of the technology disclosed herein includes a terminal device. The terminal device includes a camera that captures images of the wearer and the wearer's surroundings, a sensor that detects biometric data of the wearer, a microphone, and a control unit that sets a storage mode in a specific memory for each piece of output data collected over a certain period of time in accordance with a specific index value calculated based on the output data of the camera, the sensor, and the microphone, and performs control to record each piece of output data in accordance with the set storage mode in the memory as a life log of the wearer.

In a 25th aspect of the technology disclosed herein, the control unit further analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, and the wearer's body movements captured by the camera based on each of the output data, and calculates the index value based on the analyzed information.

A 26th aspect of the technology disclosed herein is one in which the storage mode includes at least some of the compression rate of each piece of output data, the storage format of the output data, deletion of all or part of each piece of output data, the storage period of each piece of output data, and the storage destination of each piece of output data.

A 27th aspect of the technology disclosed herein includes a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, a microphone, and a control unit that sets a storage mode for each of the output data collected over a certain period of time in a specific memory according to a specific index value calculated based on the output data of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer, and the control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, the content of utterances collected by the microphone of a person other than the wearer, and the body movements of the person photographed by the camera based on each of the output data, and calculates the value of the index based on the analyzed information.

In a 28th aspect of the technology disclosed herein, the storage mode includes at least some of the compression rate of each piece of output data, the storage format of the output data, deletion of all or part of each piece of output data, the storage period of each piece of output data, and the storage destination of each piece of output data.

A 29th aspect of the technology disclosed herein includes a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, a microphone, and a control unit that sets a storage mode for each of the output data in a specific memory according to the value of an importance index that is an index representing the degree of importance in the wearer's activities and is calculated based on the output data of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer, wherein the control unit sets the storage mode to store the output data in a large capacity if the calculated value of the importance index is high, sets the storage mode to store the output data in a small capacity if the calculated value of the importance index is low, and sets the storage mode to store the output data in a capacity between a large capacity and a small capacity if it is difficult to calculate the value of the importance index.

In a thirtieth aspect of the technology disclosed herein, the control unit associates each piece of output data with the importance index and records it in the memory.

FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a necklace-type terminal. FIG. 2 is a side view showing the configuration of the necklace-type terminal. FIG. 2 is a top view showing the configuration of the necklace-type terminal. 10 is a schematic diagram showing the functional configuration of a control unit of the necklace-type terminal. 2 shows a schematic functional configuration of a specific processing unit of the data processing device. 10 is a diagram illustrating an example of an operational flow of specific processing by a data processing device. FIG. 10 is a diagram illustrating an example of an index used in the processing of the second embodiment. FIG. 10 is a diagram illustrating an example of an index used in the processing of the second embodiment. FIG. 10 is a diagram illustrating an example of an index used in the processing of the second embodiment. FIG. 10 is a diagram schematically illustrating a block configuration of a data processing device according to a fourth embodiment. FIG. 1 is a diagram illustrating a neural network. FIG. 1 is a diagram showing a schematic diagram of neural network parameters in table form. FIG. 10 is a diagram illustrating an outline of an operation flow when the data processing device is started or reset. FIG. 10 is a diagram for explaining the calculation of the coupling coefficient of an artificial synapse. FIG. 10 is a diagram schematically showing the time evolution of a coupling coefficient when a function h _t ^ij is defined as an increase/decrease parameter of the coupling coefficient. FIG. 10 is a diagram schematically illustrating the time evolution of the coupling coefficient when further simultaneous firing occurs at time _t2 . FIG. 2 is a diagram showing an outline of influence definition information that defines a chemical influence given to a parameter. 1 shows a flowchart for calculating internal state and status. FIG. 10 is a diagram for explaining an example of calculation of the internal state when the artificial neuron does not fire. FIG. 10 is a diagram for explaining an example of calculation of an output when an artificial neuron fires. FIG. 10 is a diagram illustrating the time evolution of a coupling coefficient when a function is defined as an increase/decrease parameter of an artificial neuron. FIG. 10 is a diagram showing an example of rules stored in a switching rule in a table format. FIG. 10 is a diagram showing an example of rules stored in a switching rule in a table format.

Below, an example of an embodiment of a data processing device, data processing method, and program related to the technology disclosed herein will be described with reference to the attached drawings.

First, let me explain the terminology used in the following explanation.

In the following embodiments, the coded processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

In the following embodiments, signed RAM (Random Access Memory) is memory in which information is temporarily stored and is used by the processor as work memory.

In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

In the following embodiments, a communication I/F (Interface) with a symbol is an interface that includes a communication processor, an antenna, etc. The communication I/F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I/F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

In the following embodiments, "A and/or B" is synonymous with "at least one of A and B." In other words, "A and/or B" means that it may be just A, just B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and/or B" also applies when three or more things are expressed connected by "and/or."

[First embodiment]
FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a necklace-type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the necklace-type terminal 14 is an example of a "terminal device" according to the technology of the present disclosure. Note that the terminal device of the present disclosure is not limited to the necklace-type terminal 14. The terminal device of the present disclosure may include a robot, a doll, a stuffed animal, a wearable device (pendant, smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, etc.

The data processing device 12 includes a computer 22, a database 24, and a communication I/F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I/F 26 are also connected to the bus 34. The communication I/F 26 is connected to a network 53. Examples of the network 53 include a WAN (Wide Area Network) and/or a LAN (Local Area Network).

The necklace-type terminal 14 includes a computer 36, a microphone 38, a sensor 39, a speaker 40, a camera 42, and a communication I/F 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 38, speaker 40, and camera 42 are also connected to the bus 52.

The user 20 wearing the necklace-type terminal 14 may be, for example, a patient whose health condition is being diagnosed, or a regular user.

The microphone 38 picks up the voice emitted by the user 20 who is wearing the necklace-type terminal 14, as well as sounds around the user 20. The microphone 38 also receives instructions and the like from the user 20 by receiving the voice emitted by the user 20. The microphone 38 captures the voice emitted by the user 20 and/or people near the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs audio according to instructions from the processor 46. The speaker 40 is, for example, a directional speaker, and outputs audio toward the ears of the user 20.

Sensor 39 is a sensor that detects biometric data of user 20, who is wearing the necklace-type terminal. For example, sensor 39 is a heart rate sensor or a blood oxygen sensor.

Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user 20's surroundings (for example, an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person). Camera 42 may be capable of capturing images of the body of a person near user 20, for example. Furthermore, camera 42 can capture images of user 20's own body, including the user's facial expressions, for example.

Communication I/F 44 is connected to network 53. Communication I/Fs 44 and 26 handle the exchange of various information between processor 46 and processor 28 via network 53.

Figure 2 shows an example of the main functions of the data processing device 12 and necklace-type terminal 14.

As shown in FIG. 2, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

Storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290. Storage 32 also includes a data accumulation unit 54.

In the necklace-type terminal 14, data collection processing is performed by the processor 46. A data collection program 60 is stored in the storage 50. The processor 46 reads the data collection program 60 from the storage 50 and executes the read data collection program 60 on the RAM 48. The data collection processing is realized by the processor 46 operating as the control unit 46A in accordance with the data collection program 60 executed on the RAM 48.

3 and 4, the necklace-type terminal 14 is equipped with multiple microphones 38, multiple sensors 39, multiple speakers 40, and multiple cameras 42. Figures 3 and 4 show an example in which two microphones 38 are positioned so that they are located in front of the user 20 when the user 20 wears the necklace-type terminal 14. Also shown is an example in which two sensors 39 are positioned so that they are located on the right and left sides of the user 20 when the user 20 wears the necklace-type terminal 14. An example is shown in which two speakers 40 are positioned so that they are located on the right and left rear sides of the user 20 when the user 20 wears the necklace-type terminal 14. An example is shown in which two cameras 42 are positioned so that they are located on the right and left front sides of the user 20 when the user 20 wears the necklace-type terminal 14. An example is shown in which two sensors 39 are positioned inside the necklace-type terminal 14 so that they come into contact with the user 20's neck when the user 20 wears the necklace-type terminal 14.

Next, we will explain the processing of the control unit 46A when the necklace-type terminal 14 performs data collection processing to collect data.

In the data collection process of this embodiment, the user's biometric data is collected in real time. Furthermore, not only is this biometric data collected, but all situational data surrounding the user is also collected. This makes it possible to detect early signs of, for example, Alzheimer's disease or dementia. It also makes it possible to monitor the user's health condition (for example, heart disease).

As shown in Figure 5, the control unit 46A includes a data collection unit 100 and a communication unit 102.

The data collection unit 100 collects the outputs of the microphone 38, the sensor 39, and the camera 42.

The communication unit 102 transmits the outputs of the microphone 38, sensor 39, and camera 42 collected by the data collection unit 100 to the data processing device 12.

Next, we will explain the processing of the specific processing unit 290 when the data processing device 12 performs specific processing to obtain a response corresponding to a user utterance.

In the identification process of this embodiment, a response corresponding to the user's utterance picked up by the microphone 38 of the necklace-type terminal 14 is obtained using the data generation model 58.

As shown in FIG. 6, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

The input unit 292 stores the outputs of the microphone 38, sensor 39, and camera 42 received from the necklace-type terminal 14 in the data storage unit 54.

The input unit 292 acquires user utterances received by the necklace-type terminal 14. Specifically, it acquires user utterances picked up by the microphone 38 of the necklace-type terminal 14.

The processing unit 294 performs specific processing using the data generation model 58. Specifically, a prompt including a user utterance is input to the data generation model 58, and a generation result is obtained. At this time, the prompt may further include the outputs of the sensor 39 and the camera 42 collected by the data collection unit 100.

The output unit 296 transmits the results of the specific processing to the necklace-type terminal 14. In the necklace-type terminal 14, the control unit 46A causes the speaker 40 to output the results of the specific processing. In this way, a response corresponding to the user utterance picked up by the microphone 38 is output to the user 20 by the speaker 40. The microphone 38 further acquires the user utterance in response to the results of the specific processing. The control unit 46A transmits audio data indicating the user utterance acquired by the microphone 38 to the data processing device 12. In the data processing device 12, the identification processing unit 290 acquires the user utterance.

Data generation model 58 is what is known as generative AI (artificial intelligence). Examples of data generation model 58 include generative AI such as ChatGPT (Internet search <URL: https://openai.com/blog/chatgpt>) and Gemini (Internet search <URL: https://gemini.google.com/?hl=ja>). Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing an instruction is input to data generation model 58, and inference data such as audio data indicating speech, text data indicating text, and image data indicating an image is also input. Data generation model 58 performs inference on the input inference data in accordance with the instructions indicated by the prompt, and outputs the inference results in the form of data such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and/or summarization.

The outputs of the microphone 38, sensor 39, and camera 42 stored in the data storage unit 54 are used, for example, to diagnose the health condition of the user 20. In this case, the outputs of the microphone 38, sensor 39, and camera 42 stored in the data storage unit 54 may be transmitted to a terminal on the medical institution side. Alternatively, the data processing device 12 may analyze the outputs of the microphone 38, sensor 39, and camera 42 stored in the data storage unit 54 to diagnose the health condition of the user 20.

Next, the operation of the data processing system 10 will be explained.

First, we will explain an example of the data collection process flow.

When the user 20 is wearing the necklace-type terminal 14, the data collection unit 100 sequentially collects the output of each of the microphone 38, sensor 39, and camera 42. The communication unit 102 sequentially transmits the output of each of the microphone 38, sensor 39, and camera 42 collected by the data collection unit 100 to the data processing device 12.

Next, an example of the flow of the identification process will be explained with reference to Figure 7. Here, it is assumed that the input unit 292 of the data processing device 12 sequentially acquires the outputs of the microphone 38, sensor 39, and camera 42 received from the necklace-type terminal 14 and stores them in the data accumulation unit 54.

In step S300, the processing unit 294 determines whether a predetermined trigger condition is met. Specifically, the trigger condition may be that the user's speech picked up by the microphone 38 contains a specific word (e.g., the name of an agent installed in the necklace-type terminal 14) or phrase (e.g., "Hi! XX" (where XX is the agent's name)).

If the trigger conditions are met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger conditions are not met in step S300 (step S300; No), the data processing system 10 ends the specific processing.

In step S301, the processing unit 294 generates a prompt by adding an instruction sentence for obtaining the result of a specific process to the text representing the user's utterance picked up by the microphone 38.

For example, a prompt such as "The user is saying the following: XXX. Please respond as an agent." (XXX is the user utterance) can be generated. Alternatively, the output of each of the sensors 39 and camera 42 can be added to the prompt to generate a prompt such as "This is biometric data representing the user's heart rate and video data representing the user's surroundings. The user is also saying the following: XXX. Please respond as an agent." (XXX is the user utterance).

In step S303, the processing unit 294 inputs the generated prompt into the data generation model 58 and obtains the results of the specific processing based on the output of the data generation model 58.

In step S304, the output unit 296 outputs the results of the identification process to the necklace-type terminal 14, and the identification process ends.

[Second embodiment]

The control unit 46A of the necklace-type terminal 14 may set a storage mode in a specific memory (storage 50, database 24, etc.) for each piece of output data collected over a certain period of time by the collection unit (data collection unit 100) according to a specific index value calculated based on the output data of each of the camera 42, sensor 39, and microphone 38. The control unit 46A may also execute control to record each piece of output data according to the set storage mode in a memory (storage 50, database 24, etc.) as a life log of the wearer.

The control unit 46A of the necklace-type terminal 14 may analyze at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, the content of the voice of a person near the wearer, and the physical movements of a person near the wearer based on each output data, and calculate an index value based on the analyzed information.

The storage mode may include at least some of the compression rate of each of the output data, the storage format (data format, file format) of each of the output data, deletion of all or part of each of the output data, the storage period for each of the output data, and the storage destination for each of the output data. The storage mode may include at least some of the compression rate of each of the output data, deletion of all or part of each of the output data, the storage period for each of the output data, and the storage destination for each of the output data.

The output data may include images, sounds, biometric information, etc. collected by the necklace-type terminal 14. Images may include either still images or moving images. Biometric information may include electrocardiogram data, pulse rate, body temperature, oxygen concentration, etc. The fixed period of time may be interpreted as, for example, 1 second, 1 minute, 10 minutes, 1 hour, 2 hours, etc.

The index may include, for example, the importance index shown in Figure 8A, the excitement index shown in Figure 8B, and the emotion index shown in Figure 8C.

The importance index may be interpreted as an index representing, for example, the degree of importance in the activities of the user 20 who is wearing the necklace-type terminal 14 (the degree of importance of the situation related to the user 20). The importance index may include values such as "1," "2," and "3." The higher the importance index, the higher the importance may be interpreted as being.

For example, if the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that the user 20 in a meeting is participating in an important meeting such as a board meeting, it may set the importance index to "3." If the necklace-type terminal 14 or the data processing device 12 determines that the user 20 is participating in a regular group meeting, it may set the importance index to "2." If the necklace-type terminal 14 or the data processing device 12 determines that the user 20 is having a casual conversation at work (everyday conversation), it may set the importance index to "1."

The excitement index may be interpreted as an index representing the degree of excitement of the user 20 who is wearing the necklace-type terminal 14, for example. The excitement index may include values such as "1," "2," and "3." A higher excitement index may be interpreted as a higher level of excitement.

For example, if the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that the excitement level of a user 20 at a concert venue, etc., is very high, the excitement index may be set to "3." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that the excitement level of a user 20 riding a bus, train, etc. is relatively low, the excitement index may be set to "2." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that the excitement level of a user 20 meditating or walking is very low, the excitement index may be set to "1."

The emotion index may be interpreted as an index representing the degree of emotion (strength of emotion) of the user 20 who is wearing the necklace-type terminal 14, for example. The emotion index may include values such as "1," "2," and "3." A higher emotion index may be interpreted as a higher degree of emotion.

For example, if the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that a user 20 eating their favorite food is in an excited mood, the emotional index may be set to "3." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that a user 20 reading a book is in a state of near-calm emotion, the emotional index may be set to "2." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc., that a user 20 who is not good at exercise is in a state of near-depression while exercising, the emotional index may be set to "1."

These indexes may be associated with data storage modes. Storage modes may include storing the output data of each of the camera 42, sensor 39, and microphone 38 at a high compression rate, storing the output data at a low compression rate, storing the output data without compression in a large-capacity storage format (data format, file format), storing the output data without compression in a small-capacity storage format, or deleting specific data from the output data as unnecessary data. Storage modes may include a storage period (e.g., several months, one year, several years, ten years, etc.) for all or part of the output data of each of the camera 42, sensor 39, and microphone 38. Storage modes may include deleting all or part of the output data of each of the camera 42, sensor 39, and microphone 38 collected over a certain period of time. Storage modes may include storing the output data of each of the camera 42, sensor 39, and microphone 38 collected over a certain period of time in a location other than a specific memory. Specifically, the storage mode may include storing the output data on a specific server instead of the storage 50 of the necklace-type terminal 14. More specifically, the storage mode may include changing the storage destination of the output data to an important cloud (such as a cloud for which a separate contract has been made) that is different from the usual storage destination. The storage modes can be classified into, for example, "3," "2," and "1." Furthermore, for example, storage mode "3" may include a mode in which data is stored at a high compression rate, a mode in which data is stored in a storage format with a large capacity, and a mode in which data is stored for a long period of time. Furthermore, for example, storage mode "1" may include a mode in which data is stored at a lower compression rate than storage mode "3," a mode in which data is stored in a storage format with a smaller capacity than storage mode "3," a mode in which data is stored for a shorter period of time than storage mode "3," and a mode in which data is deleted as unnecessary data. Storage mode "2" may also include a mode in which data is stored at a compression rate lower than storage mode "3" and higher than "1", a mode in which data is stored in a storage format with a capacity smaller than storage mode "3" and larger than "1", and a mode in which the storage period is shorter than storage mode "3" and longer than "1".

More specifically, the saving mode may include changing the save destination of the output data to an important cloud (such as a cloud for which a separate contract has been signed) that is different from the usual save destination.

If the user 20 is participating in an important meeting and it is desirable to store the output data for a long period of time, the necklace-type terminal 14 or the data processing device 12 may set the storage mode "3" to the excitement level index "3" in order to increase the compression rate of each output data. This allows for effective use of memory capacity.

If a user 20 who is not good at exercise is feeling depressed, the necklace-type terminal 14 or data processing device 12 may regard the output data collected at this time as unnecessary data with little usefulness and set the excitement index to "1" and the storage mode to "1". This makes it possible to conserve memory capacity.

The life log may be interpreted as a history of actions taken by the user 20 in their daily life, and may include sounds and images associated with the user 20, specifically sounds collected by the microphone 38 in their daily life and images taken by the camera 42. The life log may record sounds and images associated with the user 20 in association with the date, time, and location at which they were acquired.

Sounds collected by microphone 38 may include the voice of the person with whom user 20 is talking, sounds occurring around user 20 while walking or cycling (voices in a meeting, cars driving by, birds chirping, the sound of a river flowing, trees rustling in the wind), etc.

Camera 42 may, for example, capture scenery within an angle of view that captures what is in front of user 20, or may capture scenery within an angle of view that captures what is not in front of user 20, such as what is to the side, behind, below, or above user 20. Images captured by camera 42 may include the image of someone user 20 is talking to, the scenery around user 20 when taking a walk or cycling, or a pet walking with user 20.

While the necklace-type terminal 14 is worn by the user 20, all or part of the collected sounds and images may be recorded as a life log in the storage 50, database 24, etc. Specifically, when the necklace-type terminal 14 is worn by the user 20, recording of sounds and images may begin, and when the necklace-type terminal 14 is removed from the user 20, recording of sounds and images may end.

The control unit 46A of the necklace-type terminal 14 may associate each piece of output data with an index and record it in memory. For example, output data collected while the user 20 was participating in an important meeting may be associated with an excitement index of "3."

The specific processing unit 290 of the data processing device 12 may include an input unit 292 that receives the wearer's speech picked up by the microphone 38, a processing unit 294 that inputs a prompt including the speech into the data generation model and obtains a response to the speech using the output of the data generation model, and an output unit 296 that plays the obtained response from the speaker of the terminal device.

Specifically, when the processing unit 294 receives, as user data, an utterance from the user 20 relating to the memories or behavior of the user 20, it may execute a process of suggesting to the user 20 information corresponding to the content of the utterance, for example by referring to the database 24 in which the life log is recorded.

(First example of information corresponding to the content of the utterance)
When a user wearing the necklace-type terminal 14 requests a message that will trigger a specific memory as the content of the utterance, the specific processing unit 290 may suggest one or more messages selected based on the life log to the user who requested the message as information corresponding to the content (request) of the utterance.

For example, if a user 20 wearing the necklace-type terminal 14 tries to recall their memory and asks, "What did I say at the meeting I attended on XX date at around XX time?", the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "I think I said something about negotiating with Company A." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

For example, if a user 20 wearing the necklace-type terminal 14 tries to recall their memory and utters, "I wonder who I was talking to at around XX time on XX month," the identification processing unit 290 inputs this message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "It seems that at that time, two friends, probably Mr. B and Mr. C, were talking." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

For example, if a user 20 wearing the necklace-type terminal 14 tries to recall their own feelings and says, "How did I feel when I was talking to Mr. A on a certain date at around XX time?", the identification processing unit 290 inputs this message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "You were laughing a lot at the time, so you seemed to like your friend and be very happy." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

(Second example of information corresponding to the content of the utterance)
When a user 20 wearing the necklace-type terminal 14 tweets a specific matter as the content of the speech, the specific processing unit 290 may suggest to the user 20 who requested the message, as information corresponding to the content of the speech (tweet), the recommended behavior of the user 20 regarding the matter based on the life log.

For example, if a user 20 wearing the necklace-type terminal 14 utters "What should I buy?" while shopping at a particular retail store, the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as "A few months ago, after purchasing product A from this store, you commented that it wasn't very tasty, so this time, how about purchasing product B or product C, which have recently been released?" This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

(Third example of information corresponding to the content of the utterance)
For example, if the user 20 wearing the necklace-type terminal 14 utters, "I wonder what the name of product A I searched for the day before yesterday" while operating a personal computer, the identification processing unit 290 inputs the message as a prompt to the data generation model 58 as an identification process. The data generation model 58 generates a specific output by referencing the life log in the database 24 and analyzing images that were displayed on the screen of the personal computer when the user 20 was operating it in the past. The identification processing unit 290 may generate a message such as "Product A is XXX" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

(Fourth example of information corresponding to the content of the utterance)
For example, if the user 20 wearing the necklace-type terminal 14 utters, while cycling, "There's a place nearby with a spectacular view, but I wonder where it is," the identification processing unit 290 inputs the message as a prompt to the data generation model 58 as an identification process. The data generation model 58 generates a specific output by referencing the life log in the database 24 and analyzing places the user 20 has previously visited and the route to those places. Based on the output obtained by the data generation model 58, the identification processing unit 290 may generate a message such as, "I think Cape XX is 500 meters from here." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

(Fifth example of information corresponding to the content of an utterance)
For example, if the user 20 wearing the necklace-type terminal 14 utters "Tell me his name" when meeting Mr. X from Company A while visiting, the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The data generation model 58 references the life log in the database 24 and generates a specific output based on the history of people the user 20 met while visiting Company A. The identification processing unit 290 may generate a message such as "I think his name is XX" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

According to the present disclosure, by combining the necklace-type terminal 14, the emotion engine (data generation model 58), and an electrocardiogram obtained by a probe (such as a sensor 39), it is possible to grasp the user's 20 state of excitement, trigger video recording or audio capture, and compress and store the life log content based on the level of excitement or the content of an important meeting, for example, to an appropriate size (such as 1/100,000 of the collected data), which can then be individually learned and used to provide useful advice.

According to the present disclosure, it is possible to reduce battery consumption in the necklace-type device 14. It is also possible to prevent the memory capacity of the necklace-type device 14 from being depleted. Furthermore, since most of the data collected in a day may be useless, it is possible to compress and store this data. For example, phrases such as "good morning" and "good night" and views from roads that are regularly used may all be compressed. Furthermore, the human brain also forgets or compresses huge amounts of data from one minute, one hour, one day, one week, one month, one year, ten years ago, etc. According to the present disclosure, it is possible to record a life log by determining a compression coefficient based on such concepts of time and an index of importance.

[Third embodiment]
In the second embodiment described above, an aspect was described in which an emotion index representing the intensity of an emotion of the user 20 is set (estimated) by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc. In the third embodiment, an aspect is described in which the emotion index is subdivided into a plurality of specific emotion indexes corresponding to any of a plurality of different emotions of the user 20, and the intensity of the plurality of emotions of the user 20 (values of the plurality of specific emotion indexes) is estimated by analyzing the output data. The plurality of emotions of the user 20 estimated using the specific emotion index can include, for example, negative emotions of the user 20 (as an example, at least one of "anger,""sadness," and "anxiety").

Furthermore, in the third embodiment, an importance index representing the importance of the user's 20 situation is set by analyzing the output data, and the format of the data to be saved in a specific memory is switched based on the values of the specific emotion index and the importance index.

In other words, as an example of setting a specific emotion index, if a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient and a first event is detected in which the user 20 is talking about a symptom that the user 20 is anxious about, or if a second event is detected in which the doctor is expressing the user 20's feelings of anxiety, the necklace-type terminal 14 or data processing device 12 will estimate that the user 20's "anxiety" has become stronger by setting the value of the specific emotion index corresponding to "anxiety" out of multiple emotions higher than when the first event or the second event is not detected.

Furthermore, as an example, if a third event is detected in which a pet owned by a user 20 wearing the necklace-type terminal 14 has died, the necklace-type terminal 14 or the data processing device 12 will estimate that the user 20's "sadness" has become stronger by increasing the value of the specific emotion index corresponding to "sadness" from among multiple emotions compared to when the third event has not been detected.

As another example of setting a specific emotion index, consider the case where a third event is detected in which a pet belonging to a user 20 wearing the necklace-type terminal 14 has died. In this case, the necklace-type terminal 14 or the data processing device 12 estimates that the user 20's "sadness" has become stronger by setting the value of the specific emotion index corresponding to "sadness" from among multiple emotions higher than when the third event has not been detected.

As an example of setting the importance index, consider a situation in which a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient, and the contents of what the user 20 and the doctor say are recognized, and a fourth event is detected in which the doctor explains his or her symptoms to the user 20, or a fifth event is detected in which the doctor listens to what the user 20 says explaining his or her symptoms. In this case, the necklace-type terminal 14 or the data processing device 12 determines that the importance of the situation related to the user 20 is higher by setting the value of the importance index higher than when the fourth event or the fifth event is not detected.

As another example of setting the importance index, consider a situation in which a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient, and the contents of what the user 20 and the doctor are saying are recognized, and a sixth event in which the user 20 is talking about their own health condition is detected, or a seventh event in which the user 20 is talking about something that makes them anxious is detected. In this case, the necklace-type terminal 14 or the data processing device 12 determines that the situation related to the user 20 is more important by setting the value of the importance index higher than when the sixth or seventh event is not detected. Events for which a high importance index value is set may include "when the user 20 blinks a lot" or "when the user 20 moves their body a lot."

As another example of setting the importance index, consider a situation in which a user 20 wearing a necklace-type terminal 14 is conversing with a doctor as a patient, with the user 20's family and others present, and the contents of what the user 20, the doctor, and the other people are saying are recognized, and an eighth event is detected in which the other person is talking about the patient's (user 20's) health condition, a ninth event is detected in which the other person is expressing the patient's anxiety, or a tenth event is detected in which the other person is expressing their own opinion. In this case, the necklace-type terminal 14 or the data processing device 12 determines that the situation related to the user 20 is more important by setting the importance index value higher than when the eighth, ninth, and tenth events are not detected. Events for which a high importance index value is set may include "when the other person blinks a lot" or "when the other person moves their body a lot."

In the third embodiment, if the values of all of the multiple specific emotion indexes are less than a first predetermined value, or if the value of the importance index is less than a second predetermined value, the necklace-type terminal 14 or data processing device 12 generates first-format data from output data collected during a period in which the value of the specific emotion index is less than the first predetermined value or the value of the importance index is less than the second predetermined value, and stores this in a specific memory. Furthermore, if the value of at least one of the multiple specific emotion indexes is equal to or greater than the first predetermined value and the value of the importance index is equal to or greater than the second predetermined value, the necklace-type terminal 14 or data processing device 12 generates second-format data, which contains more information than the first format, from output data collected during a period in which the value of at least one of the multiple specific emotion indexes is equal to or greater than the first predetermined value and the value of the importance index is equal to or greater than the second predetermined value, and stores this in a specific memory.

In the third embodiment, if the values of all of the multiple specific emotion indices are below a predetermined value, the necklace-type terminal 14 or data processing device 12 generates first-format data from output data collected during a period when the value of the specific emotion index was below the predetermined value and stores it in a specific memory. Furthermore, if the value of at least one of the multiple specific emotion indices is equal to or greater than a predetermined value, the necklace-type terminal 14 or data processing device 12 generates second-format data, which contains more information than the first format, from output data collected during a period when the value of at least one of the multiple specific emotion indices was equal to or greater than the predetermined value and stores it in a specific memory. Note that the first-format data is, for example, highly compressed data, an example of which is text data obtained by performing voice recognition on voice data, or the voice data itself. Furthermore, the second-format data is, for example, low-compression data, an example of which is video data. This makes it possible, for example, to clearly record a life log of times when the user 20 felt strong negative emotions, while generally recording a life log of times when the user did not feel strong negative emotions.

[Fourth embodiment]
The second and third embodiments have described aspects in which an emotion index is set (estimated) by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc. The fourth embodiment describes an aspect in which a neural network including an emotion artificial neuron, which is an artificial neuron that estimates the current emotion of the user 20, is used to estimate the emotion of the user 20 based on the internal state of the emotion artificial neuron.

As shown in FIG. 9, the specific processing unit 290 of the data processing device 12 according to the fourth embodiment includes an initial value setting unit 210, an external input data generation unit 230, a parameter processing unit 240, and a recording control unit 270. The parameter processing unit 240 and the recording control unit 270 are examples of processing units in the present disclosure. The storage 32 of the data processing device 12 also stores definition information 284, parameter initial values 286, the latest parameters 288, a switching rule 291, and recording data 293.

The control unit 46A of the necklace-type terminal 14 transmits output data collected by the camera 42, sensor 39, microphone 38, etc. to the data processing device 12 via the communication I/F 44. In the data processing device 12, the communication I/F 26 outputs the output data received from the necklace-type terminal 14 to the specific processing unit 290.

The initial value setting unit 210 stores the initial values of the parameters that indicate the initial state of the neural network in the parameter initial value 286 in the storage 32. Note that the initial values of the neural network parameters may be predetermined in the data processing device 12, or may be changeable by the user via the network 53.

The external input data generation unit 230 processes at least a portion of the output data received by the communication I/F 26 to generate input information from outside the neural network and outputs it to the parameter processing unit 240. The parameter processing unit 240 performs neural network calculations based on the input information and the current parameters 288 and definition information 284 of the neural network stored in the storage 32.

The artificial neurons in the neural network include multiple artificial neurons that define the user's 20 situation, multiple emotional artificial neurons that define the user's 20 emotions, and multiple endocrine artificial neurons that define the production state of the user's endocrine substances. Note that endocrine substances refer to substances that are secreted in the body and transmit signals, such as neurotransmitters and hormones. Also, endocrine refers to endocrine substances being secreted in the body.

The parameter processing unit 240 calculates parameters representing the internal states of multiple artificial neurons in the neural network based on the input information generated by the external input data generation unit 230. For example, the parameter processing unit 240 updates the current internal state parameters of multiple artificial neurons, etc., for which the user's 20 situation is defined, based on the input information generated by the external input data generation unit 230. The parameter processing unit 240 also calculates the internal state parameters of other artificial neurons in the neural network. This calculates the internal state parameters of an emotional artificial neuron that defines, for example, the emotion "happy." The internal state parameters of this emotional artificial neuron are an example of an index representing the intensity of the emotion "happy." Therefore, the parameter processing unit 240 can estimate the intensity of the user's 20's emotion based on the internal state of the emotional artificial neuron. In this way, the parameter processing unit 240 functions as an emotion estimation unit that estimates the intensity of an emotion using a neural network based on at least a portion of the output data collected by the camera 42, sensor 39, microphone 38, etc.

The neural network parameters calculated by the parameter processing unit 240 are supplied to the recording control unit 270. The recording control unit 270 processes at least a portion of the output data received from the necklace-type terminal 14 to generate record data in the first format or record data in a second format that contains more information than the first format, and records the generated record data in the storage 32 as record data 293. The recording control unit 270 also switches between generating record data in the first format or record data in the second format based on the parameters supplied from the parameter processing unit 240.

For example, if the intensity of the emotion estimated by the parameter processing unit 240 increases while record data in the first format is being generated, the recording control unit 270 switches the format of the generated record data from the first format to the second format, which contains more information. This allows detailed record data from periods when the user 20's emotions are heightened to be preserved as record data 293.

Furthermore, for example, if the intensity of the emotion estimated by the parameter processing unit 240 decreases while recording data in the second format is being generated, the recording control unit 270 switches the format of the generated recording data from the second format to the first format. This makes it possible to compress the volume of recording data during periods when the intensity of the user's 20 emotion is low.

Figure 10 shows a schematic diagram of the neural network 310. The neural network 310 is an exemplary neural network for explaining the operation of the parameter processing unit 240. The neural network 310 includes a plurality of artificial neurons, including artificial neuron 1, artificial neuron 2, artificial neuron 3, artificial neuron 4, artificial neuron 5, artificial neuron 6, artificial neuron 7, artificial neuron 8, artificial neuron 9, artificial neuron a, artificial neuron b, and artificial neuron c. Neural network 310 includes a plurality of artificial synapses, including artificial synapse 311, artificial synapse 312, artificial synapse 313, artificial synapse 314, artificial synapse 315, artificial synapse 316, artificial synapse 317, artificial synapse 318, artificial synapse 319, artificial synapse 320, artificial synapse 321, artificial synapse 322, artificial synapse 323, artificial synapse 324, artificial synapse 325, artificial synapse 326, artificial synapse 327, artificial synapse 328, and artificial synapse 329. Artificial neurons correspond to neurons in a living organism. Artificial synapses correspond to synapses in a living organism.

Artificial synapse 311 connects artificial neuron 4 and artificial neuron 1. Artificial synapse 311 is a unidirectional artificial synapse, as indicated by the arrow on artificial synapse 311. Artificial neuron 4 is an artificial neuron connected to the input of artificial neuron 1. Artificial synapse 312 connects artificial neuron 1 and artificial neuron 2. Artificial synapse 312 is a bidirectional artificial synapse, as indicated by the arrows on both ends of artificial synapse 312. Artificial neuron 1 is an artificial neuron connected to the input of artificial neuron 2. Artificial neuron 2 is an artificial neuron connected to the input of artificial neuron 1.

In this embodiment, an artificial neuron may be represented by N, and an artificial synapse may be represented by S. When identifying each artificial neuron, a superscripted reference symbol may be used as an identification letter. When representing an arbitrary artificial neuron, the letter i or j may be used as an identification letter. For example, ^Ni represents an arbitrary artificial neuron.

An artificial synapse may also be identified by the identification numbers i and j of the two artificial neurons connected to it. For example, ^S41 represents an artificial synapse connecting ^N1 and ^N4 . In general, ^Sij represents an artificial synapse that inputs the output of ^Ni to ^Nj . ^Sji represents an artificial synapse that inputs the output of ^Nj to ^Ni .

10, A to J represent the definition of the state of the user 20. The state of the user 20 includes the emotion of the user 20, the state of production of endocrine substances, the situation of the user 20, etc. As an example, ^N4 , ^N6 , and ^N7 are conceptual artificial neurons in which concepts representing the situation of the user 20 are defined.

^N1 , ^N3 , ^Nb , and ^Nc are emotion artificial neurons to which the emotions of the user 20 are defined. ^N1 is an emotion artificial neuron to which the emotion "happy" is assigned. ^N3 is an emotion artificial neuron to which the emotion "anger" is assigned. ^Nb is an emotion artificial neuron to which the emotion "sadness" is assigned. ^Nc is an emotion artificial neuron to which the emotion "anxiety" is assigned.

^N2 , ^N5 , and ^Na are endocrine artificial neurons to which the endocrine state of the user 20 is defined. ^N5 is an endocrine artificial neuron to which a dopamine generation state is assigned. Dopamine is an example of an endocrine substance involved in the reward system. That is, ^N5 is an example of an endocrine artificial neuron involved in the reward system. N2 is an endocrine artificial neuron to which a serotonin generation state is assigned. Serotonin is an example of an endocrine substance involved in the sleep system. That is, ^N2 is an example of an endocrine artificial neuron involved in the sleep system. ^Na is an endocrine artificial neuron to which a noradrenaline generation state is assigned. Noradrenaline is an example of an endocrine substance involved in the sympathetic nervous system. That is, ^Na is an endocrine artificial neuron involved in the sympathetic nervous system.

The definition information 284 in the storage 32 stores information defining the state of the user 20 as described above for each of the multiple artificial neurons that make up the neural network. Thus, the neural network 310 includes conceptual artificial neurons, emotional artificial neurons, and endocrine artificial neurons. The conceptual artificial neurons, emotional artificial neurons, and endocrine artificial neurons are artificial neurons in which the meanings of concepts, emotions, endocrine functions, etc. are explicitly defined. In contrast, ^N8 and ^N9 are artificial neurons in which the state of the user 20 is not defined. Furthermore, ^N8 and ^N9 are artificial neurons in which the meanings of concepts, emotions, endocrine functions, etc. are not explicitly defined.

The parameters of the neural network 310 include I _t ⁱ , which is an input to each N ⁱ of the neural network, E _t ⁱ , which is an input to N ⁱ from outside the neural network, parameters of N ⁱ , and parameters of S ⁱ .

The parameters of N ⁱ include S _ti ^, which represents the status of N ⁱ ; V ⁱ m _t , which represents the internal state of the artificial neuron represented by N ⁱ ; T ⁱ _t , which represents the firing threshold of N ^{i ;} t _f , which represents the final firing time of N i ; V ⁱ m _tf , which represents the internal state of the artificial neuron N ⁱ at the final firing time; and output increase/decrease parameters a _t ⁱ , b _t ⁱ , and h _t ⁱ . The output increase/decrease parameters are an example of parameters that determine the time evolution of the output when the artificial neuron fires. Note that in this embodiment, the subscript t indicates that the parameter can be updated over time. Furthermore, V ⁱ m _t is information corresponding to the membrane potential of the artificial neuron and is an example of a parameter that represents the internal state or output of the artificial neuron.

The parameters of S ^ij include BS _t ^ij , which represents the coupling coefficient of the artificial synapse of S ^ij , t _cf , which represents the last simultaneous firing time of N ⁱ and N ^j connected to S ^ij , BS ^ij _tcf , which represents the coupling coefficient at the last simultaneous firing time, and a _t ^ij , b _t ^ij , and h _t ^ij , which are coupling coefficient increase/decrease parameters. The coupling coefficient increase/decrease parameters are an example of parameters that determine the time evolution of the coupling coefficient after the last simultaneous firing of two artificial neurons connected by the artificial synapse.

The parameter processing unit 240 updates the above parameters based on inputs from the external input data generation unit 230 and the neural network to determine the activation state of each artificial neuron. The recording control unit 270 determines whether to generate record data in the first format or the second format based on the internal states or activation states of at least some of the artificial neurons in the neural network, determined by the parameter values of at least some of the artificial neurons, and the states defined for at least some of the artificial neurons by the definition information 284. The activation state can be either an activated state or an inactivated state. In this embodiment, activation is sometimes referred to as "firing," and inactivation is sometimes referred to as "non-firing." As described below, the "firing" state is divided into an "upward phase" and a "downward phase" depending on whether the internal state is rising. The "non-firing,""upwardphase," and "downward phase" are represented by status ^{S ti} _.

Fig. 11 shows a schematic table of neural network parameters. Each neuron N has a threshold _Tt and increment/decrement parameters _ht , _at , and _bt as parameters. Each artificial synapse also has a coupling coefficient _BSt and increment/decrement parameters _ht , _at , and _bt as parameters. For each N ⁱ , Fig. 11 shows a line listing the parameters of all artificial neurons directly connected to N ⁱ by artificial synapses, as well as the parameters of the artificial synapses.

12 is a schematic diagram showing the operation flow of the data processing device 12 when the data processing device 12 is started or reset. When the data processing device 12 is started or reset, the parameter processing unit 240 performs initial setting of the neural network parameters. For example, the parameter processing unit 240 obtains initial parameter values from the storage 32 and generates neural network parameter data in a predetermined data structure (S502). The parameter processing unit 240 also sets the values of the neural network parameters at time _t0 . Once the initial setting is complete, a loop for time t is started in S504.

In S510, the parameter processing unit 240 calculates parameters corresponding to changes due to electrical influences of ^the artificial synapses at time step tn ₊₁ . Specifically, BS _tij of any _Sij is calculated.

In S520, the parameter processing unit 240 calculates parameters corresponding to changes due to the chemical influence of the endocrine substance at time step tn _{+1 .} Specifically, it calculates changes in the parameters of N ⁱ and S ^ij affected by the endocrine artificial neuron. More specifically, it calculates increase/decrease parameters and thresholds for the internal state of the artificial neuron N ⁱ affected by the endocrine artificial neuron, and increase/decrease parameters and coupling coefficients for S ^ij affected by the endocrine artificial neuron at time step tn+1.

At S530, the parameter processing unit 240 acquires input from outside the neural network. Specifically, the parameter processing unit 240 acquires the output of the external input data generation unit 230.

In S540, the parameter processing unit 240 calculates the internal state of N ⁱ at time step t _n+1 . Specifically, it calculates V ⁱ m _tn+1 and status S _tt ⁱ . Then, in S550, the value of each parameter at time t _n+1 is stored in the storage 32 as parameters 288. The value of each parameter at time t _n+1 is also output to the recording control unit 270.

In S560, the recording control unit 270 determines whether the parameters of N ⁱ at time step t _n+1 satisfy the conditions for switching the format of the record data to be recorded as the record data 293. If the parameters of N ⁱ at time step t _n+1 satisfy the conditions for switching the format of the record data, the recording control unit 270 switches the format of the record data (S570) and proceeds to S506. On the other hand, if in S560 the parameters of N ⁱ at time step t _n+1 do not satisfy the conditions for switching the format of the record data, the recording control unit 270 proceeds to S506.

In S506, the parameter processing unit 240 determines whether to end the loop. For example, it determines to end the loop when the time represented by the time step reaches a predetermined time, or when output data from the necklace-type terminal 14 has not been received for a predetermined period of time. If the loop is not to be ended, the process returns to S510 and the next time step is calculated. If the loop is to be ended, this flow ends.

13 is a diagram for explaining the outline of the calculation of the coupling coefficients of the artificial synapses, where the constants a ^ij and b ^ij are defined as the initial values of the increase/decrease parameters.

In the time step of time _tn , if both N ⁱ and N ^j at both ends of S ^ij are firing, the parameter processing unit 240 calculates BS _tn+1 ^ij at time tn ₊₁ by BS _tn+1 ^ij = BS _tn ^ij + a _tn ^ij × (t _n+1 - _{t n} ). On the other hand, if neither S ⁱ nor S ^j is firing in the time step of time _tn , the coupling coefficient BS _tn+1 ^ij at time _tn+1 is calculated by BS _tn+1 ^ij = BS _tn ^ij + b _tn ^ij × (t _n+1 - _{t n} ). In addition, if BS _tn+1 ^ij is a negative value, BS _tn+1 ^ij is set to 0. Note that for S ^ij where BS ^ij is a positive value, a _t ^ij is a positive value and b _t ^ij is a negative value. In S ^ij where BS ^ij is a negative value, a _t ^ij is a positive value and b _t ^ij is a negative value.

13, since the artificial neurons at both ends fire simultaneously at time _t0 , BS _t ^ij increases at a _t0 ^ij per unit time. Furthermore, since they do not fire simultaneously at time _t1 , BS _t ^ij decreases at |b _t1 ^ij | per unit time. Furthermore, since they fire simultaneously at time t4, BS _t ^ij increases at a _t4 ^ij per unit time.

14 shows ^a schematic diagram of the time evolution of the coupling coefficient when a function _htij is defined as a parameter for increasing or decreasing ^the coupling coefficient. _htij is defined for ^the time lapse Δt (= t - _tcf ) ≥ 0 from _tcf . _htij is a function of at least Δt and takes a real value.

A function 700 shown in Figure 14 is an example of _htij . The function 700 is a function of ^the coupling coefficient _BStcfij and ^Δt at time _tcf . The function 700 monotonically increases when Δt is smaller than a predetermined value, and monotonically decreases gradually toward 0 when Δt is larger than the predetermined value. The function 700 takes on the ^value _BStcfij at Δt = 0.

14 defines a function 700 as a parameter for increasing or decreasing the coupling coefficient, and shows the coupling coefficient when N ⁱ and N ^j at both ends fire simultaneously at time t _0. The parameter processing unit 240 calculates BS _t ^ij at each time from time t ₁ to time t ₆ based on the function 700 and Δt. N ⁱ and N ^j do not fire simultaneously within the time range from time t ₁ to time t _6. Therefore, for example, the coupling coefficient decreases monotonically after time t ₂ .

FIG. 15 shows a schematic diagram of the time evolution of the coupling coefficient when N ⁱ and N ^j fire simultaneously again at time t _2. The coupling coefficient is calculated from time t ₀ to time t ₂ in the same manner as in FIG. 14. When N _i and N _j fire simultaneously again at time t ₂ , the parameter processing unit 240 calculates the coupling coefficient at each time from time t ₃ to t ₆ according to h _t ^ij (t - t ₂ , BS _t2 ^ij ). In this way, the coupling coefficient increases each time simultaneous firing occurs. This has the effect of strengthening the artificial synaptic connection, as in Hebb's law in living organisms. On the other hand, as shown in FIGS. 13 and 14, a longer period of time without simultaneous firing has the effect of weakening the artificial synaptic connection.

Figure 16 shows an outline of the effect definition information that defines the chemical effect on a parameter. This effect definition information is used to calculate the change in the parameter in S520 of Figure 12. The definition information includes conditions related to the internal state of the endocrine artificial neuron, information specifying the influencing artificial neuron or artificial synapse, and an equation that defines the effect content.

16 , endocrine artificial neuron ^N2 is an endocrine artificial neuron to which the endocrine substance of drowsiness is assigned. The definition information for endocrine artificial neuron ^N2 includes the condition "Vm _tn ² > T _tn ² ,""emotional artificial neurons N ¹ and N ³ " as artificial neurons affected by endocrine artificial neuron N ² , and the formula "T _tn+1 ⁱ = T _tn ⁱ × 1.1" defining the content of the influence. As a result, when Vm _tn ² exceeds T _tn ² , the parameter processing unit 240 increases the thresholds of emotional artificial neurons N ¹ and N ³ at time t _n+1 by 10%. This makes it possible to make the emotional artificial neurons less likely to fire when drowsiness occurs, for example.

Furthermore, endocrine artificial neuron ^N5 is an endocrine artificial neuron to which dopamine is assigned. The first definition information for endocrine artificial neuron ^N5 defines the condition "Vm _tn ⁵ > T _tn ⁵ and Vm _tn ⁴ > T _tn ⁴ ," the artificial synapses "S ⁴⁹ and S ⁹⁵ " affected by endocrine artificial neuron ^N5 , and the formula "a _tn+1 ^ij = a _tn ^ij × 1.1" defining the content of the influence. As a result, the parameter processing unit 240 increases the increase/decrease parameters of artificial synapses S ⁴⁹ and S ⁹⁵ at time t _n+1 by 10% when Vm _tn ⁵ exceeds T _tn ⁵ and Vm _tn ⁴ exceeds _T ^{tn 4} .

The second definition information for the endocrine artificial neuron ^N5 defines the condition "Vm _tn ⁵ > T _tn ⁵ ," the artificial neuron "N ¹ " affected by the endocrine artificial neuron N ⁵ , and the formula "T _tn+1 ⁱ = T _tn ⁱ × 1.1" defining the content of the influence. As a result, when Vm _tn ⁵ exceeds T _tn ⁵ , the parameter processing unit 240 reduces the increase/decrease parameter of the artificial neuron N ¹ at time t _n+1 by 10%. This makes it easier for the emotion of happiness to be ignited when the endocrine artificial neuron N ⁵ of the reward system is fired.

Note that the influence definition information is not limited to the example in Figure 16. For example, a condition may be defined that the internal state of the artificial neuron is below a threshold. Conditions regarding the status of the artificial neuron, such as an ascending phase, a descending phase, or non-firing, may also be defined. The influence range can be defined by directly specifying an artificial neuron or artificial synapse, or by defining "all artificial synapses connected to a specific artificial neuron." Regarding the influence formula, if the target is an artificial neuron, in addition to multiplying the threshold by a constant, it may also be defined to add a constant to the threshold or to multiply the increase/decrease parameter of the internal state by a constant. If the target is an artificial synapse, in addition to multiplying the increase/decrease parameter by a constant, it may also be defined to multiply the coupling coefficient by a constant.

The influence definition information is stored in the definition information 284 of the storage 32. In this way, the storage 32 stores influence definition information that defines the influence that at least one of the internal state and firing state of the endocrine artificial neuron has on the parameters of at least one of the other artificial neurons and artificial synapses that are not directly connected to the endocrine artificial neuron via an artificial synapse. The parameter processing unit 240 then updates the parameters of at least one of the other artificial neurons and artificial synapses that are not directly connected to the endocrine artificial neuron via an artificial synapse based on at least one of the internal state and firing state of the endocrine artificial neuron and the influence definition information. Furthermore, the parameters of the other artificial neurons that are influenced by at least one of the internal state and firing state of the endocrine artificial neuron can include at least one of the parameters that determine the threshold, firing state, and time evolution of the output at the time of firing of the other artificial neuron. Furthermore, the parameters of the artificial synapse affected by at least one of the internal state and firing state of the endocrine artificial neuron can include the coupling coefficient of the artificial synapse and at least one parameter that determines the time evolution of the coupling coefficient after the two artificial neurons connected by the artificial synapse last fired simultaneously. Furthermore, the influence definition information includes information that determines the influence that the firing state of the endocrine artificial neuron associated with the reward system has on the threshold of the emotional artificial neuron, and the parameter processing unit 240 updates the threshold of the emotional artificial neuron in accordance with the influence definition information when the endocrine artificial neuron fires.

Fig. 17 shows a flowchart for calculating V _tn+1 ⁱ and S _tn+1 ⁱ . The process of this flowchart can be applied to part of the process in S540 in Fig. 12. In S1100, parameter processing unit 240 determines whether S _tn ⁱ indicates non-firing.

If S _tn ⁱ indicates non-firing, the parameter processing unit 240 calculates the input I _tn+1 ⁱ to N ⁱ (S1110). Specifically, if an input from outside the neural network is not connected to N ⁱ , it is calculated by I _tn+1 ⁱ =Σ _j BS _tn+1 ^ji ×Vm _tn ^j ×f(S _tn ^j ). If an input from outside the neural network is connected to N ⁱ , it is calculated by I _tn+1 ⁱ =Σ _j BS _tn+1 ^ji ×Vm _tn ^j ×f(S _tn ^j ) + E _tn+1 ⁱ , where E _tn ⁱ is the input from outside the neural network at time t _n .

Furthermore, f(S) returns 0 if S is a value that indicates no firing, and returns 1 if S is a value that indicates an ascending or descending phase. This model corresponds to a model in which a synapse transmits an action potential only when the neuron fires. Note that f(S) = 1 may also be returned. This corresponds to a model in which a membrane potential is transmitted regardless of the neuron's firing state.

In S1112, the parameter processing unit 240 determines whether I _tn+1 ⁱ exceeds T _tn+1 ⁱ . If _{I tn+1} ⁱ exceeds T _tn+1 ⁱ , the parameter processing unit 240 calculates Vm _tn+1 ⁱ based on the increase/decrease parameters, and sets S tn+ ₁ ⁱ to a value indicating an increasing phase or a decreasing phase according to Vm _tn+1 ⁱ (S1114), and ends this flow.

In S1100, if S _tn ⁱ is in the rising phase or falling phase, the parameter processing unit 240 calculates Vm _tn+1 ⁱ (S1120). Then, if Vm _t ⁱ reaches Vmin by t _n+1 , the parameter processing unit 240 sets S _tn+1 ⁱ to a non-firing value, and if Vm _t ⁱ has not reached Vmin by t _n+1 , the parameter processing unit 240 sets S _tn+1 ⁱ to a rising phase or falling phase value, and then ends this flow. Note that if Vm _t ⁱ reaches Vmax by t _n+1 , the parameter processing unit 240 sets S _tn+1 ⁱ to a falling phase value, and if Vm _t ⁱ has not reached Vmax by t _{n+1, the} parameter processing unit 240 sets S _tn+1 ⁱ to a rising phase value.

In this way, when N ⁱ is firing, the output of N ⁱ is independent of the input, even if the output is below the threshold. This period corresponds to the absolute refractory period in biological neurons.

FIG. 18 is a diagram for explaining a schematic example of calculation of V _t ⁱ when N ⁱ does not fire.

^Ni is unfired at the time step of time _t0 . If I _t1 ⁱ at time _t1 is equal to or less than T _t1 ⁱ , the parameter processing unit 240 calculates V _t1 ⁱ at time _t1 using V _t1 ⁱ = I _t1 ⁱ , and calculates V _t ⁱ for the period from time _t0 to _t1 using V _t ⁱ = I _t0 ⁱ . Similarly, the parameter processing unit 240 maintains the value of V _tn calculated at time step _tn until the next time step, and changes it to I _{tn +1 at V tn} +1.

19 is a diagram for explaining an outline of an example of calculation of V _t ⁱ when N ⁱ ignites, in which constants a ⁱ and b ⁱ are defined.

In the time step of time _t0 , N ⁱ is unfired. If I _t1 ⁱ at time _t1 exceeds T _t1 ⁱ , the parameter processing unit 240 calculates V _t1 ⁱ at time _t1 using V _t1 ⁱ = I _t1 ⁱ , and calculates V _t ⁱ for the period from time _t0 to _t1 using V _t ⁱ = I _t0 ⁱ . Note that here, it is assumed that I _t1 ⁱ at time _t1 is equal to or less than Vmax. If I _t1 ⁱ at time _t1 exceeds Vmax, I _t1 ⁱ = Vmax.

19, the parameter processing unit 240 increases _Vt ⁱ at a _ti ^j per unit time from time _t1 until _Vt ⁱ reaches Vmax. The parameter processing unit 240 also determines the status _St ⁱ of N ⁱ during this period to be in the rising phase.

Furthermore, when _Vt ⁱ reaches Vmax, _Vt ⁱ is decreased by | _bt ⁱ | per unit time until _Vt ⁱ reaches Vmin. The parameter processing unit 240 determines the status of N ⁱ during this period to be the falling phase. When _Vt ⁱ reaches Vmin, _Vt6 ⁱ at the next time is calculated by _Vt6 ⁱ = I _t6 ⁱ . Furthermore, the status after _Vt ⁱ reaches Vmin is determined to be non-firing.

When the status of ^Ni is in a descending phase, _Vmti does not depend ^on _Iti even if _the calculated _Vmti falls below _Tti . Even if ^Vmti falls below ^Tti , the parameter processing unit ₂₄₀ ^{calculates Vmti} _according to ^the increase ^/ decrease parameters until ^Vmti reaches _Vmin ^.

20 shows a schematic diagram of the time evolution of the coupling coefficient when the function ht _i is defined as ^an increase/decrease parameter for N ⁱ . Generally, _ht ⁱ is defined for the elapsed time Δt (= t - t _f ) ≧ 0 from the firing time t _f . _ht ⁱ is a function of at least Δt. _ht ⁱ takes a real number value, and the value range of _ht ⁱ is equal to or greater than Vmin and equal to or less than Vmax.

A function 1300 shown in Figure 20 is an example of _hti . The function 1300 is a function of _Vmtfi and Δt at time _tf . The function 1300 monotonically increases when ^Δt is smaller than ^a predetermined value, and monotonically decreases when Δt is larger than the predetermined value. The function 1300 takes on the ^value _Vmtfi at Δt = 0.

20 shows the output when function 1300 is defined as an increase/decrease parameter for the internal state and N ⁱ fires at time t _1. The parameter processing unit 240 calculates Vm _t ⁱ at each of times t ₁ to t ₅ based on function 1300, Δt, and Vm _f ⁱ . Because _{Vm t} ⁱ reaches Vmin at time t ₅ , Vm _t ⁱ =I _t6 ⁱ at time t ₆ .

21 shows an example of a rule 1400 stored in the switching rule 291 in table format. The rule 1400 defines an operation to "switch the recording data format to the low-compression second format" when at least a first condition is satisfied, that is, Vm _t ⁱ of any of N ¹ , N ³ , N ^b , and N ^c exceeds a threshold. As a result, when recording data is recorded in the high-compression first format, the recording control unit 270 determines to switch the recording data format to the low-compression second format when the first condition changes from not being satisfied to being satisfied. Note that the threshold value shown is, for example, a value obtained by multiplying Vmax of each N ^j by a constant 0.9. The threshold value may be higher than T ⁱ _t .

Rule 1400 also prescribes an operation to "switch the data recording format to the low-compression second format" when at least the second condition, that is, the sum of Vm _t ⁱ of ^N5 and _Na , exceeds a threshold, is satisfied. As a result, when data is recorded in the high-compression first format, the recording control unit 270 determines to switch the recording data format to the low-compression second format when the second condition changes from not being satisfied to being satisfied. Note that the threshold value is exemplified by a value obtained by multiplying the sum of Vmax of each N ^j by a constant of 0.9. The threshold value may be higher than the sum of T ⁱ _t of each N ^j .

^N1 , ^N3 , ^Nb , and ^Nc are emotion artificial neurons in which the emotions of "happiness,""anger,""sadness," and "anxiety" are defined, respectively. Therefore, the parameter processing unit 240 estimates the strength of each of the emotions of "happiness,""anger,""sadness," and "anxiety" of the user 20 based on the internal state of the emotion artificial neuron, and can switch the format of the recorded data to a low-compression second format when the strength of at least one of the estimated emotions of "happiness,""anger,""sadness," and "anxiety" exceeds a predetermined threshold.

^N5 and ^Na are endocrine artificial neurons for which the endocrine substances "dopamine" and "noradrenaline" are defined, respectively. The sum of the parameters of the internal states of these endocrine artificial neurons is an example of an index representing the strength of the emotion of "excitement." Therefore, the parameter processing unit 240 estimates the strength of the emotion of "excitement" of the user 20 based on the internal states of the endocrine artificial neurons, and can switch the format of the recorded data to the low-compression second format when the estimated strength of the emotion of "excitement" exceeds a predetermined threshold.

Rule 1400 also prescribes an operation of ^" switching the format of the recorded _data ^to the high- ^compression first format" when a third condition is met, in which Vm _t ⁱ of all of N ¹ , N ³ , N ^b , and N ^c is equal to or less than the first threshold, and the sum of Vm t i of N 5 and Na is equal to or less than the second threshold. Therefore, when the recorded data is recorded in the low-compression second format, the recording control unit 270 determines to switch the format of the recorded data to the high-compression first format when the third condition changes from not being satisfied to being satisfied. In this way, the format of the recorded data can be switched to the high-compression first format when the estimated intensity of the emotion of user 20 becomes equal to or less than a predetermined threshold.

The first threshold for the third condition is a value obtained by multiplying Vmax of each ^Nj by a constant 0.8. The second threshold for the third condition is a value obtained by multiplying the sum of Vmax of each ^Nj by a constant 0.8. In this manner, a case is illustrated in which the first threshold for the third condition is smaller than the threshold for the first condition, and the second threshold for the third condition is smaller than the threshold for the second condition. However, the first threshold may be the same as the threshold for the first condition, and the second threshold may be the same as the threshold for the second condition. The first threshold for the third condition may be higher than T ⁱ _t of each ^Nj . The second threshold for the third condition may be higher than the sum of T ⁱ _t of each ^Nj . The thresholds for each condition are not limited to these examples, and various values can be applied.

According to the data processing system 10, the data processing device 12 generates recording data in a highly compressed format, such as text data or audio data, and continuously records it as recording data 293, while the user's 20 emotions are not high. Furthermore, when the user's 20 emotions become high, the data processing device 12 generates recording data in a low compressed format, such as video data, and continuously records it as recording data 293, while the emotion remains strong above a certain level.

In this way, the data processing system 10 can store and record video data of scenes in which the user 20 felt strong emotions as recording data 293. On the other hand, when the user 20 did not feel strong emotions, it can store and record generalized information such as text data or audio data as recording data 293. Therefore, the data processing system 10 can clearly preserve memories (records) of when the user 20 felt strong emotions, while generalizing memories (records) of when the user 20 did not feel strong emotions.

In this embodiment, "happiness," "anger," "sadness," and "anxiety" have been described as emotions, but the emotions handled by the data processing system 10 are not limited to these. In addition, in this embodiment, "dopamine," "serotonin," and "noradrenaline" have been described as endocrine substances, but the endocrine substances handled by the data processing system 10 are not limited to these.

Furthermore, the functions of the data processing device 12 may be implemented by one or more computers. At least some of the functions of the data processing device 12 may be implemented by a virtual machine. Furthermore, at least some of the functions of the data processing device 12 may be implemented in the cloud. Furthermore, the functions of the components of the data processing device 12, excluding the storage 32, can be realized by the CPU operating based on a program. For example, at least some of the processing described as the operation of the data processing device 12 can be realized by the processor controlling each piece of hardware (e.g., a hard disk, memory, etc.) that the computer has in accordance with the program. In this way, at least some of the processing of the data processing device 12 can be realized by the processor operating in accordance with the program to control each piece of hardware, and the hardware including the processor, hard disk, memory, etc. operating in cooperation with the program. In other words, the program can cause the computer to function as each component of the data processing device 12. Similarly, the functions of the control unit 46A, one of the components of the necklace-type terminal 14, can be realized by the CPU operating based on a program. In other words, the program can cause the computer to function as the control unit 46A of the necklace-type terminal 14. The computer may load a program that controls the execution of the above-described processes and operate in accordance with the loaded program to execute the processes. The computer may load the program from a computer-readable recording medium that stores the program. The program may also be supplied to the computer via a communication line, and the computer may load the program supplied via the communication line.

In the embodiment described above, a data processing device 12 separate from the necklace-type terminal 14 is responsible for neural network processing. Furthermore, a data processing device 12 separate from the necklace-type terminal 14 stores information such as video data. However, the necklace-type terminal 14 may also perform the functions of the data processing device 12, such as neural network processing. Furthermore, the necklace-type terminal 14 may also store recorded data, etc.

Fifth Embodiment
In the second embodiment described above, an aspect was described in which an emotion index representing the intensity of the emotion of the user 20 is set (estimated) by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, and the like. In the fifth embodiment, the emotion index is subdivided into multiple specific emotion indexes corresponding to multiple different emotions of the user 20, and the intensity of the multiple emotions of the user 20 (values of the multiple specific emotion indexes) is estimated by analyzing the output data. The multiple emotions of the user 20 estimated using the specific emotion index may include, for example, negative emotions of the user 20 (for example, at least one of "anger,""sadness," and "anxiety"). Furthermore, in the fifth embodiment, an importance index representing the importance of the situation of the user 20 is set by analyzing the output data, and the format of data to be saved in a specific memory is switched based on the values of the specific emotion index and the importance index.

In other words, as an example of setting a specific emotion index, consider a situation in which a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient, and a first event is detected in which the user 20 talks about a symptom that the user 20 is anxious about, or a second event is detected in which the doctor speaks on behalf of the user 20's feelings of anxiety. In this case, the necklace-type terminal 14 or the data processing device 12 estimates that the user 20's "anxiety" has become stronger by setting the value of the specific emotion index corresponding to "anxiety" out of multiple emotions higher than when the first event or the second event is not detected.

As another example of setting a specific emotion index, consider the case where a third event is detected in which a pet belonging to a user 20 wearing the necklace-type terminal 14 has died. In this case, the necklace-type terminal 14 or the data processing device 12 estimates that the user 20's "sadness" has become stronger by setting the value of the specific emotion index corresponding to "sadness" from among multiple emotions higher than when the third event has not been detected.

As an example of setting the importance index, consider a situation in which a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient, and the contents of what the user 20 and the doctor say are recognized, and a fourth event is detected in which the doctor explains his or her symptoms to the user 20, or a fifth event is detected in which the doctor listens to what the user 20 says explaining his or her symptoms. In this case, the necklace-type terminal 14 or the data processing device 12 determines that the importance of the situation related to the user 20 is higher by setting the value of the importance index higher than when the fourth event or the fifth event is not detected.

As another example of setting the importance index, consider a situation in which a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient, and the contents of what the user 20 and the doctor are saying are recognized, and a sixth event in which the user 20 is talking about their own health condition is detected, or a seventh event in which the user 20 is talking about something that makes them anxious is detected. In this case, the necklace-type terminal 14 or the data processing device 12 determines that the situation related to the user 20 is more important by setting the value of the importance index higher than when the sixth or seventh event is not detected. Events for which a high importance index value is set may include "when the user 20 blinks a lot" or "when the user 20 moves their body a lot."

As another example of setting the importance index, consider a situation in which a user 20 wearing a necklace-type terminal 14 is conversing with a doctor as a patient, with the user 20's family and others present, and the contents of what the user 20, the doctor, and the other people are saying are recognized, and an eighth event is detected in which the other person is talking about the patient's (user 20's) health condition, a ninth event is detected in which the other person is expressing the patient's anxiety, or a tenth event is detected in which the other person is expressing their own opinion. In this case, the necklace-type terminal 14 or the data processing device 12 determines that the situation related to the user 20 is more important by setting the importance index value higher than when the eighth, ninth, and tenth events are not detected. Events for which a high importance index value is set may include "when the other person blinks a lot" or "when the other person moves their body a lot."

In the fifth embodiment, if the values of all of the multiple specific emotion indexes are less than a first predetermined value, or if the value of the importance index is less than a second predetermined value, the necklace-type terminal 14 or data processing device 12 generates first-format data from output data collected during a period in which the value of the specific emotion index is less than the first predetermined value or the value of the importance index is less than the second predetermined value, and stores this in a specific memory. Furthermore, if the value of at least one of the multiple specific emotion indexes is equal to or greater than the first predetermined value and the value of the importance index is equal to or greater than a second predetermined value, the necklace-type terminal 14 or data processing device 12 generates second-format data, which contains more information than the first format, from output data collected during a period in which the value of at least one of the multiple specific emotion indexes is equal to or greater than the first predetermined value and the value of the importance index is equal to or greater than a second predetermined value, and stores this in a specific memory.

Note that the first format data is, for example, highly compressed data, an example of which is text data obtained by performing voice recognition on voice data, or the voice data itself. The second format data is, for example, low-compression data, an example of which is video data. This makes it possible, for example, to clearly record a life log of when user 20 felt strong negative emotions, while generally recording life logs of when they did not feel strong negative emotions.

Sixth Embodiment
The second and third embodiments have described aspects in which an emotion index is set (estimated) by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc. The sixth embodiment describes an aspect in which a neural network including an emotion artificial neuron, which is an artificial neuron that estimates the current emotion of the user 20, is used to estimate the emotion of the user 20 based on the internal state of the emotion artificial neuron.

As shown in FIG. 9, the specific processing unit 290 of the data processing device 12 according to the sixth embodiment includes an initial value setting unit 210, an external input data generation unit 230, a parameter processing unit 240, an importance determination unit 272, and a recording control unit 270. The parameter processing unit 240, the importance determination unit 272, and the recording control unit 270 are examples of processing units in the present disclosure. The storage 32 of the data processing device 12 also stores definition information 284, parameter initial values 286, the latest parameters 288, a switching rule 291, and recording data 293.

The control unit 46A of the necklace-type terminal 14 transmits output data collected by the camera 42, sensor 39, microphone 38, etc. to the data processing device 12 via the communication I/F 44. In the data processing device 12, the communication I/F 26 outputs the output data received from the necklace-type terminal 14 to the specific processing unit 290.

The initial value setting unit 210 stores the initial values of the parameters that indicate the initial state of the neural network in the parameter initial value 286 in the storage 32. Note that the initial values of the neural network parameters may be predetermined in the data processing device 12, or may be changeable by the user via the network 53.

The external input data generation unit 230 processes at least a portion of the output data received by the communication I/F 26 to generate input information from outside the neural network and outputs it to the parameter processing unit 240. The parameter processing unit 240 performs neural network calculations based on the input information and the current parameters 288 and definition information 284 of the neural network stored in the storage 32.

The artificial neurons in the neural network include multiple artificial neurons that define the user's 20 situation, multiple emotional artificial neurons that define the user's 20 emotions, and multiple endocrine artificial neurons that define the production state of the user's endocrine substances. Note that endocrine substances refer to substances that are secreted in the body and transmit signals, such as neurotransmitters and hormones. Also, endocrine refers to endocrine substances being secreted in the body.

The parameter processing unit 240 calculates parameters representing the internal states of multiple artificial neurons in the neural network based on the input information generated by the external input data generation unit 230. For example, the parameter processing unit 240 updates the current internal state parameters of multiple artificial neurons, etc., for which the user's 20 situation is defined, based on the input information generated by the external input data generation unit 230. The parameter processing unit 240 also calculates the internal state parameters of other artificial neurons in the neural network. This calculates the internal state parameters of an emotional artificial neuron that defines, for example, the emotion "happy." The internal state parameters of this emotional artificial neuron are an example of an index representing the intensity of the emotion "happy." Therefore, the parameter processing unit 240 can estimate the intensity of the user's 20 emotion based on the internal state of the emotional artificial neuron. In this way, the parameter processing unit 240 functions as an emotion estimation unit that estimates the intensity of an emotion using a neural network based on at least a portion of the output data collected by the camera 42, sensor 39, microphone 38, etc. The neural network parameters calculated by the parameter processing unit 240 are supplied to the recording control unit 270.

The importance determination unit 272 determines the importance of the situation related to the user 20, as exemplified in the fifth embodiment, based on at least a portion of the output data received by the communication I/F 26. The importance of the situation related to the user 20 determined by the importance determination unit 272 (importance index) is supplied to the recording control unit 270.

The recording control unit 270 processes at least a portion of the output data received from the necklace-type terminal 14 to generate recording data in the first format or recording data in a second format that contains more information than the first format, and records the generated recording data in the storage 32 as recording data 293. The recording control unit 270 also switches between generating recording data in the first format or recording data in the second format based on the parameters supplied from the parameter processing unit 240 and the importance (importance index) of the situation related to the user 20 supplied from the importance determination unit 272.

For example, when the value indicating the intensity of emotion estimated by the parameter processing unit 240 becomes equal to or greater than a first predetermined value and the value indicating the importance determined by the importance determination unit 272 becomes equal to or greater than a second predetermined value while record data in the first format is being generated, the recording control unit 270 switches the format of the record data to be generated from the first format to the second format, which contains more information. This makes it possible to keep detailed record data of a period when the user 20's emotions are heightened and the situation related to the user 20 is of high importance as record data 293.

Furthermore, for example, when the value indicating the intensity of emotion estimated by the parameter processing unit 240 becomes less than a first predetermined value, or the value indicating the importance determined by the importance determination unit 272 becomes less than a second predetermined value, while record data in the second format is being generated, the recording control unit 270 switches the format of the record data to be generated from the second format to the first format. This makes it possible to compress the volume of record data during periods when the intensity of the user's 20's emotion is low or the importance of the situation related to the user 20 is low.

The parameter processing unit 240 updates the above-mentioned parameters based on input from the external input data generation unit 230 and the neural network shown in FIG. 10, and determines the activation state of each artificial neuron. The importance determination unit 272 determines the importance of the situation related to the user 20, as exemplified in the fifth embodiment, based on at least a portion of the output data received by the communication I/F 26.

The recording control unit 270 determines whether to generate record data in the first format or the second format based on the internal state or activation state of at least some of the artificial neurons in the neural network, which is determined by the parameter values of at least some of the artificial neurons, the state defined for at least some of the artificial neurons by the definition information 284, and the importance of the situation related to the user 20 determined by the importance determination unit 272. The activation state can be either an activated state or an inactivated state. In this embodiment, activation may be referred to as "firing," and inactivation may be referred to as "unfiring." As will be described later, the "firing" state is divided into an "upward phase" and a "downward phase" depending on whether the internal state is rising. The "unfiring,""upwardphase," and "downward phase" are represented by the status ^{S ti} _.

Fig. 11 shows a schematic table of neural network parameters. Each neuron N has a threshold _Tt and increment/decrement parameters _ht , _at , and _bt as parameters. Each artificial synapse also has a coupling coefficient _BSt and increment/decrement parameters _ht , _at , and _bt as parameters. For each N ⁱ , Fig. 11 shows a line listing the parameters of all artificial neurons directly connected to N ⁱ by artificial synapses, as well as the parameters of the artificial synapses.

12 is a schematic diagram showing the operation flow of the data processing device 12 when the data processing device 12 is started or reset. When the data processing device 12 is started or reset, the parameter processing unit 240 performs initial setting of the neural network parameters. For example, the parameter processing unit 240 obtains initial parameter values from the storage 32 and generates neural network parameter data in a predetermined data structure (S502). The parameter processing unit 240 also sets the values of the neural network parameters at time _t0 . Once the initial setting is complete, a loop for time t is started in S504.

In S510, the parameter processing unit 240 calculates parameters corresponding to changes due to electrical influences of ^the artificial synapses at time step tn ₊₁ . Specifically, BS _tij of any _Sij is calculated.

In S520, the parameter processing unit 240 calculates parameters corresponding to changes due to the chemical influence of the endocrine substance at time step tn _{+1 .} Specifically, it calculates changes in the parameters of N ⁱ and S ^ij affected by the endocrine artificial neuron. More specifically, it calculates increase/decrease parameters and thresholds for the internal state of the artificial neuron N ⁱ affected by the endocrine artificial neuron, and increase/decrease parameters and coupling coefficients for S ^ij affected by the endocrine artificial neuron at time step tn+1.

At S530, the parameter processing unit 240 acquires input from outside the neural network. Specifically, the parameter processing unit 240 acquires the output of the external input data generation unit 230.

In S540, the parameter processing unit 240 calculates the internal state of ^Ni at time step tn ₊₁ . Specifically, it calculates ^Vimtn ₊₁ and status _Stt ⁱ . Then, in S550, the value of each parameter at time tn ₊₁ is stored in the storage 32 as parameters 288. The value of each parameter at time tn ₊₁ is also output to the recording control unit 270. In addition, in S555, the importance determination unit 272 determines the importance of the situation related to the user 20 based on at least a portion of the output data received by the communication I/F 26, and outputs an importance index value indicating the determined importance to the recording control unit 270.

In S560, the recording control unit 270 determines whether the parameters of N ⁱ at time step t _n+1 and the value of the importance index indicating the importance of the situation for the user 20 satisfy the conditions for switching the format of the record data to be recorded as the record data 293. If the parameters of N ⁱ at time step t _n+1 and the value of the importance index satisfy the conditions for switching the format of the record data, the recording control unit 270 switches the format of the record data (S570) and proceeds to S506. On the other hand, if in S560 the parameters of N ⁱ at time step t _n+1 and the value of the importance index do not satisfy the conditions for switching the format of the record data, the recording control unit 270 proceeds to S506.

In S506, the parameter processing unit 240 determines whether to end the loop. For example, it determines to end the loop when the time represented by the time step reaches a predetermined time, or when output data from the necklace-type terminal 14 has not been received for a predetermined period of time. If the loop is not to be ended, the process returns to S510 and the next time step is calculated. If the loop is to be ended, this flow ends.

FIG. 22 shows an example of a rule 1400 stored in the switching rule 291 in table format. The rule 1400 defines an operation to "switch the recording data format to the low-compression second format" when at least condition A, which indicates that Vm _t ⁱ of any of N ¹ , N ³ , N ^b , and N ^c exceeds a threshold, is satisfied as a first condition, and condition B, which indicates that the importance index value is equal to or greater than a predetermined value, is satisfied. As a result, when recording data is recorded in the high-compression first format, the recording control unit 270 determines to switch the recording data format to the low-compression second format when the first condition changes from not being satisfied to being satisfied. Note that the threshold value for condition A in the first condition is exemplified by a value obtained by multiplying Vmax of each N ^j by a constant 0.9. The threshold value may be higher than T ⁱ _t .

Furthermore, rule 1400 defines, as a second condition, an operation of "switching the data recording format to the low-compression second format" when at least condition A, that is, the sum of Vm _t ⁱ of ^N5 and _Na exceeds a threshold, is satisfied, and condition B, that the value of the importance index is equal to or greater than a predetermined value, is satisfied. As a result, when data is recorded in the high-compression first format, the recording control unit 270 determines to switch the recording data format to the low-compression second format when the second condition changes from not being satisfied to being satisfied. Note that, as an example of the threshold value for condition A in the second condition, a value obtained by multiplying the sum of Vmax of each N ^j by a constant 0.9 is given. The threshold value may be higher than the sum of T ⁱ _t of each N ^j .

^N1 , ^N3 , ^Nb , and ^Nc are emotion artificial neurons in which the emotions of "happiness,""anger,""sadness," and "anxiety" are defined, respectively. Therefore, the parameter processing unit 240 estimates the strength of each of the emotions of "happiness,""anger,""sadness," and "anxiety" of the user 20 based on the internal state of the emotion artificial neuron, and can switch the format of the recorded data to the low-compression second format when the strength of at least one of the estimated emotions of "happiness,""anger,""sadness," and "anxiety" exceeds a predetermined threshold and the importance index indicating the importance of the situation for the user 20 reaches a predetermined value or greater.

^N5 and ^Na are endocrine artificial neurons for which the endocrine substances "dopamine" and "noradrenaline" are defined, respectively. The sum of the parameters of the internal states of these endocrine artificial neurons is an example of an index representing the strength of the emotion of "excitement." Therefore, the parameter processing unit 240 estimates the strength of the emotion of "excitement" of the user 20 based on the internal states of the endocrine artificial neurons, and can switch the format of the recorded data to the low-compression second format when the estimated strength of the emotion of "excitement" exceeds a predetermined threshold and the importance index, which indicates the importance of the situation for the user 20, reaches or exceeds a predetermined value.

Furthermore, rule 1400 prescribes, as a third condition, an operation of ^" switching the ^format of the _recorded data to the high-compression first format ^" when at least one of condition ^A , that is, Vm _t ⁱ of ^all of N 1 , N ³ , N ^b , and N c is equal to or less than the first threshold and the sum of Vm t i of N 5 and Na is equal to or less than the second threshold, and condition B, that is, the value of the importance index is less than a predetermined value, is satisfied. Therefore, when the recorded data is recorded in the low-compression second format, the recording control unit 270 determines to switch the format of the recorded data to the high-compression first format when the third condition changes from not being satisfied to being satisfied. In this way, the format of the recorded data can be switched to the high-compression first format when the intensity of the estimated emotion of user 20 falls below a predetermined threshold or when the value of the importance index, which indicates the importance of a situation related to user 20, falls below a predetermined value.

The first threshold of condition A in the third condition is a value obtained by multiplying Vmax of each ^Nj by a constant 0.8. The second threshold of condition A in the third condition is a value obtained by multiplying the sum of Vmax of each ^Nj by a constant 0.8. In this way, a case is illustrated in which the first threshold of condition A in the third condition is smaller than the threshold of condition A in the first condition, and the second threshold of condition A in the third condition is smaller than the threshold of condition A in the second condition. However, the first threshold may be the same as the threshold of condition A in the first condition, and the second threshold may be the same as the threshold of condition A in the second condition. The first threshold of condition A in the third condition may be higher than T ⁱ _t of each N ^j . The second threshold of condition A in the third condition may be higher than the sum of T ⁱ _t of each N ^j . The thresholds of each condition are not limited to these examples, and various values can be applied.

According to the data processing system 10, the data processing device 12 generates highly compressed recording data such as text data and audio data and continuously records it as recording data 293 during a period when the user 20's emotions are not high or the importance of the situation related to the user 20 is not high. Furthermore, when the user 20's emotions are high and the importance of the situation related to the user 20 is high, the data processing device 12 generates low compressed recording data such as video data and continuously records it as recording data 293 during a period when the emotion is strong enough to exceed a certain value and the importance index is above a predetermined value.

In this way, the data processing system 10 can accumulate and record video data of scenes in which the user 20 feels strong emotions and in which the situation relating to the user 20 is of high importance as recording data 293. On the other hand, when the user 20 does not feel strong emotions or the situation relating to the user 20 is not of high importance, it can accumulate and record generalized information such as text data or audio data as recording data 293. Therefore, the data processing system 10 can clearly preserve memories (records) of when the user 20 feels strong emotions and the situation relating to the user 20 is of high importance, while generalizing memories (records) of when the user 20 does not feel strong emotions or the situation relating to the user 20 is of low importance.

As described above, the data processing system 10 according to the embodiment includes a necklace-type terminal 14 and a processing unit. The necklace-type terminal 14 includes at least a camera 42 that captures images of the wearer (user 20) and the surroundings, a sensor 39 that detects the wearer's biometric data, and a microphone 38. The processing unit estimates the strength of the wearer's emotions and determines the importance of the situation related to the wearer based on at least a portion of the output data from the camera 42, sensor 39, and microphone 38. The processing unit then generates first-format data from each piece of output data collected during a period in which the value indicating the estimated emotion strength is less than a first predetermined value or the value indicating the determined importance is less than a second predetermined value, and stores the data in a specific memory. The processing unit also generates second-format data, which contains more information than the first format, from each piece of output data collected during a period in which the value indicating the estimated emotion strength is equal to or greater than the first predetermined value and the value indicating the determined importance is equal to or greater than a second predetermined value, and stores the data in a specific memory. This allows data to be appropriately recorded according to the strength of the user's 20 emotions and the importance of the situation related to the user 20.

[Seventh embodiment]

In the seventh embodiment, the necklace-type terminal 14 or data processing device 12 sets an importance index for the user 20 by analyzing output data of a person other than the user 20 collected by the camera 42, sensor 39, microphone 38, etc. Specifically, the person other than the patient user 20 may be a medical professional such as a doctor who is involved with the patient. Below, a doctor is used as an example of a medical professional. If the necklace-type terminal 14 or data processing device 12 determines that the patient user 20 is being examined by a doctor, the necklace-type terminal 14 or data processing device 12 may set the importance index to "3" if the doctor's comments, the physical movements of the doctor and patient, and the patient's biometric information meet predetermined conditions. That is, for example, if the doctor makes a statement such as "explaining the patient's symptoms" or "listening to the patient's symptoms," the necklace-type terminal 14 or data processing device 12 may set the importance index to "3." For example, if the patient blinks a lot or moves their body a lot, the necklace-type terminal 14 or data processing device 12 may set the importance index to "3."

"Explaining about the patient's symptoms" includes, for example, explaining information about the patient's test results, information about the causes, symptoms, and prognosis of the patient's illness, future treatment plans, information about medications that have been or will be prescribed to the patient, and points to keep in mind in the patient's daily life.

"Asking about the patient's symptoms" includes, for example, asking about symptoms present at the time of the examination, information about the patient's daily physical and mental condition, information about the patient's medical history, information about medications the patient takes on a daily basis, information about the patient's appetite, information about the patient's drinking or smoking habits, information about the type and frequency of the patient's usual exercise, and information about the patient's sleep.

"When the patient blinks a lot" refers to, for example, when the number of blinks within a specified period of time is equal to or exceeds a specified threshold number. When there is a lot of blinking, it is considered that there is a high possibility that the patient is nervous. In other words, when there is a lot of blinking, there is a high possibility that the patient is making an important statement.

"When the patient is moving a lot" includes, for example, when the patient is gesturing or using hand movements while speaking. In such cases, it is highly likely that the patient is desperately trying to communicate their thoughts to the doctor.

When biological information meets specified conditions, this refers to a state in which, for example, electrocardiogram data, pulse rate, body temperature, oxygen concentration, etc., are outside the range of specified thresholds.

On the other hand, if the doctor determines that he is exchanging seasonal greetings or engaging in small talk that is unrelated to the state of "explaining the patient's symptoms" or "listening to the patient's symptoms," the necklace-type terminal 14 or data processing device 12 may set the importance index to "1."

"Seasonal greetings and small talk" includes, for example, talking about today's or recent weather, talking about social topics that are being covered prominently on television, talking about the doctor's or patient's hobbies, and talking about the transportation the patient used to get to the hospital.

As described above, the importance index for the states in which the doctor is "explaining the patient's symptoms" and "listening to the patient's symptoms" is set to "3." In these cases, the save state is set to "3" for importance index "3," and the output data for importance index "3" is saved in a format with a large capacity. For example, the image data and audio data included in the output data in these cases is saved as video data.

On the other hand, when the doctor is offering seasonal greetings or making small talk, the importance index is set to "1." In this case, the save state is set to "1" for importance index "1," and the output data for importance index "1" is saved in a format with a small capacity. For example, the image data included in the output data in these cases may be saved as photograph (still image) data, or only the audio data may be saved without saving the image data, or it may be saved as text data.

In this way, by setting importance indexes and changing the storage format of output data based on the doctor's comments and body movements, memory can be used efficiently.

In the seventh embodiment, when the processing unit 294 receives, as user data, an utterance from the user 20 relating to the memories or behavior of the user 20, it may perform a process of suggesting to the user 20 information corresponding to the content of the utterance, for example, by referring to the database 24 in which the life log is recorded.

(Examples of information corresponding to the content of the utterance)
For example, if a user 20 wearing the necklace-type terminal 14 tries to recall his or her memory and utters, "I wonder what kind of conversation I had with Dr. X, who examined me at XX Clinic on XX date at around XX time," the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "At that time, it seems that you and Dr. X discussed your symptoms, physical and mental condition, medical history, appetite, drinking, smoking, sleep, test results, causes of illness, symptoms, prognosis, future treatment plans, information about medications, points to note in daily life, etc., in the following manner." The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

According to the present disclosure, by combining the necklace-type terminal 14, the emotion engine (data generation model 58), and an electrocardiogram obtained by a probe (such as a sensor 39), it is possible to grasp the user's 20 state of excitement, trigger video recording or audio capture, and compress and store the life log content to an appropriate size (such as 1/100,000 of the collected data) based on the level of excitement or the content of an important meeting, for example, or record it in an appropriate storage format, and then individually learn from this information and use it to provide useful advice.

According to the present disclosure, it is possible to reduce battery consumption in the necklace-type device 14. It is also possible to prevent the memory capacity of the necklace-type device 14 from being depleted. Furthermore, since most of the data collected in a day may be useless, it is possible to compress and store this data. For example, phrases such as "good morning" and "good night" and views from roads that are regularly used may all be compressed. Furthermore, the human brain also forgets or compresses huge amounts of data from one minute, one hour, one day, one week, one month, one year, ten years ago, etc. According to the present disclosure, it is possible to record a life log by determining a compression coefficient based on such concepts of time and an index of importance.

[Eighth embodiment]

In the eighth embodiment, the necklace-type terminal 14 or data processing device 12 sets an importance index for the user 20 by analyzing output data collected by the camera 42, sensor 39, microphone 38, etc. If the user 20 of the necklace-type terminal is a patient, the objects from which output data is collected may include output data from medical professionals such as doctors who are involved with the patient, and from attendants who are present at the doctor's examination, etc. Specifically, for example, if it is determined that the patient user 20 is being examined by a doctor, the importance index may be set to "3" when the patient's comments, body movements, and biometric information satisfy specified conditions. In other words, for example, the necklace-type terminal 14 or data processing device 12 may set the importance index to "3" when the patient "talks about their health condition," "talks about their concerns," "blinks frequently," or "moves frequently." Furthermore, for example, if the doctor says something like "I am explaining the patient's symptoms" or "I am listening to the patient's symptoms," the necklace-type terminal 14 or data processing device 12 may set the importance index to "3." For example, if the patient blinks a lot or moves their body a lot, the necklace-type terminal 14 or data processing device 12 may set the importance index to "3."

When a patient "talks about their health condition" includes, for example, when they talk about information regarding their physical and mental condition, their medical history, medications they take on a daily basis, their appetite, their drinking or smoking habits, the type and frequency of their usual exercise, and their sleep habits.

When a patient "talks about their own feelings of anxiety" includes, for example, when the patient talks about their feelings of anxiety regarding their current or future health condition, their feelings of anxiety regarding their future lifestyle, and their feelings of anxiety regarding their job.

"When the patient blinks a lot" refers to, for example, when the number of blinks within a specified period of time is equal to or exceeds a specified threshold number. When there is a lot of blinking, it is considered that there is a high possibility that the patient is nervous. In other words, when there is a lot of blinking, there is a high possibility that the patient is making an important statement.

"When the patient is moving a lot" includes, for example, when the patient is gesturing or using hand movements while speaking. In such cases, it is highly likely that the patient is desperately trying to communicate their thoughts to the doctor.

When biological information meets specified conditions, this refers to a state in which, for example, electrocardiogram data, pulse rate, body temperature, oxygen concentration, etc., are outside the range of specified thresholds.

When a doctor "explains about a patient's symptoms," this includes, for example, explaining information about the patient's test results, information about the causes, symptoms, and prognosis of the patient's illness, future treatment plans, information about medications that have been or will be prescribed to the patient, and points that the patient should be aware of in their daily lives.

When a doctor "asks about a patient's symptoms," this includes, for example, asking about symptoms present at the time of the examination, information about the patient's daily physical and mental condition, information about the patient's medical history, information about medications the patient takes on a daily basis, information about the patient's appetite, information about the patient's drinking or smoking habits, information about the type and frequency of the patient's usual exercise, and information about the patient's sleep habits.

On the other hand, if it is determined that the patient or doctor is engaging in seasonal greetings or small talk unrelated to their own health condition, psychological state, or condition, the necklace-type terminal 14 or data processing device 12 may set the importance index to "1."

"Seasonal greetings and small talk" includes, for example, talking about today's or recent weather, talking about social topics that are being covered prominently on television, talking about the doctor's or patient's hobbies, and talking about the transportation the patient used to get to the hospital.

As described above, the importance index is set to "3" when the patient is talking about "their health condition" or "their feelings of anxiety." The importance index is also set to "3" when the patient blinks frequently and when the patient's body movements are frequent. It is possible that other attendants are making important comments about the patient in the time periods before and after the time periods when the number of blinks is greater than the threshold number. Therefore, with regard to blinking, the importance index may be set to "3" for a first predetermined period before and after this time period, in addition to the time period when the number of blinks is equal to or greater than the threshold number. This first predetermined period is, for example, five seconds. It is also possible that the patient is desperately trying to communicate his or her thoughts to the doctor in the time periods before and after the time periods when the patient's body movements are frequent. Therefore, with regard to the patient's body movements, the importance index may be set to "3" for a second predetermined period before and after this time period, in addition to the time period when the patient's body movements are frequent. This second predetermined period is, for example, five seconds. In these cases, the importance index "3" is set to the storage state "3," and the output data for the importance index "3" is saved in a storage format with a large capacity. For example, the image data and audio data included in the output data in these cases are saved as video data.

On the other hand, when the patient is exchanging seasonal greetings or making small talk, the importance index is set to "1." In this case, the save state is set to "1" for importance index "1," and the output data for importance index "1" is saved in a format with a small capacity. For example, the image data included in the output data in these cases may be saved as photograph (still image) data, or only the audio data may be saved without saving the image data, or it may be saved as text data.

In this way, by setting an importance index and changing the storage format of output data based on the patient's speech and physical movements, memory can be used efficiently.

In the eighth embodiment, when the processing unit 294 receives, as user data, an utterance from the user 20 relating to the memories or behavior of the user 20, it may perform a process of suggesting to the user 20 information corresponding to the content of the utterance, for example, by referring to the database 24 in which the life log is recorded.

(Examples of information corresponding to the content of the utterance)
For example, if the user 20 wearing the necklace-type terminal 14 tries to recall his or her memory and utters, "I wonder what I said when I was examined at XX Clinic on XX date at around XX time," the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "At that time, it seems that the doctor told you something like XXX about your health condition, your concerns, etc." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

According to the present disclosure, by combining the necklace-type terminal 14, the emotion engine (data generation model 58), and an electrocardiogram obtained by a probe (such as a sensor 39), it is possible to grasp the user's 20 state of excitement, trigger video recording or audio capture, and compress and store the life log content to an appropriate size (such as 1/100,000 of the collected data) based on the level of excitement or the content of an important meeting, for example, or record it in an appropriate storage format, and then individually learn from this information and use it to provide useful advice.

[Ninth embodiment]

In the ninth embodiment, the necklace-type terminal 14 or data processing device 12 sets an importance index for the user 20 by analyzing output data of a person other than the user 20 collected by the camera 42, sensor 39, microphone 38, etc. Specifically, if the necklace-type terminal 14 or data processing device 12 determines that the patient user 20 is being examined by a doctor at a hospital with another person (hereinafter, a companion), the necklace-type terminal 14 or data processing device 12 may set the importance index to "3" if the companion's comments or movements satisfy predetermined conditions. That is, for example, the necklace-type terminal 14 or data processing device 12 may set the importance index to "3" when the companion "talks about the patient's health condition," "speaks on behalf of the patient's anxieties," "talks about their own feelings," or when the companion blinks frequently or moves their body a lot.

"Patient's health condition" includes, for example, information regarding the patient's physical and mental state, information regarding the patient's medical history, information regarding medications the patient takes on a daily basis, information regarding the patient's appetite, information regarding the patient's drinking or smoking habits, information regarding the type and frequency of the patient's usual exercise, and information regarding the patient's sleep.

"Patient's feelings of anxiety" include, for example, patient's feelings of anxiety about their current or future health condition, patient's feelings of anxiety about their future lifestyle, and patient's feelings of anxiety about their job.

"What the attendees themselves think" includes, for example, their thoughts about the patient's health condition, their thoughts about what kind of treatments and tests the patient should receive in the future, their thoughts about what kind of lifestyle the patient should lead in the future, and their thoughts about what aspects of the patient's life they should reflect on.

"When a person in attendance blinks frequently" refers to, for example, when the number of blinks within a specified period of time exceeds a specified threshold. When there is a lot of blinking, it is highly likely that the person in attendance is nervous. In other words, when there is a lot of blinking, there is a high possibility that the person in attendance is making important comments about the patient.

"When the attendee is making a lot of physical movements" includes, for example, when the attendee is making gestures and hand movements while speaking. In such cases, it is highly likely that the attendee is desperately trying to convey their thoughts to the doctor.

On the other hand, if it is determined that the attendee is engaging in seasonal greetings or small talk that is unrelated to the patient's health condition, psychological state, or the attendee's thoughts about the patient's condition, the necklace-type terminal 14 or data processing device 12 may set the importance index to "1."

"Seasonal greetings and small talk" includes, for example, talking about today's or recent weather, talking about social topics that are being covered prominently on television, talking about the hobbies of the patient or other attendees, and talking about the transportation used to arrive at the hospital.

As described above, the importance index is set to "3" when a patient's companion "talks about the patient's health condition," "speaks on behalf of the patient's anxieties," or "talks about their own thoughts." Furthermore, the importance index is set to "3" when the companion blinks frequently and when the companion's body movements are frequent. It is possible that the companion is making important comments about the patient in the time periods before and after the time periods when the number of blinks is greater than the threshold number. Therefore, with regard to blinking, the importance index may be set to "3" not only for the time periods when the number of blinks is equal to or greater than the threshold number, but also for a first predetermined period before and after this time period. This first predetermined period is, for example, five seconds. Furthermore, it is possible that the companion is desperately trying to convey their thoughts to the doctor in the time periods before and after the time periods when the companion's body movements are frequent. Therefore, with regard to the companion's body movements, the importance index may be set to "3" not only for the time periods when the companion's body movements are frequent, but also for a second predetermined period before and after this time period. This second predetermined period is, for example, five seconds. In these cases, the save state "3" is set for importance index "3," and the output data for importance index "3" is saved in a storage format with a large capacity. For example, the image data and audio data included in the output data in these cases is saved as video data.

On the other hand, when attendees are exchanging seasonal greetings or making small talk, the importance index is set to "1." In this case, the save state is set to "1" for importance index "1," and the output data for importance index "1" is saved in a format with a small capacity. For example, the image data included in the output data in these cases can be saved as photo (still image) data, or only the audio data can be saved without saving the image data, or it can be saved as text data.

In this way, memory can be used efficiently by changing the storage format of output data based on what other people say and their body movements.

In the ninth embodiment, when the processing unit 294 receives, as user data, an utterance from the user 20 relating to the memories or behavior of the user 20, it may perform a process of suggesting information corresponding to the content of the utterance to the user 20, for example, by referring to the database 24 in which the life log is recorded.

(Examples of information corresponding to the content of the utterance)
For example, if the user 20 wearing the necklace-type terminal 14 tries to recall his/her memory and utters, "I wonder what Mr./Ms. X, who accompanied me to my medical examination at XX Clinic at around XX o'clock, said," the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and generate a message such as, "At that time, Mr./Ms. X seemed to have told the doctor about your health condition, what he/she thinks you are worried about, and what he/she thinks about it." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

According to the present disclosure, by combining the necklace-type terminal 14, the emotion engine (data generation model 58), and an electrocardiogram obtained by a probe (such as a sensor 39), it is possible to grasp the user's 20 state of excitement, trigger video recording or audio capture, and compress and store the life log content to an appropriate size (such as 1/100,000 of the collected data) based on the level of excitement or the content of an important meeting, for example, or record it in an appropriate storage format, and then individually learn from this information and use it to provide useful advice.

[Tenth embodiment]

The control unit 46A of the necklace-type terminal 14 may set the storage mode to save a large amount of output data when the importance index value calculated based on the output data of each of the camera 42, sensor 39, and microphone 38 is high. The control unit 46A may also set the storage mode to save a small amount of output data when the importance index value calculated based on the output data of each of the camera 42, sensor 39, and microphone 38 is low.

For example, in FIG. 8A, if the importance index is set to "3," storage mode "3" is set. This storage mode "3" may be a storage mode for saving a large amount of output data including video data captured over a certain period of time by camera 42 and audio data picked up by microphone 38.

On the other hand, if the importance index is set to "1", storage mode "1" is set. This storage mode "1" may be a storage mode that does not include video data captured by camera 42, but stores relatively small amounts of output data such as image data of still images. It may also be a storage mode that does not include audio data picked up by microphone 38, but stores text data converted from audio data.

Furthermore, if the importance index is set to "2", storage mode "2" is set. This storage mode "2" may be a storage mode that stores output data with a larger capacity than storage mode "1" but a smaller capacity than storage mode "3".

Here, if the control unit 46A of the necklace-type terminal 14 finds it difficult to calculate the importance index value based on information obtained by analyzing at least one of the wearer's emotions, the content of the wearer's voice, and the wearer's biometric information based on each output data, the storage mode may be set to "2," which is between "1" and "3."

For example, even if user 20 is participating in a conference, if it is not possible to determine whether the conference is important from the output data collected by camera 42, sensor 39, microphone 38, etc., it will be difficult to calculate the value of the importance index. Furthermore, even if it is determined that user 20 is participating in a regular group meeting, if the content being discussed in the meeting is important, it will be difficult to calculate the value of the importance index.

Furthermore, if the importance cannot be determined because there is insufficient output data from any of the camera 42, sensor 39, and microphone 38, i.e., if it is difficult to calculate the value of the importance index, the output medium with insufficient data may be stored in a large capacity, and the output medium with sufficient data may be stored in a small capacity. For example, if there is sufficient information from the camera 42 and sensor 39 but insufficient information from the microphone 38, i.e., if there is insufficient audio information, the output from the microphone 38 may be stored in a large capacity, and the output from the camera 42 and sensor 39 in a small or medium capacity.

Furthermore, even if it is determined that user 20 is having an everyday conversation at work, if it is determined that one of the people participating in the conversation is meeting for the first time, it will be difficult to calculate the importance index value. In these cases, it may be determined that it is difficult to calculate the importance index value, and the storage mode may be set to "2."

Incidentally, even when the same video data is saved in storage mode "2" and storage mode "3," the control unit 46A may set the storage mode so that higher quality video data is saved when storage mode "3" is set, and compressed video data is saved when storage mode "2" is set.

Furthermore, in the case of the excitement index, as with the importance index, if it is difficult to calculate the value, the storage mode may be set to "2" and the data may be saved in a medium capacity; and in the case of the emotion index, if it is difficult to calculate the value, the storage mode may be set to "2" and the data may be saved in a medium capacity.

The system according to the present disclosure has been described above primarily in terms of the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may also be implemented as a general information processing system. The present disclosure may also be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone, etc. The method according to the present disclosure may also be provided to users in a SaaS (Software as a Service) format.

In the above embodiment, an example was given in which a specific process is performed by a single computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may also be performed by multiple computers, including computer 22.

In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 53, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

It is not necessary to store the entire specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 53, or to store the entire specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

The hardware resource for executing a specific process can be any of the following types of processors. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), and ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration specifically designed to execute a specific process. Each of these processors has built-in or connected memory, and each processor executes a specific process by using the memory.

The hardware resource that executes the specific processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

As an example of a configuration using a single processor, first, there is a configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes specific processing. Second, there is a configuration in which a processor is used to realize the functions of an entire system, including multiple hardware resources that execute specific processing, on a single IC chip, as typified by SoC (System-on-a-chip). In this way, specific processing is realized using one or more of the various processors listed above as hardware resources.

Furthermore, the hardware structure of these various processors can be, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices. Furthermore, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, as long as it does not deviate from the spirit of the invention.

The above-described written content and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above explanation of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or substitutions may be made to the above-described written content and illustrations, as long as they do not deviate from the spirit of the technology of the present disclosure. Furthermore, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above-described written content and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

All publications, patent applications, and technical standards mentioned in this specification are incorporated by reference herein to the same extent as if each individual publication, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

The disclosures of Japanese Patent Application No. 2024-107796 filed on July 3, 2024, Japanese Patent Application No. 2024-135454 filed on August 14, 2024, Japanese Patent Application No. 2024-135455 filed on August 14, 2024, Japanese Patent Application No. 2024-137784 filed on August 19, 2024, Japanese Patent Application No. 2024-137785 filed on August 19, 2024, Japanese Patent Application No. 2024-135717 filed on August 15, 2024, and Japanese Patent Application No. 2024-130197 filed on August 6, 2024 are incorporated herein by reference in their entireties.

10 Data processing system 12 Data processing device 14 Necklace type terminal 38 Microphone 39 Sensor 40 Speaker 42 Camera 46A Control unit 100 Data collection unit 102 Communication unit 240 Parameter processing unit 270 Recording control unit 290 Specific processing unit 292 Input unit 294 Processing unit 296 Output unit

Claims (23)

A camera that captures the wearer's surroundings;
a sensor for detecting biological data of the wearer;
A microphone and
a control unit that sets a storage mode in a specific memory for each of the output data collected over a certain period of time according to a value of a specific index calculated based on the output data of each of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer;
A terminal device including: The terminal device according to claim 1, wherein the control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, and the wearer's biometric information based on each of the output data, and calculates the index value based on the analyzed information. The terminal device of claim 1, wherein the storage mode includes at least some of the compression rate of each of the output data, deletion of all or part of each of the output data, storage period of each of the output data, and storage destination of each of the output data. The terminal device according to claim 1, wherein the control unit links each piece of output data to the index and records it in the memory. The terminal device described in claim 1 is a necklace-type terminal worn around the wearer's neck. A data processing system including the terminal device according to claim 1 and a data processing device,
The data processing device includes:
an input unit that receives the wearer's speech picked up by the microphone;
a processing unit that inputs a prompt including the utterance into a data generation model and obtains a response to the utterance using an output of the data generation model;
an output unit that reproduces the acquired response from a speaker of the terminal device;
2. The data processing system of claim 1, comprising: a terminal device including at least a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, and a microphone;
a processing unit that estimates the intensity of emotion of the wearer based on at least a portion of the output data of the camera, the sensor, and the microphone, generates data in a first format from each of the output data collected during a period in which a value indicating the estimated intensity of emotion is less than a predetermined value, and stores the data in a specific memory, and generates data in a second format having a larger amount of information than the first format from each of the output data collected during a period in which a value indicating the estimated intensity of emotion is equal to or greater than a predetermined value, and stores the data in the specific memory;
A data processing system comprising: The data processing system of claim 7, wherein the processing unit estimates the wearer's negative emotions as the wearer's emotions. The data processing system of claim 8, wherein the negative emotion is at least one of "anger," "sadness," and "anxiety." The data processing system of claim 7, wherein the processing unit estimates the emotion using a neural network based on at least a portion of each of the output data. The plurality of artificial neurons constituting the neural network include emotion artificial neurons that are artificial neurons in which a current emotion is defined,
The data processing system of claim 10 , wherein the processing unit estimates the intensity of the emotion based on an internal state of the emotion artificial neuron. The data processing system of claim 7, wherein the terminal device is a necklace-type terminal worn around the wearer's neck. an input unit that receives the wearer's speech picked up by the microphone;
an acquisition unit that inputs a prompt including the utterance into a data generation model and acquires a response to the utterance using an output of the data generation model;
an output unit that reproduces the acquired response from a speaker of the terminal device;
8. The data processing system of claim 7, further comprising: a terminal device including at least a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, and a microphone;
a processing unit that estimates the strength of the wearer's emotion and determines the importance of a situation related to the wearer based on at least a portion of the output data of the camera, the sensor, and the microphone, generates data in a first format from each of the output data collected during a period in which a value indicating the estimated strength of the emotion is less than a first predetermined value or a value indicating the determined importance is less than a second predetermined value, and stores the data in a specific memory, and generates data in a second format having a larger amount of information than the first format from each of the output data collected during a period in which the value indicating the estimated strength of the emotion is equal to or greater than the first predetermined value and the value indicating the determined importance is equal to or greater than the second predetermined value, and stores the data in the specific memory;
A data processing system comprising: The data processing system of claim 14, wherein the processing unit recognizes the content of statements made by the wearer or the other person when the wearer is conversing with another person based on at least a portion of the output data, and determines the importance level based on the recognized content of statements. A camera that captures the wearer's surroundings;
a sensor for detecting biological data of the wearer;
A microphone and
a control unit that sets a storage mode in a specific memory for each of the output data collected over a certain period of time according to a value of a specific index calculated based on the output data of each of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer;
Including,
The control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, and the content of the medical professional's remarks collected by the microphone based on each of the output data, and calculates the value of the index related to the content of the voice related to the wearer's symptoms based on the analyzed information. The terminal device of claim 16, wherein the storage mode includes at least some of the compression rate of each piece of output data, the storage format of the output data, deletion of all or part of each piece of output data, the storage period of each piece of output data, and the storage destination of each piece of output data. a camera that photographs the wearer and the surroundings of the wearer;
a sensor for detecting biological data of the wearer;
A microphone and
a control unit that sets a storage mode in a specific memory for each of the output data collected over a certain period of time according to a value of a specific index calculated based on the output data of each of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer;
Including,
The control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, and the wearer's body movements captured by the camera based on each of the output data, and calculates the value of the index based on the analyzed information. The terminal device of claim 18, wherein the storage mode includes at least some of the compression rate of each of the output data, the storage format of the output data, deletion of all or part of each of the output data, the storage period of each of the output data, and the storage destination of each of the output data. A camera that captures the wearer's surroundings;
a sensor for detecting biological data of the wearer;
A microphone and
a control unit that sets a storage mode in a specific memory for each of the output data collected over a certain period of time according to a value of a specific index calculated based on the output data of each of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer;
Including,
The control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, the content of statements collected by the microphone of a person other than the wearer, and the body movements of the person photographed by the camera based on each of the output data, and calculates the value of the index based on the analyzed information. The terminal device of claim 20, wherein the storage mode includes at least some of the compression rate of each of the output data, the storage format of the output data, deletion of all or part of each of the output data, the storage period of each of the output data, and the storage destination of each of the output data. A camera that captures the wearer's surroundings;
a sensor for detecting biological data of the wearer;
A microphone and
a control unit that sets a storage mode of each of the output data in a specific memory according to a value of an importance index that is an index representing a degree of importance in the wearer's activity, calculated based on the output data of each of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer;
Including,
The control unit sets the storage mode to store the output data in a large capacity when the calculated value of the importance index is high, sets the storage mode to store the output data in a small capacity when the calculated value of the importance index is low, and sets the storage mode to store the output data in a capacity between a large capacity and a small capacity when it is difficult to calculate the value of the importance index. The terminal device of claim 22, wherein the control unit associates each piece of output data with the importance index and records it in the memory.