JP2023183280A - Electronic equipment, control method of electronic equipment, and control program of electronic equipment - Google Patents

Abstract

To provide electronic equipment, a control method of electronic equipment, and a control program of electronic equipment capable of improving estimation accuracy for an internal state of a target person.SOLUTION: Electronic equipment 1 includes an output unit 40 for outputting presentation information to be presented to a target person, an acquisition unit 30 for acquiring target person information on the target person, and a control unit 10 for estimating an internal state of the target person on the basis of the presentation information and the target person information.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an electronic device, a method of controlling the electronic device, and a control program.

In asynchronous remote conferencing, a system is known that displays a variety of psychological state estimation information extracted from the user's voice, facial expressions/movements, and biological information to the remote conversation partner in addition to video and audio information ( For example, see Patent Document 1, etc.).

Japanese Patent Application Publication No. 2006-262010

There is a need to improve the accuracy of estimating a subject's internal state.

An object of the present disclosure is to provide an electronic device, a control method for the electronic device, and a control program that can improve the estimation accuracy of a subject's internal state.

An electronic device according to an embodiment of the present disclosure includes an output section, an acquisition section, and a control section. The output unit outputs presentation information to be presented to the target person. The acquisition unit acquires target person information regarding the target person. The control unit estimates the internal state of the subject based on the presentation information and the subject information.

A method for controlling an electronic device according to an embodiment of the present disclosure includes: outputting presentation information to be presented to a target person, acquiring target person information regarding the target person, and based on the presentation information and the target person information. and estimate the internal state of the subject.

A control program for an electronic device according to an embodiment of the present disclosure causes the electronic device to output presentation information to be presented to a target person. The control program causes the electronic device to acquire target person information regarding the target person. The control program causes the electronic device to estimate the internal state of the subject based on the presentation information and the subject information.

According to the electronic device, the control method for the electronic device, and the control program according to the present disclosure, the accuracy of estimating the internal state of the subject can be improved.

FIG. 1 is a block diagram showing a schematic configuration of an electronic device according to an embodiment. FIG. 2 is a diagram showing an example of a model showing the relationship between internal states and biological information. It is a flowchart which shows the example of a procedure of the control method which estimates an internal state based on stimulus information. 2 is a flowchart illustrating an example of a procedure of a control method for estimating an internal state by adjusting a stimulus. It is a flowchart which shows the example of a procedure of the control method which measures a stimulus and estimates an internal state.

Hereinafter, an embodiment of an electronic device 1 (see FIG. 1) to which the present disclosure is applied will be described with reference to the drawings. The following description may also serve as an explanation of an embodiment of a control method and a control program for the electronic device 1 to which the present disclosure is applied.

In the present disclosure, the "electronic device 1" may be a device driven by electric power. The electronic device 1 according to one embodiment estimates the internal state of the subject, such as the subject's emotion or concentration level. Here, the "target person" may be a person (typically a human being) whose internal state is estimated by the electronic device 1 according to one embodiment. Furthermore, in the present disclosure, a "user" may be a person (typically a human) who uses the electronic device 1 according to one embodiment. The "user" may be the same person as the "target person" or may be a different person. Further, the "user" and the "target person" may be humans or animals other than humans.

The electronic device 1 according to one embodiment may be various types of devices. For example, the electronic device 1 according to one embodiment may be any device such as a specially designed terminal, a general-purpose smartphone, a tablet, a phablet, a notebook computer, a computer, or a server. good. Furthermore, the electronic device 1 according to one embodiment may have a function of communicating with other devices, such as a mobile phone or a smartphone. Here, the above-mentioned "other device" may be a device such as a mobile phone or a smartphone, or may be any device such as a base station, server, dedicated terminal, or computer. good. Further, "other equipment" in the present disclosure may be equipment, devices, etc. that are driven by electric power. The electronic device 1 according to one embodiment may be configured to be able to communicate with other devices by wire and/or wirelessly.

An example of an embodiment in which the electronic device 1 is a device such as a server or a terminal that connects to an online conference will be described below. In this case, the electronic device 1 takes into consideration the stimulus for content viewers included in content such as images or audio of the online conference, and determines a predetermined internal state (for example, a predetermined psychological state, etc.) of the participants in the online conference. emotion or concentration level, etc.) may be estimated. The electronic device 1 notifies the organizer or facilitator of the online conference, or the participants themselves, of the estimation results of the participants' internal states, or outputs an alarm based on the estimation results of the participants' internal states. It's fine.

Content such as images or audio of the online conference provided by the electronic device 1 is also referred to as presentation information. Information regarding stimulation for the viewer of the presentation information, which is included in the presentation information, is also referred to as stimulation information.

(Example of configuration of electronic device 1)
As shown in FIG. 1, an electronic device 1 according to an embodiment may include a control section 10, a storage section 20, an acquisition section 30, and an output section 40. Further, the control unit 10 may be configured to include a stimulation adjustment unit 12 and a stimulation measurement unit 14, as shown in FIG. The electronic device 1 according to one embodiment may include all the functional units shown in FIG. 1 or may not include at least some of the functional units shown in FIG. 1. For example, the electronic device 1 according to one embodiment may include only the control unit 10 shown in FIG. 1. In this case, the electronic device 1 according to one embodiment may be configured to be connected to the storage section 20, the acquisition section 30, the output section 40, etc., which are prepared as external devices.

The control unit 10 controls and/or manages the entire electronic device 1, including each functional unit that constitutes the electronic device 1. Control unit 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities to perform various functions. The control unit 10 may be implemented by a single processor, several processors, or individual processors. A processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). A processor may be implemented as a plurality of communicatively connected integrated and discrete circuits. The processor may be implemented based on various other known technologies.

Control unit 10 may include one or more processors and memory. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for specific processing. A dedicated processor may include an Application Specific Integrated Circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The control unit 10 may be either an SoC (System-on-a-Chip) or an SiP (System In a Package) in which one or more processors cooperate. The control unit 10 controls the operation of each component of the electronic device 1.

The control unit 10 may be configured to include at least one of software and hardware resources, for example. Furthermore, in the electronic device 1 according to one embodiment, the control unit 10 may be configured by specific means in which software and hardware resources cooperate. At least one of the stimulation adjustment section 12 and the stimulation measurement section 14 included in the control section 10 may be configured to include at least one of software and hardware resources. Furthermore, in the electronic device 1 according to one embodiment, at least one of the stimulation adjustment section 12 and the stimulation measurement section 14 may be configured by concrete means in which software and hardware resources cooperate.

The stimulus adjustment unit 12 adjusts the stimulus included in the presentation information. The stimulus adjustment unit 12 may strengthen or weaken the stimulus included in the presentation information. The stimulus adjustment unit 12 may add a stimulus to the presentation information. The stimulus measurement unit 14 measures the stimulus included in the presentation information.

The stimulus may be presented by displaying a predetermined image including characters, symbols, figures, or the like. The predetermined image may include a still image or a moving image. The predetermined image may include, for example, an image of a timer. Stimuli may be presented by changing the display. For example, the stimulus may be presented by brightening or dimming the display brightness of at least a portion of the screen. The stimulus may also be presented by flashing the display on at least a portion of the screen.

The stimulus may be presented by outputting a predetermined sound. The predetermined sound may include, for example, a warning or notification sound such as a buzzer or a chime. Stimuli may be presented by changing audio. For example, the stimulus may be presented by increasing or decreasing the volume. Stimuli may also be presented by changing the frequency of the audio. Further, the stimulus adjustment unit 12 may define as a stimulus the time when the sound in a meeting or the like is temporarily interrupted, or the time when the volume of the sound or the like becomes lower than a predetermined value. Further, the stimulus adjustment unit 12 may define, as a stimulus, a sound associated with a specific emotion, such as laughter or yelling at a meeting or the like.

The storage unit 20 stores various information. The storage unit 20 may store, for example, programs executed by the control unit 10, results of processes executed by the control unit 10, and the like. Furthermore, the storage unit 20 may function as a work memory for the control unit 10. The storage unit 20 may be connected to the control unit 10 by wire and/or wirelessly, as shown in FIG. The storage unit 20 may include, for example, at least one of a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage unit 20 may include, for example, a semiconductor memory, but is not limited thereto, and may include any storage device. For example, the storage unit 20 may be configured as a storage medium such as a memory card inserted into the electronic device 1 according to one embodiment. The storage unit 20 may be an internal memory of a CPU used as the control unit 10, or may be a memory connected to the control unit 10 as a separate body.

The acquisition unit 30 is configured to be able to acquire information regarding the target person. Information regarding the target person is also referred to as target person information. The acquisition unit 30 outputs the target person information to the control unit 10.

The acquisition unit 30 may be configured to include a camera that photographs the face or body of the target person, for example, as the target person information. The camera may be configured to include an image sensor that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera may be a visible light camera or various other cameras such as an infrared camera.

If the target person is a participant in an online conference, the acquisition unit 30 acquires an image of the participant using a terminal connected to the online conference. In one embodiment, the acquisition unit 30 may capture still images of the subject at predetermined time intervals (eg, 30 frames per second), for example. Further, the acquisition unit 30 may capture images of the subject as a continuous moving image, for example. The acquisition unit 30 may acquire the image of the subject in various data formats such as RGB data and/or infrared data.

The acquisition unit 30 may be configured as, for example, an in-camera of a terminal in order to capture images of participants in an online conference. The acquisition unit 30 may be configured as a camera connected to the terminal.

The acquisition unit 30 outputs the captured image of the participant to the control unit 10. The acquisition unit 30 may be connected to the control unit 10 by wire and/or wirelessly, as shown in FIG. The control unit 10 may analyze, for example, the movement of the subject's gaze direction or viewpoint from the image acquired by the acquisition unit 30. The acquisition unit 30 may output the result of analyzing the movement of the target person's gaze direction, viewpoint, etc. from the photographed image of the target person to the control unit 10 as target person information.

The acquisition unit 30 may include a sensor that detects various biological information such as the subject's heart rate, blood pressure, respiratory rate, body temperature, or amount of perspiration as subject information. The acquisition unit 30 is a movement acquisition unit that acquires movement information of the target person's hands, fingers, body, feet, head, etc. as target person information, or a voice acquisition unit that acquires voice information emitted from the target person. may include.

The acquisition unit 30 may be configured to be able to acquire information or data input from a user or a target person using an input device. The acquisition unit 30 may be configured to be able to acquire information or data through communication with other devices.

When the electronic device 1 functions as a client of an online conference, the output unit 40 may output conference content such as audio or images of the conference to participants of the online conference. Further, the output unit 40 may output a predetermined alarm for calling attention to the user of the electronic device 1 based on a predetermined signal such as an alarm signal output from the control unit 10. Therefore, as shown in FIG. 1, the output section 40 may be connected to the control section 10 by wire and/or wirelessly. The output unit 40 may output a predetermined alarm in a manner that stimulates at least one of the user's auditory sense, visual sense, or tactile sense, such as sound, audio, light, text, video, or vibration.

The output unit 40 may include, for example, an audio output device such as a buzzer or a speaker. The output unit 40 is configured to include at least one of various output devices such as a light emitting device such as an LED (Light Emitting Diode), a display device such as an LCD (Liquid Crystal Display), or a tactile sensation presentation device such as a vibrator. It's fine.

In one embodiment, the output unit 40 outputs a signal indicating that the target person's own concentration level has decreased, for example, when it is estimated that the target person's own concentration level has decreased to a predetermined threshold value or less as an internal state of the target person. It is possible to output a corresponding alarm. For example, in one embodiment, the output unit 40 that outputs visual information may notify the subject that the subject's concentration level has decreased by emitting light, a predetermined display, or the like. In one embodiment, the output unit 40 that outputs the auditory information may notify the subject that the subject's concentration level has decreased by using a predetermined sound, voice, or the like. Further, in one embodiment, the output unit 40 that outputs tactile information may notify the subject that the subject's concentration level has decreased by using predetermined vibrations or the like. In this way, the subject himself/herself can know that, for example, his or her level of concentration on the online meeting is decreasing.

The output unit 40 outputs information to the host or facilitator of the online conference, or other participants when it is estimated that the concentration level of the participants in the online conference as the target audience has decreased below a predetermined threshold. It may be reported that there are participants whose concentration level has decreased. The output unit 40 may notify that there are participants whose concentration level has decreased in various ways.

(Example of internal state estimation operation)
An example of the operation of estimating the internal state of a subject by the electronic device 1 according to an embodiment will be described below.

The electronic device 1 according to one embodiment uses an auto encoder to perform machine learning based on images of participants in an online conference as targets, so as to determine the degree of concentration of participants, etc. Estimate the internal state. An autoencoder is an architecture for neural networks. The autoencoder may be a neural network that includes an encoder and a decoder. In the electronic device 1 according to one embodiment, the control unit 10 may include a function as a self-encoder. That is, the control unit 10 of the electronic device 1 according to one embodiment may have functions as an encoder and a decoder.

As shown in FIG. 2, the neural network that functions as an autoencoder in the electronic device 1 according to one embodiment may be represented as a deep generative model 50. The deep generative model 50 includes a block 51 corresponding to information Y representing the subject's internal state such as concentration level or emotion, a block 52 corresponding to information E representing noise such as observation noise, and a block 52 corresponding to information Y representing the subject's internal state such as concentration level or emotion, and a block 52 corresponding to information E representing noise such as observation noise. A block 53 corresponding to information S representing the target person and a block 54 corresponding to information X representing the target person's response including target person information such as biological information of the target person are provided. Block 54 is influenced by blocks 51, 52 and 53. That is, information X representing the subject's response is generated due to information Y representing the subject's internal state, information E representing noise, and information S representing the stimulus received by the subject.

The control unit 10 uses the causal relationship represented by the deep generative model 50 to determine the target person based on information X representing the target person's response, information S representing the stimulus received by the target person, and information E representing noise. Information Y representing the internal state of can be estimated. The control unit 10 may generate a trained model for estimating the information Y representing the internal state of the subject using the causal relationship represented by the deep generative model 50 by performing machine learning.

In the present embodiment, the control unit 10 estimates unknown information regarding the target person based on the target person information and a temporary internal state of the target person, and reconstructs the target person information based on the estimation result of the unknown information. Suppose we use a model that The control unit 10 sets various temporary internal states to be input to the model, and sets the state when the biological information reconstructed by the model has the highest degree of reproduction of the original biological information input to the model. The hypothetical internal state of the subject may be estimated as the subject's internal state. The control unit 10 is not limited to the above-mentioned example, and may estimate the internal state of the subject using a model in another mode.

For example, the control unit 10 may intentionally generate a situation in which the subject's internal state becomes a predetermined state, and obtain information representing the predetermined state as information Y representing the subject's internal state. For example, the control unit 10 may intentionally create a state in which the target participants in the online conference are completely focused only on the content of the online conference. Further, the control unit 10 may, for example, intentionally create a state in which the participants in the online conference who are the target participants are not able to fully concentrate on the content of the online conference. Information Y indicating the internal state may be set, for example, so that Y=0 in a state of concentration and Y=1 in a state of not concentration.

A state in which a participant in an online conference cannot fully concentrate on the content of the online meeting may be a state in which the participant's attention is taken up by tasks other than the online conference. For example, a state may be generated in which a participant simultaneously performs work unrelated to the online conference, such as predetermined mental arithmetic, as a task other than the online conference. By adjusting the complexity of the task, the degree to which participants are unable to concentrate on the content of the online meeting may be adjusted. The complexity of the task can be adjusted, for example, by the difficulty level of the mental arithmetic problem. For example, a situation in which participants in an online meeting simultaneously perform very simple mental arithmetic during the online meeting is a state in which participants are not fully focused on the content of the online meeting, but their level of concentration is relatively high. It may be assumed that there is. Further, a state in which participants simultaneously perform fairly complex mental calculations during an online conference may be considered to be a state in which the participants' concentration level is relatively low.

The control unit 10 may acquire subject information in a situation where the subject's internal state is in a predetermined state as information X representing the subject's response. The control unit 10 may intentionally generate a predetermined stimulus that the subject receives, and obtain information representing the predetermined stimulus as information S representing the stimulus that the subject receives. The control unit 10 may acquire information representing various types of noise such as white noise as the information E representing noise. The control unit 10 may generate a learned model by learning the acquired information as teacher data.

As described above, the control unit 10 determines, depending on the degree to which the subject information reconstructed by the model based on the information Y indicating various internal states reproduces the original subject information input into the model. The validity of the information Y indicating the internal state may be determined. For example, if the subject information reconstructed based on information Y1 indicating a certain internal state has a high degree of reproducing the original subject information, the validity of the information Y1 indicating the internal state is high (i.e., close to the correct answer). ). On the other hand, if the subject information reconstructed based on the information Y2 indicating a certain internal state has a low degree of reproducing the original subject information, the validity of the information Y2 indicating the internal state is low (that is, it is far from the correct answer). ).

<Estimation of internal information considering stimulus>
As described above, when a subject receives a stimulus, subject information may change under the influence of the stimulus. Further, the influence of stimulation on biological information can become larger or smaller depending on the internal state of the subject. The control unit 10 of the electronic device 1 according to the present embodiment estimates the internal state of the subject using the deep generative model 50 that takes into account the stimulation given to the subject. Hereinafter, an operation example will be described in which the internal state of a participant is estimated by taking into consideration the stimulation that the participant receives from information presented in the online conference, using the participant of the online conference as the target.

It is assumed that the electronic device 1 according to this embodiment is a client connected to a host that provides an online conference. The host may be configured as a server device or a cloud system. The electronic device 1 acquires the contents of the online conference from the host, and presents the contents of the online conference from the electronic device 1 to the participants participating in the online conference. The electronic device 1 presents information such as voices spoken by other participants, videos of other participants, and images to be shared as the content of the online conference. Information such as the content of the online conference presented by the electronic device 1 is included in the presentation information.

The content of the online conference may include content that provides stimulation to participants. Participants can be stimulated, for example, when presented with content that differs from what they expected. Contents that can stimulate participants include, for example, an unexpected remark or laughter during a meeting, or a silent period in which no one speaks.

The control unit 10 may estimate the internal state of the target person while the online conference is being held. The control unit 10 can estimate the internal state of the target person based on the target person information and the content of the online conference. The stimulation received by the subject from the content of the online meeting can influence the subject's information. Furthermore, the magnitude of the influence of the stimulus on the subject's information may vary depending on the subject's internal state. In other words, the degree to which certain content stimulates a subject depends on the subject's internal state. When the presented information includes content that can stimulate the subject, the control unit 10 can estimate the internal state of the subject based on the influence of the stimulus on the subject's information.

The control unit 10 extracts content that can stimulate the subject from the presentation information. Information representing content that can stimulate the subject is included in the stimulation information. The control unit 10 acquires target person information. When there is a possibility that the subject has received a stimulus, the control unit 10 extracts biological information for a period in which the influence of the stimulus may have appeared.

The control unit 10 generates a plurality of states as temporary internal states of the subject. The control unit 10 inputs the stimulus information of the subject and the extracted biometric information into a model to which each of the plurality of states generated as a temporary internal state of the subject is applied, and inputs the biometric information reconstructed by each model. get.

The control unit 10 may determine that the higher the degree of reproducibility of the reconstructed biological information, the higher the validity of estimating the temporary internal state as the internal state of the subject. The control unit 10 may compare the reproducibility of biological information reconstructed by each model, and estimate the temporary internal state applied to the model with the highest reproducibility as the internal state of the subject. .

For example, the control unit 10 may use the subject's concentration level as the information Y indicating the subject's internal state. In this case, the control unit 10 may assume a plurality of values for the degree of concentration. A value based on an assumed concentration degree value is also referred to as an assumed value. The control unit 10 inputs the stimulus information of the subject and the extracted biological information to a model to which each of the plurality of assumed values is applied, and obtains the biological information reconstructed by each model. The control unit 10 may estimate the assumed value applied to the model with the highest degree of reproducibility as the value of the concentration level of the participants.

The subject's internal state may include the subject's emotions. The subject's emotion may include the subject's concentration level. The subject's emotion is not limited to concentration level, and may be expressed as, for example, happiness, anger, sadness, or the like. The subject's emotions may be indicated by, for example, an index that combines degrees of happiness, anger, sadness, and happiness. The emotion of the subject may be indicated by the degree of pleasure or discomfort, the degree of security or anxiety, or the like. Information indicating the emotion of the subject is also referred to as an emotion label. The subject's internal state may include the subject's willingness to participate in an online meeting. The subject's internal state may include the subject's understanding of the online meeting.

The control unit 10 may generate the temporary internal state as a combination of a plurality of parameters such as the subject's concentration level or emotion. The control unit 10 may apply to the model a temporary internal state obtained by combining all possible values for each parameter in a round-robin manner. The control unit 10 may appropriately set the number of assumed values for each parameter, and apply to the model a temporary internal state that is a combination of the set number of assumed values for each parameter. The control unit 10 may narrow down and generate several combinations of values of each parameter, and apply the temporary internal state of each generated combination to the model. The control unit 10 may reconstruct the subject information for each of the temporary internal states applied to the model, and determine the temporary internal state indicating the estimation result of the subject's internal state based on the degree of reproducibility.

The control unit 10 may output the estimation result of the subject's internal state to the subject, the user, or the like. For example, a facilitator of an online conference can adjust the progress of an online conference by understanding the concentration level, motivation, understanding level, etc. of each target person based on the estimation result of each target person's internal state.

The control unit 10 may output a warning to the subject, the user, etc. based on the estimation result of the subject's internal state. The control unit 10 may output a warning when the estimation result of the subject's internal state satisfies the warning condition. The alarm condition may include, for example, that the concentration level of the subject is equal to or higher than the determination threshold. The warning condition may include, for example, that the value of the subject's emotion label is outside the determination range. For example, the control unit 10 determines that the alarm condition is satisfied when it is estimated that the target person's concentration level is decreasing, and instructs the target person or user, etc., to take measures to increase the target person's concentration level. An alarm may be output to prompt the user. For example, when it is estimated that the subject's emotions are worsening, the control unit 10 determines that the alarm condition is satisfied, and instructs the subject or user to take measures to improve the subject's emotions. A warning may be output to prompt the user. The control unit 10 may output a warning through the output unit 40.

<Example of control method procedure>
The control unit 10 of the electronic device 1 may execute a method of controlling the electronic device 1 including the procedure of the flowchart illustrated in FIG. 3 . The method for controlling the electronic device 1 may be implemented as a control program for the electronic device 1 that is executed by a processor that constitutes the control unit 10. The control program for the electronic device 1 may be stored in a non-transitory computer-readable medium.

The control unit 10 of the electronic device 1 functioning as a client of the online conference outputs presentation information including stimulation from the output unit 40 (step S1). The control unit 10 acquires subject information (step S2). The control unit 10 acquires stimulus information regarding the stimulus included in the presentation information (step S3).

The control unit 10 estimates the internal state of the subject based on the stimulus information and the subject information (step S4). The control unit 10 determines whether the estimation result of the subject's internal state satisfies the alarm condition (step S5). If the alarm condition is satisfied (step S5: YES), the control unit 10 outputs an alarm through the output unit 40 (step S6). When the alarm condition is not satisfied (step S5: NO), the control unit 10 skips the procedure of step S6 and proceeds to the procedure of step S7. The control unit 10 outputs the estimation result (step S7). After executing the procedure of step S7, the control unit 10 ends the execution of the procedure of the flowchart of FIG.

The control unit 10 may omit the step S7 to output the estimation result after the warning output step S6. The control unit 10 may output the estimation result without executing the alarm condition determination procedure in step S5 and the alarm output procedure in step S6.

<Summary>
As described above, the subject's internal state can be influenced by the stimuli received by the subject. The control unit 10 of the electronic device 1 according to the present embodiment can estimate the internal state of the subject in consideration of the stimulus received by the subject. The accuracy of estimating the subject's internal state can be improved compared to the case where the stimulus is not considered.

(Other embodiments)
Other embodiments will be described below.

<Stimulus adjustment>
In the embodiments described above, the control unit 10 of the electronic device 1 outputs presentation information including stimulation. The control unit 10 may adjust the stimulus included in the presentation information and estimate the subject's internal state based on the adjusted stimulus. The control unit 10 may generate information for adjusting stimulation using the stimulation adjustment unit 12. Information for adjusting stimulation is also referred to as stimulation adjustment information. The control unit 10 may cause the output unit 40 to output information obtained by adding the stimulation adjustment information to the presentation information. Information obtained by adding stimulation adjustment information to presentation information is also referred to as adjusted presentation information.

For example, the control unit 10 may generate, as the stimulus adjustment information, an instruction to increase a timer that displays the elapsed time of the online conference, which is included in the screen that is displayed as the content of the online conference. The control unit 10 causes the output unit 40 to display a screen with a larger timer. If a participant in the online conference notices a change in the timer, he or she may input an operation on the timer or watch the timer. In other words, subject information may change.

For example, the control unit 10 may generate an instruction to gradually reduce the volume of the online conference as the stimulation adjustment information. The control unit 10 causes the output unit 40 to output the sound with a reduced volume. If a participant in an online conference notices a change in volume, he or she may input an operation to adjust the volume or move his or her ear closer to the speaker. In other words, subject information may change.

The control unit 10 can estimate the internal state of the subject based on the stimulation adjustment information and the subject information. The control unit 10 can estimate the subject's concentration level, for example, based on the subject's reaction to the adjusted stimulus. If the subject responds to the adjusted stimulus, the control unit 10 can estimate the subject's internal state as a state in which the subject is able to concentrate on the screen or audio of the online conference. Conversely, if the subject's reaction to the adjusted stimulus is small, the control unit 10 can estimate the subject's internal state as a state in which the subject is unable to concentrate on the screen or audio of the online conference. The control unit 10 may estimate the subject's internal state based on the time that the subject reacts to the stimulus.

The control unit 10 may execute a method of controlling the electronic device 1, including the procedure of the flowchart illustrated in FIG. 4. The method for controlling the electronic device 1 may be implemented as a control program for the electronic device 1 that is executed by a processor that constitutes the control unit 10. The control program for the electronic device 1 may be stored in a non-transitory computer-readable medium.

The control unit 10 of the electronic device 1 functioning as a client for the online conference generates stimulation adjustment information (step S11). The control unit 10 outputs adjusted presentation information in which the stimulation adjustment information is added to the presentation information (step S12). The control unit 10 acquires subject information corresponding to the adjusted stimulation (step S13).

The control unit 10 estimates the internal state of the subject based on the stimulus adjustment information and the subject information (step S14). The control unit 10 determines whether the estimation result of the subject's internal state satisfies the alarm condition (step S15). If the alarm condition is satisfied (step S15: YES), the control unit 10 outputs an alarm through the output unit 40 (step S16). If the alarm condition is not satisfied (step S15: NO), the control unit 10 skips the procedure of step S16 and proceeds to the procedure of step S17. The control unit 10 outputs the estimation result (step S17). After executing the procedure of step S17, the control unit 10 ends the execution of the procedure of the flowchart of FIG.

The control unit 10 may omit the step S17 to output the estimation result after the warning output step S16. The control unit 10 may output the estimation result without executing the alarm condition determination procedure in step S15 and the alarm output procedure in step S16.

<Measurement of stimulation>
In the embodiments described above, the control unit 10 of the electronic device 1 outputs presentation information including stimulation. The control unit 10 may use the stimulation measuring unit 14 to measure the stimulation included in the presentation information. Information regarding the measured stimulation is also referred to as stimulation measurement information. The control unit 10 may estimate the subject's internal state based on the stimulus measurement information.

The control unit 10 may measure, for example, unexpected sounds such as sudden laughter included in the audio of the online conference, or characteristic sounds such as a state where participants are silent. A participant in an online conference as a target person may take some action in response to the characteristic sound. For example, participants may be tempted to laugh at the sound of laughter or be confused by the silence. The participant's reaction may appear in the participant's facial expressions, expressions, or eye movements. In other words, subject information may change.

The control unit 10 may measure characteristic displays, such as changes in display content or reactions of other participants, included in images of the online conference, for example. A participant in an online conference as a target person may take some action in response to the characteristic display. The participant's reaction may appear in the participant's facial expressions, expressions, or eye movements. In other words, subject information may change.

The control unit 10 may measure the time from when a characteristic sound or display included in the online conference occurs until the target person reacts.

The control unit 10 can estimate the internal state of the subject based on the stimulus measurement information and the subject information. The control unit 10 can estimate the degree of concentration of the subject, for example, based on the subject's reaction to the measured stimulus. If the subject responds to the measured stimulus, the control unit 10 can estimate the internal state of the subject, assuming that the subject is able to concentrate on the screen or audio of the online conference. Conversely, if the subject's reaction to the adjusted stimulus is small, the control unit 10 can estimate the subject's internal state as a state in which the subject is unable to concentrate on the screen or audio of the online conference. The control unit 10 may estimate the subject's internal state based on the time that the subject reacts to the stimulus.

The control unit 10 may execute a method of controlling the electronic device 1, including the procedure of the flowchart illustrated in FIG. The method for controlling the electronic device 1 may be implemented as a control program for the electronic device 1 that is executed by a processor that constitutes the control unit 10. The control program for the electronic device 1 may be stored in a non-transitory computer-readable medium.

The control unit 10 of the electronic device 1 functioning as a client for the online conference outputs presentation information (step S21). The control unit 10 acquires subject information (step S22). The control unit 10 measures the stimulation included in the presentation information and generates stimulation measurement information (step S23).

The control unit 10 estimates the internal state of the subject based on the stimulus measurement information and the subject information (step S24). The control unit 10 determines whether the estimation result of the subject's internal state satisfies the alarm condition (step S25). If the alarm condition is satisfied (step S25: YES), the control unit 10 outputs an alarm through the output unit 40 (step S26). If the alarm condition is not satisfied (step S25: NO), the control unit 10 skips the procedure of step S26 and proceeds to the procedure of step S27. The control unit 10 outputs the estimation result (step S27). After executing the procedure of step S27, the control unit 10 ends the execution of the procedure of the flowchart of FIG.

The control unit 10 may omit the step S27 to output the estimation result after the warning output step S26. The control unit 10 may output the estimation result without executing the alarm condition determination procedure in step S25 and the alarm output procedure in step S26.

<Example of execution entity of internal state estimation>
As described above, the electronic device 1 according to an embodiment of the present disclosure is configured as a terminal for participating in an online conference. Further, a predictive model generation method or an internal state estimation method according to an embodiment of the present disclosure is executed by a terminal.

Here, the online conference may be held by a server device or a cloud system connected via a network or the like. The electronic device 1 according to an embodiment of the present disclosure may be configured as a server device or a cloud system that provides online conferences to terminals. Further, the predictive model generation method or internal state estimation method according to an embodiment of the present disclosure may be executed by a server device or a cloud system that provides online conferences to terminals.

In other words, the electronic device 1 according to an embodiment of the present disclosure may be configured as a device that hosts an online conference. Further, the predictive model generation method or internal state estimation method according to an embodiment of the present disclosure may be executed by a device that hosts an online conference.

A server device, a cloud system, or a host device that provides an online conference may output an internal state estimation result to a terminal, or may output an alarm based on the internal state estimation result to a terminal.

Those skilled in the art can make various changes and modifications to the contents of the present disclosure based on the present disclosure. Accordingly, these variations and modifications are included within the scope of this disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. may be added to other embodiments so as not to be logically contradictory, or each functional unit, each means, each step, etc. of other embodiments may be added to other embodiments so as not to be logically inconsistent. It is possible to replace it with Furthermore, in each embodiment, it is possible to combine or divide a plurality of functional units, means, or steps into one. Further, each embodiment of the present disclosure described above is not limited to being implemented faithfully to each described embodiment, but may be implemented by combining each feature or omitting a part as appropriate. You can also do that.

The electronic device 1 according to an embodiment of the present disclosure may be mounted on a moving object, for example. The moving object may include, for example, a vehicle, a ship, or an aircraft. When the electronic device 1 according to an embodiment of the present disclosure is mounted on a moving body, the electronic device 1 is configured to detect a predetermined internal state (for example, a predetermined psychological state) of a person (driver or fellow passenger) on board the moving body such as a passenger car. state, emotion, concentration level, etc.) can be estimated. The electronic device 1 may estimate, as the internal state of the driver who drives the mobile object, the internal state such as the driver's emotion or concentration level while driving. The electronic device 1 may be configured as a server device or a cloud system that is communicably connected to a terminal mounted on a mobile object.

The vehicle may include, for example, an automobile, an industrial vehicle, a railroad vehicle, a residential vehicle, a fixed wing aircraft traveling on a runway, or the like. Motor vehicles may include, for example, cars, trucks, buses, motorcycles, trolleybuses, and the like. The industrial vehicle may include, for example, an agricultural or construction industrial vehicle. Industrial vehicles may include, for example, forklifts or golf carts. Agricultural industrial vehicles may include, for example, tractors, tillers, transplanters, binders, combines, lawn mowers, and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, shovels, crane trucks, dump trucks, road rollers, and the like. Vehicles may include those that are powered by human power. The classification of vehicles is not limited to the example described above. For example, a motor vehicle may include a road capable industrial vehicle. The same vehicle may be included in multiple classifications. A watercraft may include, for example, a personal watercraft (PWC), a boat, or a tanker. The aircraft may include, for example, a fixed wing aircraft or a rotary wing aircraft. Further, the "user" and "target person" of the present disclosure may be a person driving a moving body such as a vehicle or an aircraft, or may be a fellow passenger who is not driving the moving body.

In one embodiment, (1) an electronic device includes an output unit that outputs presentation information to be presented to a target person, an acquisition unit that acquires target person information regarding the target person, and the presentation information and the target person information. and a control unit that estimates the internal state of the subject based on.

(2) In the electronic device according to (1) above, the control unit may estimate the internal state of the subject based on stimulus information regarding stimulation to the subject included in the presentation information.

(3) In the electronic device of (1) or (2) above, the control unit may generate stimulus adjustment information that adjusts the stimulus included in the presentation information. The control unit may cause the output unit to output adjusted presentation information obtained by adjusting the presentation information based on the stimulation adjustment information. The control unit may estimate the internal state of the subject based on the subject information corresponding to the case where the adjusted presentation information is output.

(4) In any one of the electronic devices described in (1) to (3) above, the control unit may be configured to target the target person based on stimulus measurement information that measures the stimulus included in the presentation information and the target person information. The internal state of the person may be estimated.

(5) In any one of the electronic devices according to (1) to (4) above, the control unit may send at least one of an alarm or a notification to the output unit based on the estimation result of the internal state of the subject. You can decide whether to output it.

In one embodiment, (6) a method for controlling an electronic device outputs presentation information to be presented to a target person, acquires target person information regarding the target person, and based on the presentation information and the target person information. and estimate the internal state of the subject.

In one embodiment, (7) a control program for an electronic device causes the electronic device to output presentation information to be presented to a target person, acquires target person information regarding the target person, and combines the presentation information with the target person. The internal state of the subject is estimated based on the information.

1 Electronic equipment (10: control unit, 12: stimulation adjustment unit, 14: stimulation measurement unit, 20: storage unit, 30: acquisition unit, 40: output unit)
50 Deep generative model (51-54: block)

Claims (7)

an output unit that outputs presentation information to be presented to the target person;
an acquisition unit that acquires target person information regarding the target person;
An electronic device comprising: a control unit that estimates an internal state of the subject based on the presentation information and the subject information. The electronic device according to claim 1, wherein the control unit estimates the internal state of the subject based on stimulus information regarding stimulation to the subject included in the presentation information. The control unit includes:
generating stimulus adjustment information that adjusts the stimulus included in the presentation information;
causing the output unit to output adjusted presentation information obtained by adjusting the presentation information based on the stimulation adjustment information;
The electronic device according to claim 1, wherein the internal state of the target person is estimated based on the target person information corresponding to the case where the adjusted presentation information is output. The electronic device according to claim 1, wherein the control unit estimates the internal state of the subject based on stimulus measurement information obtained by measuring a stimulus included in the presentation information and the subject information. 5. The control unit according to claim 1, wherein the control unit determines whether to cause the output unit to output at least one of a warning or a notification based on an estimation result of the internal state of the subject. Electronics. Output the presentation information to be presented to the target person,
Obtaining target person information regarding the target person,
estimating the internal state of the subject based on the presentation information and the subject information;
How to control electronic equipment. Make the electronic device output presentation information to be presented to the target person,
causing the electronic device to acquire target person information regarding the target person;
causing the electronic device to estimate the internal state of the subject based on the presentation information and the subject information;
Control program for electronic equipment.

Images