JP7369884B1 - Information processing device, information processing method, and information processing program - Google Patents

Abstract

The present invention provides a technology that can improve the comfort of a user when viewing content. An information processing device according to the present disclosure automatically adjusts the volume of content viewed by a user. This information processing device acquires first photographed image data, which is photographed image data photographed by a predetermined photographing device, and represents a facial expression image of a user when the user views the content; The comfort state, which is the state of pleasure and displeasure felt by the user when viewing, is obtained by inputting the first photographed image data to a pre-learning model constructed by performing learning using predetermined image data. and automatically adjusting the volume of the content based on the user's comfort state so as to improve the user's comfort level when viewing the content. [Selection diagram] Figure 3

Description

The present invention relates to an information processing device, an information processing method, and an information processing program that automatically adjust the volume of content that a user views.

BACKGROUND ART Input interfaces using devices such as mice and touch panels have conventionally been used as user interfaces for operating mobile terminals, tablet terminals, smartphones, wearable terminals, personal computers, and the like. However, the user sometimes finds it troublesome to use such an input interface that is operated by the user.

Further, for example, when a user is driving a vehicle, the user cannot operate an input interface such as a mouse or a touch panel. Therefore, it is preferable that the above-mentioned terminal be operated automatically without the user's operation using an input interface.

Patent Document 1 discloses a content providing device that improves the uncomfortable feeling when the passenger becomes uncomfortable due to the content provided according to the passenger's feeling. In this technology, the first content is changed from the first content to the second content according to the passenger emotion estimated after the content output unit outputs the first content. A change in the content of is commanded.

Japanese Patent Application Publication No. 2018-101341

When operating mobile terminals, tablet terminals, smartphones, wearable terminals, personal computers, etc., users may find it troublesome to operate using input interfaces such as a mouse or touch panel. It is preferable that

Here, according to the technology described in Patent Document 1, for example, if the occupant's emotions deteriorate due to the output of the first content, the content is automatically changed from the first content to the second content, so that the user can It also seems that the hassle of operation can be reduced. However, the comfort that a user feels with content is affected not only by the genre of the content but also by the volume of the content. As described above, there is still room for improvement in technology for improving the comfort of users when they view content.

An object of the present disclosure is to provide a technology that can improve the comfort of a user when viewing content.

An information processing device of the present disclosure is an information processing device that automatically adjusts the volume of content that a user views. The information processing device acquires first photographed image data that is photographed by a predetermined photographing device and represents a facial expression image of the user when the user views the content; Input the first captured image data into a pre-learning model constructed by learning a comfort state, which is a state of pleasure and displeasure felt by the user when viewing the content, using predetermined image data. and automatically adjusting the volume of the content based on the comfort state so as to improve the user's comfort level when viewing the content by the user. Department.

In the above information processing device, the comfort state of the user is acquired by inputting the first captured image data representing the facial expression image of the user when the user views the content into the pre-learning model. Based on this comfort state, the volume of the content is automatically adjusted to improve the comfort of the user when viewing the content. If the user is feeling the same, the volume of the content will be automatically adjusted without any operation by the user. Thereby, the user's comfort can be improved while reducing the troublesome operation by the user.

The information processing device of the present disclosure includes the steps of acquiring first captured image data captured by a camera and representing a facial expression image of a user when the user views content; The first photographed image data is applied to a pre-learning model that is constructed by learning a comfort state, which is a state of pleasure and displeasure that the user feels when viewing the content, using image data that has been photographed in advance. and automatically adjusting the volume of the content based on the comfort state so as to improve the user's comfort level when viewing the content by the user. The control unit is configured to play an initial content for initial settings before the user views the content, and automatically changes the volume of the initial content during the playback, and according to the change in the volume. Further performing the following steps: obtaining a timing at which the user feels uncomfortable; and obtaining second captured image data, which is captured image data captured by the camera and represents a facial expression image of the user at the timing. However, the pre-learning model may be caused to perform learning using the second captured image data as training data. In this case, the control unit controls the captured image data generated by processing the second captured image data, in which the position of the person included in the second captured image data is arbitrarily changed. or/and represents an image of the user in which the color of the background included in the second captured image data has been arbitrarily changed, and/and the clothing of the person included in the second captured image data has been arbitrarily changed. The third captured image data may be further automatically generated, the third captured image data may be added to the teacher data, and the pre-learning model may be caused to perform learning. According to this, by automatically generating multiple pieces of third photographed image data based on one second photographed image data, it is possible to generate multiple pieces of photographed image data with different impressions, and to apply learning to the pre-learning model. It is possible to efficiently increase the amount of training data required for the process.

Further, in the information processing device of the present disclosure, the control unit may display captured image data captured by the imaging device, and the expression of the user when the user is viewing other content different from the content. further performing periodic acquisition of fourth photographed image data representing the image during playback of the other content, and causing the pre-learning model to perform learning using the fourth photographed image data as training data. It's okay.

In this case, the control unit labels, with respect to the fourth captured image data, an expression image of the user when the user adjusts the volume of the other content as an uncomfortable state, and The pre-learning model may perform learning by labeling the facial expression image of the user after a predetermined period of time has elapsed after adjusting the volume of other content as being in a pleasant state. Furthermore, the control unit may acquire the comfort state based on a matching ratio with the uncomfortable state and a matching ratio with the pleasant state. According to this, it is possible to suppress erroneous recognition of the user's comfort state as much as possible.

Further, the present disclosure can be understood from the aspect of an information processing method by a computer. That is, the information processing method of the present disclosure is an information processing method that automatically adjusts the volume of content viewed by a user, in which a computer receives photographed image data taken by a camera , and when the user views the content. a first acquisition step of acquiring first photographed image data representing a facial expression image of the user; and a comfort state, which is a state of pleasure and displeasure felt by the user when the user views the content, which has been photographed in advance. a second acquisition step of acquiring the first photographed image data by inputting the first photographed image data into a pre-learning model constructed by performing learning using the image data obtained by the user; and a second acquisition step of acquiring the content by the user based on the comfort state. automatically adjusting the volume of the content so as to improve the user's comfort when viewing the content; Playing content, automatically changing the volume of the initial content at the time of playback, and obtaining timing at which the user feels uncomfortable due to the change in volume, and photographed image data taken by the camera, , obtaining second captured image data representing an expression image of the user at the timing, and causing the pre-learning model to perform learning using the second captured image data as training data. Execute.

Further, the present disclosure can be viewed from the aspect of an information processing program. In other words, the information processing program of the present disclosure is an information processing program that automatically adjusts the volume of content viewed by a user, and the information processing program automatically adjusts the volume of content viewed by a user, and stores photographed image data captured by a camera on a computer when the user views the content. a first acquisition step of acquiring first photographed image data representing a facial expression image of the user; and a comfort state, which is a state of pleasure and displeasure felt by the user when the user views the content, which has been photographed in advance. a second acquisition step of acquiring the first photographed image data by inputting the first photographed image data into a pre-learning model constructed by performing learning using the image data obtained by the user; and a second acquisition step of acquiring the content by the user based on the comfort state. an automatic adjustment step of automatically adjusting the volume of the content so as to improve the comfort of the user when viewing the content; Reproducing initial content, automatically changing the volume of the initial content during playback, and obtaining timing at which the user feels uncomfortable due to the change in volume, and capturing image data taken by the camera. and acquiring second captured image data representing an expression image of the user at the timing, and causes the pre-learning model to perform learning using the second captured image data as training data. Execute.

According to the present disclosure, it is possible to improve the comfort of the user when the user views content.

FIG. 1 is a diagram showing a schematic configuration of an information processing system in a first embodiment. FIG. 2 is a diagram illustrating in more detail the components of a server included in the information processing system and the components of a user terminal that communicates with the server in the first embodiment. FIG. 3 is a diagram illustrating the flow of operation of the information processing system in the first embodiment. FIG. 2 is a diagram illustrating an initial setting screen for using the information processing system. FIG. 2 is a diagram for explaining identification results obtained from input to a pre-learning model and a neural network forming the pre-learning model in the first embodiment. FIG. 7 is a diagram illustrating the flow of operation of the information processing system in the second embodiment.

Embodiments of the present disclosure will be described below based on the drawings. The configurations of the following embodiments are illustrative, and the present disclosure is not limited to the configurations of the embodiments.

<First embodiment>
An overview of the information processing system in the first embodiment will be explained with reference to FIG. FIG. 1 is a diagram showing a schematic configuration of an information processing system in this embodiment. The information processing system 100 according to the present embodiment includes a network 200, a server 300, and a user terminal 400. Note that the information processing system of the present disclosure is a system that automatically adjusts the volume of the content that the user views, and the volume of the content that is being played on the user terminal 400 is adjusted according to a command from the server 300.

Network 200 is, for example, an IP network. As long as the network 200 is an IP network, it may be wireless, wired, or a combination of wireless and wired. For example, in the case of wireless communication, the user terminal 400 may be connected to a wireless LAN access point ( (not shown) and communicate with the server 300 via a LAN or WAN. Further, the network 200 is not limited to these examples, and may be, for example, a public switched telephone network, an optical line, an ADSL line, a satellite communication network, or the like.

Server 300 is connected to user terminal 400 via network 200. In addition, in FIG. 1, in order to simplify the explanation, one server 300 and four user terminals 400 are shown, but it goes without saying that the present invention is not limited to these.

The server 300 may be any electronic device as long as it has the processing capacity for arithmetic processing and processing such as data acquisition, generation, and updating; for example, a personal computer, server, mainframe, or other electronic device. It may be a device. That is, the server 300 can be configured as a computer having a processor such as a CPU or a GPU, a main storage device such as a RAM or ROM, and an auxiliary storage device such as an EPROM, a hard disk drive, or a removable medium. Note that the removable medium may be, for example, a USB memory or a disk recording medium such as a CD or a DVD. The auxiliary storage device stores an operating system (OS), various programs, various tables, and the like.

Furthermore, the server 300 can be configured as a cloud server using SaaS (Software as a Service), Paas (Platform as a Service), IaaS ( infrastructure as a service) may be used as appropriate.

The user terminal 400 may be any electronic device such as a mobile terminal owned by a user who uses the information processing system 100, and may be any other terminal device such as a mobile terminal, a tablet terminal, a smartphone, a wearable terminal, a personal computer, etc. Good too.

Next, based on FIG. 2, a detailed explanation will be given mainly of the components of the server 300. FIG. 2 is a diagram showing in more detail the components of the server 300 included in the information processing system 100, and also shows the components of the user terminal 400 that communicates with the server 300, in the first embodiment.

The server 300 has a communication unit 301, a storage unit 302, and a control unit 303 as functional units, and loads a program stored in an auxiliary storage device into a work area of a main storage device and executes it. By controlling each functional unit, it is possible to realize each function that meets a predetermined purpose in each functional unit. However, some or all of the functions may be realized by a hardware circuit such as an ASIC or FPGA.

Here, the communication unit 301 is a communication interface for connecting the server 300 to the network 200. The communication unit 301 includes, for example, a network interface board and a wireless communication circuit for wireless communication. The server 300 is communicably connected to the user terminal 400 and other external devices via the communication unit 301 .

The storage unit 302 includes a main storage device and an auxiliary storage device. The main storage device is a memory in which programs executed by the control unit 303 and data used by the control program are expanded. The auxiliary storage device is a device that stores programs executed by the control unit 303 and data used by the control programs. Note that the server 300 acquires data transmitted from the user terminal 400 or the like via the communication unit 301, and the storage unit 302 stores captured image data, which will be described later. Further, the storage unit 302 stores teacher data and a pre-learning model for acquiring a comfort state, which will be described later.

The control unit 303 is a functional unit that manages control performed by the server 300. The control unit 303 can be realized by an arithmetic processing device such as a CPU. The control unit 303 further includes four functional units: a first acquisition unit 3031, a second acquisition unit 3032, a volume adjustment unit 3033, and a learning unit 3034. Each functional unit may be realized by executing a stored program by a CPU. Note that since the learning unit 3034 requires a large amount of calculations associated with machine learning, it may be implemented by executing a stored program using a GPU. In this way, when the GPU is used for arithmetic processing associated with machine learning, high-speed processing becomes possible. In addition, in order to perform faster processing, a computer cluster is constructed using multiple computers equipped with such GPUs, and multiple computers included in this computer cluster perform parallel processing. Good too.

The first acquisition unit 3031 acquires first photographed image data representing a facial expression image of a user using the information processing system 100 when the user views content. Here, the above content is content such as videos and songs. The first photographed image data described above is photographed by a photographing device included in the user terminal 400 while the user is viewing the content using the user terminal 400. Note that a predetermined application for using the information processing system 100 is installed in the user terminal 400 in advance, and the application executes the process of photographing the above-mentioned image in the background when the content is played back. The photographed data is then uploaded to the server 300. Then, the first acquisition unit 3031 acquires the first captured image data and stores it in the storage unit 302.

Here, the user terminal 400 in this embodiment has a communication section 401, an input/output section 402, and a storage section 403 as functional sections. The communication unit 401 is a communication interface for connecting the user terminal 400 to the network 200, and includes, for example, a network interface board and a wireless communication circuit for wireless communication. The input/output unit 402 is a functional unit that displays information transmitted from the outside via the communication unit 401 and inputs the information when transmitting information to the outside via the communication unit 401. It is. Like the storage unit 302 of the server 300, the storage unit 403 includes a main storage device and an auxiliary storage device.

The input/output unit 402 further includes a display unit 4021, an operation input unit 4022, and an image/audio input/output unit 4023. The display unit 4021 has a function of displaying various information, and is realized by, for example, an LCD (Liquid Crystal Display) display, an LED (Light Emitting Diode) display, an OLED (Organic Light Emitting Diode) display, or the like. The operation input unit 4022 has a function of accepting operation input from the user, and is specifically realized by soft keys or hard keys of a touch panel or the like. The image/audio input/output unit 4023 has a function of receiving input images such as still images and videos. This is realized by a camera using an image sensor such as Oxide-Semiconductor (CMOS). Further, the image/audio input/output unit 4023 has a function of receiving input/output of audio, and is specifically realized by a microphone or a speaker.

Then, in the content (for example, a moving image), an image can be displayed by the display unit 4021, and an audio can be outputted by the speaker. The above image may then be captured by the camera.

The second acquisition unit 3032 acquires a comfort state that is a state of pleasure or displeasure felt by a user using the information processing system 100 when the user views content. Here, the second acquisition unit 3032 acquires the comfort state by inputting the first photographed image data described above to a pre-learning model described below.

The volume adjustment unit 3033 automatically adjusts the volume of the content based on the comfort state so as to improve the comfort of the user when the user views the content.

The learning unit 3034 is a functional unit that constructs a pre-learning model used in the processing by the second acquisition unit 3032, and its details will be described later.

Note that the control unit 303 functions as a control unit according to the present disclosure by executing the processes of the first acquisition unit 3031, the second acquisition unit 3032, the volume adjustment unit 3033, and the learning unit 3034.

Here, the flow of operation of the information processing system 100 in this embodiment will be explained. FIG. 3 is a diagram illustrating the flow of operations of the information processing system 100 in this embodiment. In FIG. 3, the flow of operations between the server 300 and the user terminal 400 in the information processing system 100 in this embodiment, and the processing executed by the server 300 and the user terminal 400 will be described.

In this embodiment, first, initial settings for using the information processing system 100 are performed. The server 300 transmits the initial content to the user terminal 400 in order to play the initial content for initial settings on the user terminal 400 of the user using the information processing system 100 (S101). Then, the initial content is played back on the user terminal 400 (S102). At this time, the initial content is played back with the volume automatically changing. Then, the discomfort timing is input by the user into the user terminal 400 (S103), and by transmitting it to the server 300, the server 300 can acquire the timing at which the user feels uncomfortable due to the change in the volume. (S104).

Here, FIG. 4 is a diagram illustrating an initial setting screen for using the information processing system 100. The screen SC1 illustrated in FIG. 4 is displayed on the display unit 4021 of the user terminal 400 of the user using the information processing system 100, and the initial setting start button SC11 is shown on the screen SC1 of FIG. 4(a). When the initial setting start button SC11 is pressed on the screen SC1 of FIG. 4(a), the screen transitions to the screen SC1 of FIG. 4(b), and the initial content is reproduced in the initial content reproduction field. At this time, the audio is played so that the volume of the initial content gradually increases, and if the user feels that the audio is too loud, the user can press the volume minus button SC12 to determine the timing at which the volume changes make the user feel uncomfortable. can be entered. In addition, on the screen SC1 of FIG. 4(c), the audio is played so that the volume of the initial content gradually decreases, and if the user feels that the audio is too low, the user can press the volume plus button SC13. , you can input the timing at which you feel uncomfortable due to changes in volume.

Then, returning to FIG. 3, the server 300 transmits a photographing command to the user terminal 400 for photographing second photographed image data representing the user's facial expression image at the above timing (S105). Then, the user terminal 400 acquires the information (S106) and photographs the second photographed image data (S107). Note that the second captured image data may be captured by a camera included in the user terminal 400. The second captured image data is then transmitted from the user terminal 400 to the server 300, and the server 300 acquires the second captured image data (S108).

Then, the server 300 causes the pre-learning model to perform learning using the second captured image data as training data (S109). As described above, the second photographed image data represents the facial expression image of the user at a time when the user feels uncomfortable due to a change in the volume of the content, so the pre-learning model is made to perform learning using this as training data. This makes it possible to identify the user's comfort state when viewing content using the pre-learning model.

Note that the server 300 may automatically generate third captured image data based on the second captured image data, add the third captured image data to the teacher data, and cause the pre-learning model to perform learning. good. Here, the above-mentioned third photographed image data is photographed image data generated by processing the second photographed image data, and the position of the person included in the second photographed image data has been arbitrarily changed. , or/and the background color included in the second captured image data has been arbitrarily changed, and/and the clothing of the person included in the second captured image data has been arbitrarily changed.

Here, even if the facial expressions of the users are the same, the impression of the photographed image data may differ depending on the surrounding environment of the user (background color, clothing, differences in brightness depending on position, etc.). Therefore, by automatically generating multiple pieces of third photographed image data based on one second photographed image data, it is possible to generate multiple pieces of photographed image data with different impressions, and to make the pre-learning model perform learning. The number of training data can be efficiently increased.

Further, the initial setting and learning process from S101 to S109 may be executed each time the user uses the information processing system 100, or may be executed only the first time the user uses the information processing system 100.

Then, in a state in which the above-described initial settings are completed, arbitrary content is played back on the user terminal 400 of the user who uses the information processing system 100 (S110). At this time, in the user terminal 400, a predetermined application installed in advance executes a process of photographing a facial expression image of the user in the background during reproduction of the content (S111). The first captured image data captured in this manner is then transmitted from the user terminal 400 to the server 300.

Then, the server 300 acquires the first captured image data transmitted from the user terminal 400 (S112), and stores the acquired first captured image data in the storage unit 302.

Then, the server 300 acquires the comfort state based on the first captured image data. This will be explained below.

The server 300 executes a process of calling the pre-learning model (S113). Here, the pre-learning model is a machine learning model used to obtain the comfort state based on the first captured image data, and is trained by the learning unit 3034 using the second captured image data as training data. It is pre-built by

Here, FIG. 5 is a diagram for explaining the identification results obtained from the input to the pre-learning model and the neural network forming the pre-learning model in this embodiment. In this embodiment, a neural network model generated by deep learning is used as the pre-learning model. The pre-learning model 30 in this embodiment includes an input layer 31 that receives input image data, and an intermediate layer (hidden layer) that extracts feature quantities representing unpleasant facial expressions of a person from the image data input to the input layer 31. 32, and an output layer 33 that outputs identification results based on feature amounts. In the example of FIG. 5, the pre-learning model 30 has one intermediate layer 32, the output of the input layer 31 is input to the intermediate layer 32, and the output of the intermediate layer 32 is input to the output layer 33. has been done. However, the number of intermediate layers 32 is not limited to one layer, and the pre-learning model 30 may have two or more intermediate layers 32.

Further, according to FIG. 5, each layer 31 to 33 includes one or more neurons. For example, the number of neurons in the input layer 31 can be set depending on input image data. Furthermore, the number of neurons in the output layer 33 can be set according to the comfort state that is the identification result.

Neurons in adjacent layers are then appropriately connected, and a weight (connection weight) is set for each connection based on the results of machine learning. In the example of FIG. 5, each neuron is connected to all neurons in adjacent layers, but the connection of neurons does not need to be limited to this example and can be set as appropriate.

Such a pre-learning model 30 performs supervised learning using, for example, teacher data that is a combination of image data including an image representing a person's facial expression and a label of an image representing an unpleasant facial expression of the person. Constructed with. Specifically, a combination of a feature amount and a label is given to a neural network, and the weights of connections between neurons are tuned so that the output of the neural network is the same as the label. In this way, a pre-trained model for learning the features of the training data and estimating the result from the input is obtained inductively.

Returning to FIG. 3, the server 300 acquires the comfort state by inputting the first photographed image data to the above pre-learning model (S114). Then, the server 300 determines whether the comfort state of the user when the user views the content is an uncomfortable state (S115). If an affirmative determination is made in the process of S115, the flow proceeds to the process of S116, and if a negative determination is made in the process of S115, the present flow returns to the process of S111.

When an affirmative determination is made in the process of S115, next, in S116, a volume adjustment process is executed. In the process of S116, the server 300 automatically adjusts the volume of the content so as to improve the comfort of the user when the user views the content. For example, if it is determined that the user's comfort state is in an uncomfortable state due to the audio of the content being too loud, the server 300 executes processing to lower the volume of the content. Further, for example, if it is determined that the user's comfort state is in an uncomfortable state due to the audio of the content being too low, the server 300 executes a process of increasing the volume of the content. Then, by transmitting such a command for volume adjustment processing from the server 300 to the user terminal 400, the volume is automatically adjusted in the user terminal 400 (S117). After the process in S117, the flow returns to the process in S111.

Then, when playing the content, the processes from S111 to S117 are repeatedly executed at a predetermined cycle, and when the playing of the content is finished, the execution of this flow is finished. According to the process described above, when the user feels uncomfortable due to the influence of the volume of the content, the volume of the content is automatically adjusted without any operation by the user. Therefore, the user's comfort can be improved while reducing the troublesomeness of the user's operations.

According to the information processing system 100 described above, it is possible to improve the comfort of the user when the user views content.

<Second embodiment>
An information processing system in the second embodiment will be described based on FIG. 6. In the present embodiment, the server 300 further acquires fourth captured image data. Here, the above-mentioned fourth photographed image data is photographed image data that represents a facial expression image of the user when the user is viewing any other content different from the content for which the volume is automatically adjusted. The camera of the user terminal 400 periodically takes pictures while the other content is being played back. Then, the server 300 causes the pre-learning model to perform learning using this fourth captured image data as teacher data.

Here, FIG. 6 is a diagram illustrating the flow of operation of the information processing system 100 in this embodiment. FIG. 6 describes the flow of operations between the server 300 and the user terminal 400 in the information processing system 100 in this embodiment, and the processing executed by the server 300 and the user terminal 400. Note that in each process shown in FIG. 6, the processes that are substantially the same as the processes shown in FIG.

In the example shown in FIG. 6, when any other content different from the content whose volume is automatically adjusted is played on the user terminal 400 (S201), the information is transmitted to the server 300 (S202). Here, in the user terminal 400, a predetermined application related to the information processing system 100 installed in advance is executed in the background, and when arbitrary content is played, the information is transmitted to the server 300 by the application.

Then, upon acquiring the above information (S202), the server 300 transmits a photographing command for photographing fourth photographed image data representing the user's facial expression image to the user terminal 400 (S203). Here, the server 300 transmits the above-mentioned photographing command so as to periodically photograph the fourth photographed image data while the above-mentioned other content is being played back. Then, the user terminal 400 acquires the information (S204) and periodically photographs the fourth photographed image data (S205). The fourth captured image data is then transmitted from the user terminal 400 to the server 300, and the server 300 acquires the fourth captured image data (S206).

Then, the server 300 causes the pre-learning model to perform learning using the fourth captured image data as training data (S207).

Here, as described above, the pre-learning model 30 uses teacher data that is a set of image data including an image representing a person's facial expression and a label of an image representing an unpleasant facial expression of the person. It can be constructed by performing learning. Therefore, in the present embodiment, an image of the user's facial expression when the user adjusts the volume of other content is labeled as an unpleasant state in the fourth captured image data, and when the user adjusts the volume of other content, The facial expression image of the user after a predetermined period of time has passed is labeled as a pleasant state, and the pre-learning model is caused to perform learning.

Note that the fourth photographed image data is periodically photographed. The above app also allows you to monitor when the user adjusts the volume of other content. Therefore, by extracting photographed image data that matches the above-mentioned timing from the fourth photographed image data that is periodically photographed, an image of the user's facial expression when the user adjusts the volume of other content is obtained. becomes possible. Then, in this embodiment, at the timing when the user adjusts the volume of other content in this way, it is estimated that the user is feeling uncomfortable, and the facial expression image of the user at that timing is labeled as being in an uncomfortable state. .

Additionally, once a user starts adjusting the volume while viewing content, there is a tendency to continue adjusting the volume until a comfortable volume is reached. In other words, the user finishes adjusting the volume when the volume is comfortable. Therefore, in the present embodiment, after the user adjusts the volume of other content and a predetermined period of time has elapsed, it is estimated that the volume has become comfortable due to the volume adjustment and the user feels comfortable, and the user at that time Label the facial expression image as a pleasurable state. Note that the above predetermined time is, for example, 30 seconds to 1 minute.

In the example shown in FIG. 6, in the process of S114, the comfort state is adjusted by inputting the first captured image data to the pre-learning model trained using the teacher data labeled as described above. be obtained. At this time, in this embodiment, for the identification results output from the pre-learning model for the first photographed image data, a matching ratio with the unpleasant state and a matching ratio with the pleasant state are output. In other words, the match ratio with the two labels described above is output. Then, based on the matching ratio with the unpleasant state and the matching ratio with the pleasant state, for example, if the matching ratio with the unpleasant state is higher than the matching ratio with the pleasant state, the server 300 determines the comfort state. Get discomfort state as the state.

According to such processing, erroneous recognition of the user's comfort state can be suppressed as much as possible.

The information processing system 100 described above can also improve the user's comfort level when the user views content.

<Other variations>
The embodiments described above are merely examples, and the present disclosure may be implemented with appropriate changes within the scope of the gist thereof. For example, the processes and means described in this disclosure can be implemented in any combination as long as no technical contradiction occurs.

Further, the processing described as being performed by one device may be shared and executed by a plurality of devices. For example, the learning unit 3034 may be formed in a processing device separate from the server 300. At this time, the other arithmetic processing device is configured to suitably cooperate with the server 300. Further, processes described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) that implements each function can be flexibly changed.

The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors included in the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the computer's system bus, or may be provided to the computer via a network. The non-transitory computer-readable storage medium may be any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), Includes read only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.

100... Information processing system 200... Network 300... Server 301... Communication unit 302... Storage unit 303... Control unit 400... User terminal

Claims (4)

Obtaining first captured image data that is captured image data captured by a camera and represents a facial expression image of the user when the user views the content;
The first photograph is applied to a pre-learning model constructed by learning a comfort state, which is a state of pleasure and displeasure felt by the user when the user views the content, using image data photographed in advance. Obtaining by inputting image data,
automatically adjusting the volume of the content based on the comfort state so as to improve the user's comfort level when the user views the content;
Equipped with a control unit that executes
The control unit includes:
Before the user views the content, initial content for initial settings is played back, and during playback, the volume of the initial content is automatically changed, and the timing at which the user feels uncomfortable due to the change in volume is obtained. And,
acquiring second photographed image data that is photographed image data photographed by the camera and represents an expression image of the user at the timing;
further execute,
causing the pre-learning model to perform learning using the second captured image data as training data;
Information processing device. The control unit includes:
Photographed image data generated by processing the second photographed image data, wherein the position of a person included in the second photographed image data has been arbitrarily changed, and/or the position of the person included in the second photographed image data has been changed. Automatically generate third photographed image data representing an image of the user in which the color of the included background has been arbitrarily changed and/or the clothing of the person included in the second photographed image data has been arbitrarily changed. do more of that,
adding the third captured image data to the teacher data and causing the pre-learning model to perform learning;
The information processing device according to claim 1 . The computer is
a first acquisition step of acquiring first photographed image data, which is photographed image data photographed by a camera and represents a facial expression image of the user when the user views the content;
The first photograph is applied to a pre-learning model constructed by learning a comfort state, which is a state of pleasure and displeasure felt by the user when the user views the content, using image data photographed in advance. a second acquisition step of acquiring image data by inputting it;
an automatic adjustment step of automatically adjusting the volume of the content based on the comfort state so as to improve the comfort of the user when the user views the content;
Run
The computer includes:
Before the user views the content, initial content for initial settings is played back, and during playback, the volume of the initial content is automatically changed, and the timing at which the user feels uncomfortable due to the change in volume is obtained. And,
acquiring second photographed image data that is photographed image data photographed by the camera and represents an expression image of the user at the timing;
further execute,
causing the pre-learning model to perform learning using the second captured image data as training data;
Information processing method. to the computer,
a first acquisition step of acquiring first photographed image data, which is photographed image data photographed by a camera and represents a facial expression image of the user when the user views the content;
The first photograph is applied to a pre-learning model constructed by learning a comfort state, which is a state of pleasure and displeasure felt by the user when the user views the content, using image data photographed in advance. a second acquisition step of acquiring image data by inputting it;
an automatic adjustment step of automatically adjusting the volume of the content based on the comfort state so as to improve the comfort of the user when the user views the content;
run the
to the computer;
Before the user views the content, initial content for initial settings is played back, and during playback, the volume of the initial content is automatically changed, and the timing at which the user feels uncomfortable due to the change in volume is obtained. And,
acquiring second photographed image data that is photographed image data photographed by the camera and represents an expression image of the user at the timing;
further execute
causing the pre-learning model to perform learning using the second captured image data as training data;
Information processing program.

Images