WO2022196769A1

WO2022196769A1 - Information processing method, program and information processing device

Info

Publication number: WO2022196769A1
Application number: PCT/JP2022/012347
Authority: WO
Inventors: 未知佐藤
Original assignee: 株式会社チカク
Priority date: 2021-03-19
Filing date: 2022-03-17
Publication date: 2022-09-22
Also published as: JP2022144667A

Abstract

The present invention appropriately ascertains the state at a call destination before calling. This information processing method involves execution, by a communication device, of: the acquisition of image data and/or audio data; the determination of the state of a user who is using the communication device on the basis of the image data and/or audio data; and the output of information pertaining to the state of the user to another communication device which is associated with said communication device before calling.

Description

Information processing method, program, and information processing device

The present disclosure relates to an information processing method, a program, and an information processing device.

Conventionally, there is a technology that informs the other party whether or not it is possible to talk before starting a voice call. For example, the caller operates the telephone, connects to the Internet using the other party's email address obtained from the notification destination, and obtains information on whether or not the call is possible, or by installing a presence sensor. , is known to know whether or not the other party is available for communication (for example, see Patent Document 1).

JP-A-2005-159748

However, in the conventional technology, it is possible to know whether or not the callee is available for communication, but it is not possible to know what state the callee is actually in. For example, you may not be at your desk even though the person you are calling has an internet connection. Further, if a presence sensor is provided to determine whether or not a person is at a seat, the cost increases due to the provision of the seat presence sensor. Therefore, there is a demand for a technique for appropriately grasping the state of the called party before making a call.

Therefore, it is an object of the present invention to provide an information processing method, a program, and an information processing apparatus capable of appropriately grasping the state of a callee before making a call.

An information processing method according to an aspect of the present disclosure includes: obtaining image data and/or sound data by a communication device; determining a state of a user using the communication device based on the image data and/or the sound data; and outputting the information about the state of the user to another communication device associated with the communication device before the call.

According to certain aspects of the present disclosure, it is possible to appropriately grasp the state of the call destination before the call.

1 is a diagram for explaining a system overview of the present disclosure; FIG. It is a figure showing an example of a schematic structure of information processing system 1 concerning a 1st embodiment. It is a figure which shows an example of the hardware constitutions of the server which concerns on 1st Embodiment. It is a figure which shows an example of the hardware constitutions of the user terminal which concerns on 1st Embodiment. 3 is a diagram illustrating an example of functions of each device of the information processing system according to the first embodiment; FIG. FIG. 10 is a diagram showing a screen example 1 including a state of a call destination user according to the first embodiment; FIG. 10 is a diagram showing a screen example 2 including a state of a call destination user according to the first embodiment; FIG. 11 is a diagram showing a screen example 3 including a state of a call destination user according to the first embodiment; FIG. 11 is a diagram showing a screen example 4 including a state of a call destination user according to the first embodiment; 4 is a flowchart showing an example of operation processing of the server according to the first embodiment; 8 is a flowchart showing an example of operation processing on the side that displays a state according to the first embodiment; FIG. 10 is a diagram illustrating an overview of a system according to a second embodiment of the present disclosure; FIG.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

<System Overview>
FIG. 1 is a diagram for explaining the system outline of the present disclosure. In the example shown in FIG. 1, it is assumed that user UB (eg, child) wants to initiate a voice call, video call, or video call (hereinafter collectively referred to as "call") with user UA (eg, parent). . A user UA uses an information processing device 20A (first communication device) such as a mobile terminal, and a user UB uses an information processing device 20B (second communication device) such as a mobile terminal. The server 10 is an information processing device capable of controlling IP (Internet Protocol) phone calls and videophone calls. Note that the outline shown in FIG. 1 includes an example in which a call is realized by P2P communication using IP telephony, WebRTC (Web Real-time Communication) technology, or the like.

(1) Transmission of Image Data/Sound Data First, the information processing device 20A turns on the camera and the microphone to acquire images and sounds around the information processing device 20A. The information processing device 20</b>A transmits media data related to the acquired image data and/or sound data (hereinafter also referred to as “image data/sound data”) to the server 10 . The information processing device 20A obtains image data/sound data by turning on at least one of the camera and the microphone at all times or periodically.

(2) Transmission of state information The server 10 receives media data related to image data/sound data from the information processing device 20A, and determines the state of the user UA who uses the information processing device 20A based on this media data. The server 10 performs image analysis and sound analysis, for example, to determine the current state of the user UA. The current state of the user UA is, for example, a state regarding whether or not the user UA is around the information processing apparatus 20A based on an image, a state regarding emotions based on facial expressions of the user UA, and a state regarding the user UA's surroundings. It includes at least one of a place state based on sound information, a state related to liveliness, a state indicating whether or not the user UA is available for communication, and the like.

The server 10 transmits information about the determined state of the user UA (hereinafter also referred to as "state information") to the information processing device 20B associated with the identification information (ID) of the information processing device 20A. The server 10 holds, for example, call handling information that associates identification information between devices that can communicate with each other.

(3) Sending a call request after confirming the state of the user When the information processing device 20B used by the user UB receives the state information of the user UA, it displays a screen informing the user UB of the state of the user UA on the display screen. . For example, the information processing device 20B displays a UI component M1 including a video, animation, text, or the like indicating the current state of the user UA. The UI component M1 is an icon, window, popup, or a predetermined area within the application screen. Also, current or past image data including the user UA acquired by the information processing device 20A may be displayed. At this time, in order to protect privacy, the images of those other than the user UA may be mosaiced or may have a different background.

It is assumed that the information processing device 20B accepts an operation to make a call from the user UB to the user UA. At this time, the information processing device 20B transmits a call request to the server 10 .

(4) Call Session Establishment When the server 10 receives a call request from the information processing device 20B, the server 10 transmits the call request to the information processing device 20A associated with the identification information of the information processing device 20B. When the information processing device 20A receives the call request and accepts an operation to answer the call from the user UA, the information processing device 20A transmits a call response to the call request to the server 10 . After that, when a session is established between the information processing device 20A and the information processing device 20B, the server 10 starts transmitting/receiving data for a call (hereinafter also referred to as "call data"). Further, if the server 10 does not receive a call response within a predetermined period of time, the server 10 may send a message to that effect to the information processing device 20B and discard the call request.

As a result, the user UB can appropriately grasp the current state of the user UA who is the callee before the call, just by looking at the display screen of the information processing device 20B. For example, if a state in which the user UA is in a room and is relaxed is displayed on the UI component M1, the user UB determines that the call may be made, and instructs the information processing device 20B to start the call. can give instructions.

In addition, the user UA is notified of his/her own state to the user UB without being particularly conscious of it, and the user UB can confirm the state of the user UA and grasp the timing of the call without performing any special operation. can. In addition, since image data and sound data are used to determine the state, it is possible to determine the appropriate state of the user UA. be able to start.

Note that similar processing may be performed from the user UB side to the user UA side. That is, the state of the user UB is displayed on the information processing device 20A of the user UA, and the user UA can grasp the current state of the user UB.

[First embodiment]
Next, a system configuration example for realizing the system described above will be described. FIG. 2 is a diagram showing an example of a schematic configuration of the information processing system 1 according to the first embodiment. As shown in FIG. 2, a server 10 and

user terminals

20A, 20B, 20C, 20D, . A system 1 is configured. Henceforth, the code|symbol 20 is used when it is not necessary to distinguish a user terminal individually. Further, the number of servers 10 connected to the network N may be plural.

The server 10 is, for example, an information processing device capable of transmitting and receiving IP-packetized voice data and image data, and is a device that performs call control such as calling (outgoing), receiving (incoming), answering, and disconnecting a telephone call. But also. Also, for call control, the H.264 standard is used. Signaling protocols such as H.323, MGCP (Media Gateway Control Protocol), SIP (Session Initiation Protocol) may be used.

The user terminal (communication device) 20 is an information processing device that can access a network. Media computer platforms (examples without limitation include set-top boxes, digital video recorders, etc.), handheld computing devices (examples without limitation include personal digital assistants (PDAs), email clients, etc.), wearable terminals (examples without limitation , glasses-type devices, watch-type devices, etc.), or other types of computers.

The network N can be configured to include multiple types of communication lines, communication networks, and various network devices. For example, the network N includes a base station wirelessly connected to the server 10, a wireless LAN access point (WiFi router, etc.), a mobile communication network connected to the base station, and a network connected from the access point via a router or modem. It may also include a public line such as a telephone line, a cable television line, or an optical communication line, the Internet connected to the user terminal 20, a mobile communication network, or a gateway device that connects the public line and the Internet. Note that the configuration of the network N is not limited to the above example.

In the system configuration shown in FIG. Displayed on the user terminal 20B.

The user terminal 20B receives the state information of the first user acquired via the server 10, and generates state notification information capable of notifying this state information. It is assumed that the second user, for example, sees the status notification information displayed on the image display screen, determines whether or not the first user can make a call, and performs an operation to start a call. At this time, the user terminal 20B transmits a call request to the user terminal 20A via the server 10. FIG.

When the user terminal 20A acquires the call request, the user terminal 20A transmits a call response to the server 10 when the second user performs an operation to respond. As a result, a call session is established, call data including call content such as voice and images is IP-packetized, and the IP-packetized call data is transmitted and received between the

user terminals

20A and 20B, A call is made.

<Hardware configuration>
Next, the hardware configuration of each device according to the information processing system 1 of the present disclosure will be described. FIG. 3 is a diagram showing an example of the hardware configuration of the server 10 according to the first embodiment. As shown in FIG. 3 , the server 10 has a control section 102 , a communication interface 104 and a storage section 106 , each section being connected via a bus line 112 .

The control unit 102 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like. Further, the control unit 102 executes an application or the like stored in the storage unit 106, in addition to the function of a general web server, the function of performing call control such as call origination, call reception, answering, and disconnection. may be configured to achieve Further, the control unit 102 may be capable of executing a voice recognition function on acquired sound data and an object recognition function on acquired image data.

The communication interface 104 controls communication with the user terminal 20 via the network N.

The storage unit 106 stores applications and data (not shown) for realizing a server function that performs call control, including, for example, a large-capacity HDD (Hard Disk Drive) or SSD (Solid State Drive). The storage unit 106 also stores a control program 108 . Storage unit 106 also has information storage unit 110 .

The control program 108 is a program that determines the user's state based on image data/sound data, performs call control, and executes applications. The control program 108 is a program for transmitting a request from the calling side to the called side, receiving a response to the request from the called side, and returning the response to the calling side.

The information storage unit 110 stores information of each user terminal that uses the function of determining the user's status. For example, the information storage unit 110 may function as a location server that stores a URI (Uniform Resource Identifier) corresponding to the IP address of the user terminal 20 . The information storage unit 110 also stores image data/sound data included in media data transmitted from the user terminal 20 in association with identification information of the user terminal 20 or the like.

Next, the hardware configuration of the user terminal 20 will be explained. FIG. 4 is a diagram showing an example of the hardware configuration of the user terminal 20 according to the first embodiment. As shown in FIG. 4, the user terminal 20 includes a control unit 202, a communication interface 206, a storage unit 208, a display unit 214, an input unit 216, a microphone 220, a speaker 222, and an imaging device 224. Each part is connected via a bus line 218 .

The control unit 202 includes a CPU, ROM, RAM 204, and the like. The control unit 202 is configured to implement, for example, an IP call function in addition to functions as a general information processing apparatus by executing applications and the like stored in the storage unit 208 . Further, the control unit 202 may be capable of executing a voice recognition function on input sound data and an object recognition function on input image data.

In addition, the RAM 204 temporarily holds various information and is used as a work area when the CPU executes various processes.

The communication interface 206 controls communication with the server 10 via the network N.

The storage unit 208 includes, for example, an HDD or SSD, and stores application programs 210 in addition to storing applications and data (not shown) for realizing functions as a general information processing device. The storage unit 208 also has an information storage unit 212 .

The application program 210 is a program for executing the media data transmission function related to the image data/sound data described above and the user status display function. The application program 210 is a program for transmitting a request to the called party via the server 10, receiving a response to this request from the called party via the server 10, and starting a call.

The information storage unit 212 stores information (call handling information) that associates the identification information of devices that can communicate with each other and the identification information of users. The call handling information may further hold the user status information transmitted from the server 10 in association with the identification information of the transmitted device.

The display unit 214 is a display such as a touch panel or liquid crystal monitor, and displays information to the user. For example, the display unit 214 displays an application execution screen, and more specifically, displays a screen showing the status of the call destination, a screen during IP call, and the like.

The input unit 216 receives input from the user and receives instructions from the user. Note that the display unit 214 and the input unit 216 may be configured as a touch panel.

The microphone 220 is a device that collects sound such as voice, and may have a noise canceling function. The speaker 222 is a device that converts sound data into physical vibrations and outputs sounds such as music and voice. The imaging device 224 is, for example, a camera, and images the surroundings of the information processing device 20 .

<Functional configuration>
Next, functions of each device of the information processing system 1 according to the first embodiment will be described with reference to FIG. In the example shown in FIG. 5, the user terminal 20A is the side that transmits media data and the side that receives call requests. The user terminal 20B is the side that displays the state information of the user using the user terminal 20A, and is the side that transmits the call request. Note that each user terminal 20 may be a calling side and a called side, and may have respective functions.

As described above, the server 10 has the function of controlling calls using the Internet. For example, when using SIP communication, the server 10 performs call control by exchanging a request (SIP method) based on HTTP (Hyper Text Transfer Protocol) and a response (response code).

In the call function of the present disclosure, image data/sound data is transmitted and user status information is displayed on the screen before the above-described request and response are exchanged. With regard to the user terminal 20A shown in FIG. 5, mainly the function of transmitting image data/sound data will be explained, and with respect to the user terminal 20B, mainly the function of displaying user status information will be explained.

In addition, since the user terminal 20A and the user terminal 20B have the same functions, the symbols A and B are omitted when the respective functions are not distinguished. For example, it is possible to express that the user terminal 20 has the image acquisition unit 302 . Note that the user terminal 20 may have either the function of transmitting media data or the function of displaying user status information.

The user terminal 20A has an image acquisition section 302A, a sound acquisition section 304A, a transmission section 306A, a reception section 312A, a data acquisition section 314A, a generation section 316A, an output control section 318A, and a call processing section 320A. The function of each unit in the user terminal 20A is realized by executing the application program 210 by the control unit 202 shown in FIG.

Note that when the user terminal 20A is executing an application related to the call function of the present disclosure, the camera function and microphone function are set to ON, and image data and sound data around the user terminal 20A are input.

(Data transmission function of user terminal 20A)
302 A of image acquisition parts acquire the image data imaged by the imaging device 224. FIG. For example, the image data is image data including the first user or image data including the state of the room. The acquired image data is stored in a buffer, for example, and deleted after a predetermined period of time has elapsed.

The sound acquisition unit 304A acquires sound data collected by the microphone 220. For example, the sound data is sound data including the voice of the first user, sound data including life sounds, and the like. Acquired sound data is stored in a buffer, for example, and deleted after a predetermined period of time has elapsed.

The transmission unit 306A transmits media data including image data acquired by the image acquisition unit 302A and/or sound data acquired by the sound acquisition unit 304A to the server 10. A destination is set in the server 10 in advance. In addition, when transmitting data to the server 10, the transmission unit 306A may also transmit the identification information of the user terminal 20A or the identification information of the first user.

Through the above processing, the first user can include image data and sound data in media data and transmit the media data to the server 10 without performing any special operation. For example, even an elderly first user who is unfamiliar with computers or the like does not require any special operation, so the transmission process can be easily executed.

(Session Establishment Function of User Terminal 20A)
As will be described later, when a call request is transmitted from the user terminal 20B, the receiving unit 312A receives the call request via the server 10. FIG. When the user terminal 20A receives the call request, the user terminal 20A notifies the first user that there is a call request from the second user of the user terminal 20B. Here, when the first user executes an operation of responding to the call request, the user terminal 20A transmits the call response to the user terminal 20B via the server 10. FIG.

When a session is established between the user terminal 20A and the user terminal 20B via the server 10 by transmitting a call response from the receiving unit 312A, the call processing unit 320A converts call data into call packets. send and receive

When the receiving unit 312A directly or indirectly receives a call termination request from the user terminal 20B, the call processing unit 320A directly or indirectly transmits a response to the termination request to the user terminal 20B. This ends the session and ends the call. Note that the call termination request may be transmitted from the user terminal 20A.

(Call control function of server 10)
The server 10 has a reception unit 402 , a call control unit 404 , an acquisition unit 406 , a determination unit 408 and a transmission unit 410 . The function of each unit in the server 10 is realized by executing the control program 108 by the control unit 102 shown in FIG.

The receiving unit 402 of the server 10 receives the image data/sound data transmitted from the transmitting unit 306A of the user terminal 20A. For example, the image data/sound data included in the received media data are stored in the RAM of the control unit 102, for example.

The acquisition unit 406 acquires image data/sound data stored in RAM or the like. Note that the acquisition unit 406 may acquire the image data/sound data transmitted by the user terminal 20A from the reception unit 402 .

The determination unit 408 determines the state of the first user of the user terminal 20A based on the acquired image data/sound data. For example, the determination unit 408 analyzes the image data/sound data and determines the state of the first user, such as whether the first user is around the user terminal 20A. The state of the user may be, for example, one state selected from a plurality of states according to the analysis result of the image data/sound data.

Also, the determination unit 408 determines the state each time image data/sound data is acquired, but the updated state information may be transmitted from the transmission unit 410 only when the state changes. . Thereby, the load of the transmission processing of the server 10 can be reduced.

Also, the determination unit 408 may determine the state of the first user using a learning model that has learned the state of the first user using image data/sound data as learning data.

The transmission unit 410 transmits information (state information) about the state of the first user to the user terminal 20B that is associated with the identification information of the user terminal 20A and that can communicate with the user terminal 20A. For example, the transmission unit 410 transmits state information including whether or not the first user is available for communication to the user terminal 20B. Alternatively, the transmission unit 410 may transmit the state information to the user terminal 20B only when the state is updated.

By the above processing, the state of the first user is determined using image data and sound data, and the state information of the first user who is the callee is notified to the user who is the caller. , the status of the called party can be easily grasped.

Further, the determining unit 408 may perform object recognition on the acquired image data and determine the state of the user based on the result of the object recognition. For example, the determination unit 408 may detect whether or not a person is included by object recognition for image data. Object recognition may be performed using a learning model that has undergone machine learning for each object in advance. Object recognition may be performed using a learning model that has undergone supervised learning using images of the first user as learning data.

In this case, the determination unit 408 determines whether or not there is a user (for example, the first user) around the user terminal 20A based on the object recognition result. The determination unit 408 determines that the call is possible when the user is present, and determines that the call is impossible when the user is absent. If the determination unit 408 detects that the user is sitting on a sofa or the like and has not moved for a predetermined time as a result of object recognition, the determination unit 408 determines that the user is in a relaxed state, and determines that the user is moving. If detected, it may be determined that the user is busy. The determination unit 408 may identify the state from the recognition result by associating each recognition result with each state.

Through the above processing, the determining unit 408 can determine not only whether or not the first user can make a call, but also a more detailed state. For example, the determination unit 408 can determine states such as whether the first user is absent or moving.

Further, the determination unit 408 may perform facial expression recognition of the first user's face included in the image data, and determine the state of the first user based on the result of the facial expression recognition. For example, when detecting that the first user is included in the image data, the determination unit 408 recognizes the facial expression of the first user. A known technique may be used for facial expression recognition. The determination unit 408 uses AI (artificial intelligence) to estimate the first user's joy, surprise, anger, sadness, serious face, and the like.

Through the above processing, the determination unit 408 can not only determine whether or not the first user is able to make a call, but also estimate a more detailed emotional state. For example, the determination unit 408 can estimate what kind of emotional state the first user is currently in, depending on what the facial expression of the first user is.

Also, the determining unit 408 may perform frequency analysis on the sound data and determine the state of the first user based on the result of this frequency analysis. For example, as a result of the frequency analysis, the determination unit 408 determines whether the first user's voice is included, whether the user is on a train, whether the user is outside, whether the user is listening to music, and the like. judge.

Through the above processing, the determination unit 408 can estimate the detailed sound-related state in addition to simply determining whether or not the first user can make a call. For example, the determination unit 408 can estimate whether the surroundings of the first user are lively or quiet.

The call control unit 404 identifies the user terminal 20A based on the call request received by the receiving unit 402 from the user terminal 20B. For example, the call control unit 404 identifies call destination information (for example, URI) from the call request, and acquires the IP address of the user terminal 20A using the identified call destination information. The call control unit 404 may query a location server (not shown) to identify the IP address from the destination information.

The transmission unit 410 transmits a call request to the user terminal 20A identified by the call control unit 404. For example, the call control unit 404 transmits a call request to the user terminal 20A having the identified IP address.

The receiving unit 402 also receives a call response from the user terminal 20A in response to the transmitted call request. After that, the call control unit 404 establishes a call session and controls mutual transmission and reception of call data.

(Status reporting function of user terminal 20B)
The user terminal 20B has an image acquisition section 302B, a sound acquisition section 304B, a transmission section 306B, a reception section 312B, a data acquisition section 314B, a generation section 316B, an output control section 318B, and a call processing section 320B. The function of each unit in the user terminal 20B is realized by executing the application program 210 by the control unit 202 shown in FIG. Also, the functions of the user terminal 20B are the same as those of the user terminal 20A. The status notification function will be mainly described below.

The receiving unit 312B receives the state information of the first user of the user terminal 20A from the server 10. For example, the received state information of the first user is stored in the RAM of the control unit 202 or the like. At this time, the receiving unit 312B may also receive the identification information of the user terminal 20A that transmitted the state information of the first user or the identification information of the first user, and store it in the RAM.

The data acquisition unit 314B acquires the state information of the first user stored in the RAM, for example. Also, the data acquisition unit 314B may acquire the state information of the first user from the reception unit 312B.

The generation unit 316B generates state notification information that can notify the state of the first user based on the state information of the first user. For example, the generation unit 316B generates a UI component including character data, image data, etc. representing the state of the first user indicated by the state information.

The output control unit 318B controls the output of the state notification information by the generation unit 316B. For example, the output control unit 318B controls display on the display screen of the display unit 214 if the state notification information is state display information, and controls so that sound is output from the speaker 222 if the state notification information is state sound information. For example, the output control unit 318B controls the state notification information to be displayed as a popup.

By the above process, the second user can grasp the current situation of the first user by looking at the screen of the user terminal 20B. As a result, if the first user is available for communication, the second user should call the first user.

Also, the output control unit 318B may control the display of the status notification information within a predetermined area of the display screen. For example, the output control unit 318B controls to display the state notification information in a window set in advance within the display screen. Further, the output control unit 318B may control to display the status notification information in a predetermined area set in the execution screen of the application that realizes the disclosed call function. Also, the predetermined area may be an area in which a widget capable of executing the application described above is displayed. For example, status notification information is displayed on this widget, and this status notification information is appropriately updated on the widget.

By the above processing, the second user can grasp the current situation of the first user by looking at the predetermined area in the screen.

In addition, when the state notification information is updated, the output control unit 318B may control display of the updated state notification information within a predetermined area. For example, when the state notification information is updated from the first state to indicate the second state, the output control unit 318B controls display of the state notification information indicating the second state within a predetermined area.

With the above processing, the display screen is updated each time the status notification information is updated, not each time the status notification information is generated. 20B processing load can be reduced.

In addition, the generating unit 316B may include generating an image of the icon as the status notification information when an icon of an application that executes a call via the server 10 exists on the display screen. For example, using a function for dynamically changing the display of icons such as calendar icons, the generation unit 316B sets icon images or characters corresponding to each state indicated by the state notification information, and the server 10 Identify the image or text of the icon according to the status information obtained from. In this case, the output control unit 318B performs control so that the image or characters specified by the generation unit 316B are displayed on the icon.

By the above processing, the second user can grasp the state of the first user by looking at the icon of the application that executes the functions of the present disclosure. For example, by looking at icons displayed on the home screen, it is possible to determine whether or not the first user is ready to talk.

The generation unit 316B may also generate state notification information including a moving image or animation indicating the state of the first user. For example, the generation unit 316B may use a moving image, an animation, or the like representing the state of the first user indicated by the state information as the state notification information. As a specific example, the moving image is a past moving image of the first user captured in a predetermined room. In this case, if the state information indicates that the first user is relaxed (such as sitting on a sofa), the generating unit 316B generates a past moving image of the first user sitting on the sofa in the room. If it is selected and indicates that the first user is absent, a past moving image in which the first user is not in the room is selected.

In addition, when acquiring image data captured by the user terminal 20A as state information, the generation unit 316B applies an effect (mosaic, another background, etc.) to images of users other than the first user to protect privacy. can be

In addition, the generation unit 316B may express the state of the first user as an animation with the movement of a predetermined object. For example, when the first user is moving or exercising, the generating unit 316B moves the object quickly, when the first user is absent, hides the object, or makes the first user relax. If you are in a state where you are moving the object slowly.

With the above processing, it is possible to protect the privacy when displaying the state of the first user, and the first user can safely permit the execution of the functions of the present disclosure.

In addition, the generation unit 316B may include specifying a UI component for starting call processing for the user terminal 20A when the information regarding the state of the first user indicates that a call is possible. In this case, the output control unit 318B may control the display of the UI component on the display screen together with the output of the status notification information.

For example, when the status information indicates that a call is possible, the generation unit 316B identifies or selects a button to call the first user. Note that the state information may include a plurality of states, and whether or not a call is possible may be set in advance for each state. In addition, the output control unit 318B may perform control so that the UI component is displayed in association with the status notification information. For example, control is performed so that the UI component is displayed at a position adjacent to the display information.

With the above processing, if the first user is in a callable state, the second user can confirm the first user's state by operating the UI parts, and then smoothly make a call.

When the second user operates the UI component for making a call to the first user, the transmission unit 306B transmits a call request to the user terminal 20A via the server 10. If an OK call response is returned from the user terminal 20A in response to this call request, the call is started.

As described above, by executing the functions of each device, the user who is the calling party can grasp the current state of the user who is the calling party before starting the call. In addition, when the current state includes the emotions and movements of the user to whom the call is being made, it is possible to make a call while grasping the state, so that the start of the call can proceed smoothly.

<Screen example>
Next, with reference to FIG. 6, an example screen including the state of the user who is the callee according to the first embodiment will be described. FIG. 6A is a diagram showing a screen example 1 including a state of a call destination user according to the first embodiment. In the example shown in FIG. 6A, the status notification information is displayed on the screen as a popup A1. Also, since the state information indicates that the user (Mr. A) is sitting on the sofa, it is determined that the call is possible. Also, when it is determined that the call is possible, the UI component A2 of the "call" button is displayed on the screen.

FIG. 6B is a diagram showing Screen Example 2 including the state of the called user according to the first embodiment. In the example shown in FIG. 6B, the status notification information is displayed on the icon B1 of the application that executes the call function of the present disclosure. In this case, after the user clicks this icon B1 and transitions to the execution screen of this application, a button to make a call may be displayed.

FIG. 6C is a diagram showing a screen example 3 including the state of the called user according to the first embodiment. In the example shown in FIG. 6C, the status notification information is displayed in a predetermined area C1 of the execution screen of the application that executes the call function of the present disclosure. Also, since the state information indicates that the user is sitting on the sofa, it is determined that the call is possible. Further, when it is determined that the call is possible, the UI component A2 of the "call" button is displayed in the execution screen.

FIG. 6D is a diagram showing a screen example 4 including the state of the called user according to the first embodiment. In the example shown in FIG. 6D, there are a plurality of predetermined areas on the execution screen of the application that executes the call function of the present disclosure, and different user's current states are displayed in the respective predetermined areas C1 and D1. For example, Mr. B is absent, and in the predetermined area D1 where the status notification information is displayed, a black screen or the like is displayed to give the impression that the room is dark.

In the example shown in FIG. 6D, the server 10 sets a plurality of users (Mr. A and Mr. B in the example shown in FIG. 6D) in the call handling information as call destination candidates for a predetermined user. The server 10 acquires the image data/sound data from the user terminals 20 of Mr. A and Mr. B, and determines the state of each. Upon receiving the state information of each user from the server 10, the user terminal 20 can display the state notification information of a plurality of users on the screen as shown in FIG. 6D.

As a result, the user who is the caller can easily grasp the current status of multiple callee candidates.

<Action>
Next, each operation of the information processing system 1 will be described. FIG. 7 is a flowchart showing an example of operation processing of the server 10 according to the first embodiment. Hereinafter, an operation will be described for an example in which the user terminal 20A of the first user, who is the callee, transmits image data and the like, and the user terminal 20B of the second user, who is the caller, displays the state of the first user.

At step S102, the receiving unit 402 of the server 10 receives the image data/sound data included in the media data transmitted from the user terminal 20A of the first user. Next, the acquisition unit 406 of the server 10 acquires the received image data/sound data.

At step S104, the determination unit 408 of the server 10 determines the current state of the first user based on the acquired image data/sound data.

In step S106, the determination unit 408 of the server 10 determines whether the determined state has changed from the current state. If the state changes (step S106-YES), the process proceeds to step S108, and if the state does not change (step S106-NO), the process returns to step S102.

In step S106, the determination unit 408 of the server 10 is associated with the identification information of the user terminal 20A that transmitted the media data including the image data/sound data or the identification information of the first user, and can communicate with the user terminal 20A. The user terminal 20B is specified as the notification destination. The determination unit 408 can identify the notification destination by referring to the call handling information described above, for example.

In step S108, the transmission unit 410 of the server 10 transmits information (state information) regarding the determined state of the first user to the user terminal 20B specified as the notification destination.

FIG. 8 is a flowchart showing an example of operation processing on the side that displays the state according to the first embodiment. In the example shown in FIG. 8, the processing performed by the user terminal 20B will be mainly described. The example shown in FIG. 8 is an example in which the state notification information is state display information, and the output control unit 318B controls the display of the state display information on the display screen.

In step S202, the receiving unit 312B of the user terminal 20B receives the state information of the first user from the server 10. Also, the data acquisition unit 314B of the user terminal 20B acquires the received state information.

In step S204, the generation unit 316B of the user terminal 20B generates state display information based on the acquired state information of the first user. For example, the generation unit 316B generates a UI component or the like including an image, characters, or the like indicating the state of the first user.

At step S206, the output control unit 318B of the user terminal 20B controls display of the generated status display information on the display screen.

In step S208, the user terminal 20B determines whether or not the second user has operated a UI component for starting a call. If there is a call start operation (step S208-YES), the transmission unit 306B transmits a call request to the user terminal 20A via the server 10, the process proceeds to step S210, and if there is no call start operation (step S2108-NO), the process returns to step S202.

At step S210, the receiving unit 312B of the user terminal 20B determines whether or not a call response has been received from the user terminal 20A via the server. If a call response is received (step S210-YES), the process proceeds to step S212, and if a call response is not received (step S210-NO), the call process ends because the first user did not answer. , the process returns to step S202.

At step S212, the user terminal 20B starts a call session, transmits/receives call data to/from the user terminal 20A, and starts a call.

As described above, according to the information processing system in the first embodiment, the user who is the caller can grasp the current state of the user who is the callee before starting the call.

Further, the information processing described in the first embodiment may be implemented as a program to be executed by a computer. By installing this program from a server or the like and causing a computer to execute the program, the information processing described above can be realized.

It is also possible to record this program on a recording medium and cause a computer to read the recording medium on which this program is recorded, thereby realizing the information processing described above. The recording medium can store the program in a "non-temporary tangible medium".

Recording media include recording media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks; and recording media that record information electrically, such as ROMs and flash memories. Various types of recording media can be used, such as a semiconductor memory that

[Second embodiment]
FIG. 9 is a diagram explaining an overview of the system 2 according to the second embodiment of the present disclosure. A system 2 shown in FIG. 2 is a system in which the system described in the first embodiment is applied to a system for managing images, and includes a management device 10A, a server 10B, a user terminal 20, and an image output device 30. , and a camera 60 . The image output device 30 is also connected to the display device 40 , and the display device 40 is controlled by the remote control device 50 . A camera 60 is attached to the display device 40 , and an image acquired by the camera 60 is input to the image output device 30 . For example, the camera 60 is a web camera and may have a built-in microphone. The server 10B has the functions of the server 10 described in the first embodiment.

The management device 10A or the server 10B and the user terminal 20 are mutually connected via a network N1 such as a wireless LAN, a fourth or fifth generation mobile communication system (4G or 5G), or LTE (Long Term Evolution) as an example of communication technology. Communication is possible. Also, the management device 10A and the image output device 30 can communicate with each other via a wireless network N2, such as a third generation mobile communication system (3G), which has a lower communication charge than the network N1 but is slower than the network N1. be. Although the network N1 and the wireless network N2 are described separately for the sake of explanation, these networks can be connected to each other via the Internet. Also, the network configuration is not limited to the above example.

The management device 10A first acquires information such as the name and address of the viewer or poster of the image from the poster of the image through the website on the Internet. Information such as the name and address of the viewer or contributor is not essential information. At this time, the management device 10A generates an identifier (referred to as a device ID) of the image output device 30 used by the viewer. Next, the device ID is notified to the contributor of the image, for example, by e-mail.

The management device 10A sets the generated device ID in the image output device 30. Thereafter, the image output device 30 is shipped to the address of the viewer of the image by the administrator of the management device 10A. The image output device 30 incorporates, for example, a 3G communication module, and is configured to immediately start communication with the management device 10A using the set device ID when the power is turned on.

On the other hand, the person who posted the image downloads an application for image sharing that operates on the user terminal 20, such as a smartphone or tablet. This application uses the notified device ID to access the management device 10A. The management device 10A can associate the user terminal 20 (contributor) and the image output device 30 (viewer) using the device ID notified from the application as a key. This information corresponds to the call handling information described above.

After that, the contributor can use the application to shoot various subjects. The application automatically transmits image data obtained by shooting to the management device 10A via the network N1. The contributor is not required to perform any special operation in order to transmit the image data to the management device 10A.

The management device 10A accumulates the image data transmitted from the application of the user terminal 20 and sequentially distributes it to the image output device 30. The image output device 30 displays the image data on the display device 40 according to an instruction from the viewer. Also, the image output device 30 may have a microphone and a speaker, and IP calls are possible.

Here, the display device 40 is, for example, a television widely used in ordinary homes, and the remote control device 50 is a remote controller. The image output device 30 is connected to the display device 40 via, for example, HDMI (High-Definition Multimedia Interface) (registered trademark), and can acquire control signals emitted from the remote control device 50 via HDMI. .

The image output device 30 can acquire the control signal issued from the remote control device 50 and grasp the content of the operation input by the viewer. In other words, the viewer can view the distributed image data using the familiar TV remote control.

The camera 60 is, for example, a web camera, and may have a built-in microphone. Image (still image or moving image) data/sound data captured by the camera 60 is input to the image output device 30 . The image output device 30 transmits media data related to the input image data/sound data to the server 10 .

The function for starting a call described in the first embodiment is implemented in the application for image sharing described above. In this case, the user terminal 20A shown in FIG. 1 etc. corresponds to the image output device 30. FIG. For example, the image output device 30 has a transmission section 306 , a reception section 312 , a data acquisition section 314 , a generation section 316 , an output control section 318 and a call processing section 320 of the user terminal 20 .

Specifically, media data including image data/sound data acquired by the camera 60 is transmitted to the server 10B via the image output device 30. The server 10B determines the state of the viewer of the image based on the image data/sound data included in the acquired media data. For example, the server 10B analyzes the image data/sound data to determine whether the viewer is moving, sitting on a chair, not in the room, busy, or quiet in the room where the display device 40 is located. determine the state.

The server 10B refers to the information held in the management device 10A, identifies the user terminal 20 associated with the image output device 30, and transmits the status information of the viewer to this user terminal 20. As described above, the user terminal 20 generates state notification information (for example, state display information) based on the state information and controls to display it on the display screen (for example, FIG. 6).

When the user terminal 20 detects that the contributor has operated the call button, the user terminal 20 transmits a call request to the image output device 30 via the server 10B. When receiving the call request, the image output device 30 displays on the display device 40 whether or not to respond. For example, a first button on remote control 50 is assigned to answer and a second button is assigned to decline.

When the viewer operates the first button, the image output device 30 detects the operation of the first button and transmits a call response to the user terminal 20 via the server 10B. After this, the call is started.

As described above, according to the second embodiment, a system for sharing images between viewers and contributors can have the call function disclosed in the first embodiment. Also, the management device 10A and the server 10B may be the same server.

[Third embodiment]
Next, a third embodiment of the present disclosure will be described. In the third embodiment, the user terminal 20 is provided with the function of the determination unit 408 described in the first embodiment. For example, the determination unit of the user terminal 20 determines the state of the user using the own terminal based on the image data acquired by the image acquisition unit 302 and/or the sound data acquired by the sound acquisition unit 304. .

The method for determining the user's status is as described above. For example, the determination unit of the user terminal 20 performs object recognition on image data and/or frequency analysis on sound data, and determines the state of the user using the respective results.

The transmission unit 306 of the user terminal 20 transmits information (state information) regarding the state of the user determined by the determination unit to other user terminals before the call. For example, the transmitting unit 306 may first transmit the state information to the server 10 in order to transmit the state information to other user terminals. After acquiring the status information, the server 10 identifies a notification destination (another user terminal) associated with the user terminal 20 that transmitted the status information, and transmits the status information to the identified notification destination. Upon receiving the status information, the user terminal 20 of the notification destination executes the same processing as in the first embodiment.

As described above, according to the third embodiment, the load of the server 10 can be distributed. In addition, since the user terminal 20 processes image data and sound data by itself and does not transmit these image data and sound data to the outside, it is possible to protect privacy.

[Fourth embodiment]
Next, a fourth embodiment of the present disclosure will be described. In the fourth embodiment, the user terminal 20 is provided with some functions of the determination unit 408 described in the first embodiment. For example, the preprocessing unit of the user terminal 20 performs object recognition on image data acquired by the image acquisition unit 302 and/or performs frequency analysis on sound data acquired by the sound acquisition unit 304. , object detection and frequency feature extraction.

The transmission unit 306 of the user terminal 20 transmits to the server 10 media data including processing result information including object recognition results and/or feature amounts (frequency analysis results) processed by the preprocessing unit. When the server 10 acquires the media data, the determining unit 408 determines the state of the user based on the processing result information included in the acquired media data. For example, the determination unit 408 can determine the user's state from the result information by holding a table or the like in which the user's state is associated with each obtainable result information.

After determining the state of the user, the server 10 identifies the notification destination associated with the user terminal 20 that transmitted the media data, and transmits information (state information) regarding the user state to the identified notification destination. . Upon receiving the status information, the user terminal 20 of the notification destination executes the same processing as in the first embodiment.

As described above, according to the fourth embodiment, it is possible to reduce the amount of communication compared to the first embodiment, and it is possible to share the computational resources of the determination unit that is subject to the processing load.

As described above, each embodiment described above is an example for explaining the present invention, and is not intended to limit the present invention only to the embodiment, and the present invention can be variously modified without departing from the gist thereof. is.

[Modification]
For example, as Modification 1, the output control unit 318 of the user terminal 20 may control the operation of the device according to the state information, in addition to controlling the output of the state notification information generated by the generation unit 316 . As a specific example, the output control unit 318 controls the brightness of the lighting in the room, the volume of the device that is playing music (own terminal or music player), the device that is watching TV or video (own terminal or TV), depending on the state information. etc.) may also function as an operation control unit that controls devices connected via a network, such as pausing or volume.

For example, the output control unit 318 may brighten the lighting in the room, reduce the volume of the music, or pause the video being played, if the call destination is in a state where the call can be made. As a result, it is possible to create an environment in which the caller can easily make a call. In addition, the output control unit 318 may store control commands corresponding to each state of the user in association with each other, and execute the control command specified by the state information.

Also, as a modification 2, in each of the above-described embodiments, the generation unit 316 is provided in the user terminal 20, but may be provided in the server 10. For example, the generation unit of the server 10 generates state notification information according to the state of the user determined by the determination unit 408 .

In this case, the transmitting unit 410 of the server 10 transmits state notification information to the user terminal 20 as information (state information) regarding the state of the user. That is, in the case of Modification 2, the information about the state of the user transmitted from the server 10 includes the state notification information. The output control unit 318 of the user terminal 20 controls to display the status notification information on the screen when the status notification information is acquired. As a result, the server 10 can easily add or modify the status notification information.

Further, as a modification 3, the determination unit 408 uses information acquired from a predetermined sensor (for example, an illuminance sensor, a photo sensor such as a motion sensor, an ultrasonic sensor, etc.) instead of or in addition to the media data. may be used to determine the state of the user. For example, the determination unit 408 can determine that there is a user to be called when the detection signal of a predetermined sensor indicates ON. With this, although it may be costly to introduce a predetermined sensor, it is possible to determine the state of the called party user before the call in the same manner as in the above-described embodiments. In addition, by using media data and a signal acquired by a predetermined sensor, it becomes possible to more appropriately determine the state of the user of the called party before the call.

Reference Signs List 1 information processing system 2 system 10 server 10A management device 10B server 20 user terminal 30 image output device 40 display device 60 camera 102 control unit 106 Memory unit 202 Control unit 204 RAM 208 Storage unit 220 Microphone 222 Speaker 224 Imaging device 302 Image acquisition unit 304 Sound acquisition unit 306 Transmission unit 312 Reception Unit 314 Data acquisition unit 316 Generation unit 318 Output control unit 320 Call processing unit 402 Reception unit 404 Call control unit 406 Acquisition unit 408 Determination unit 410 Transmission unit

Claims

the communication device
obtaining image data and/or sound data;
Determining the state of the user using the communication device based on the image data and/or sound data;
outputting information about the user's state toward another communication device associated with the communication device before a call;
Information processing method that performs
The determining
2. The information processing method according to claim 1, comprising performing object recognition on said image data and determining said user's state based on at least a result of said object recognition.
The determining
3. The information processing method according to claim 1, comprising recognizing facial expressions of said user included in said image data, and determining a state of said user based on at least a result of said facial expression recognition.
The determining
4. The information processing method according to any one of claims 1 to 3, comprising performing frequency analysis on said sound data and determining said user's state based on at least a result of said frequency analysis.
to the communication device,
obtaining image data and/or sound data;
Determining the state of the user using the communication device based on the image data and/or sound data;
outputting information about the user's state toward another communication device associated with the communication device before a call;
program to run.
an acquisition unit that acquires image data and/or sound data;
a determination unit that determines a state of a user using the communication device based on the image data and/or the sound data;
a transmission unit that outputs information about the user's state toward another communication device associated with the communication device before a call;
A communication device comprising:
the communication device
Acquiring information about the state of the user using the other communication device, determined based on media data relating to image data and/or sound data acquired by the other communication device;
generating state notification information capable of notifying the state of the user based on the information about the state of the user;
controlling the output of the state notification information;
Information processing method that performs
The output control includes
controlling display of the state notification information within a predetermined area of a display screen;
8. The information processing method according to claim 7, further comprising, when the state notification information is updated, displaying the updated state notification information within the predetermined area.
The generating includes:
9. The information processing method according to claim 8, comprising generating an image of said icon as said status notification information when an icon of an application for executing a call via said information processing device is present on said display screen.
The generating includes:
10. The information processing method according to any one of claims 7 to 9, comprising generating state notification information including a moving image or animation indicating the user's state.
The generating includes:
identifying a UI component that initiates call processing for the other communication device when the information about the user's status indicates that a call is possible;
The output control includes
10. The information processing method according to any one of claims 7 to 9, comprising controlling display of said UI component on a display screen together with outputting said status notification information.
to the communication device,
Acquiring information about the state of the user using the other communication device, determined based on media data relating to image data and/or sound data acquired by the other communication device;
generating state notification information capable of notifying the state of the user based on the information about the state of the user;
controlling the output of the state notification information;
program to run.
an acquisition unit that acquires information about the state of a user using the other communication device, which is determined based on media data related to image data and/or sound data acquired by the other communication device;
a generating unit that generates state notification information that can notify the state of the user based on the information about the state of the user;
an output control unit that controls output of the state notification information;
A communication device comprising:
The information processing device
obtaining media data related to image data and/or sound data from the first communication device;
determining a state of a user of the first communication device based on the media data;
An information processing method, comprising transmitting information about the state of the user to a second communication device that is associated with the identification information of the first communication device and that can communicate with the first communication device.
The determining
Determining the state of the user based on at least the processing result information contained in the media data, which includes object recognition results of the image data and/or frequency analysis results of the sound data. 15. The information processing method of claim 14, comprising:
information processing equipment,
Acquiring media data related to image data and/or sound data transmitted from the first communication device;
determining a state of a user of the first communication device based on the media data;
Sending information about the state of the user to a second communication device that is associated with the identification information of the first communication device and that can communicate with the first communication device.
an acquisition unit that acquires media data related to image data and/or sound data from the first communication device;
a determination unit that determines a state of the user of the first communication device based on the media data;
and a transmitting unit that transmits information about the state of the user to a second communication device that is associated with the identification information of the first communication device and that can communicate with the first communication device.