JP2020010107A - Server, control method of server, control program of server, and remote support system - Google Patents

Abstract

To provide a server which allows an operator to confirm the screen of a support application provided to a user by the operator, when support by the operator is provided to the user together with the support application.SOLUTION: A server 2 is configured to include a user information storage section for storing user information acquired by performing voice recognition processing of voice received from a user terminal 4, an automatic voice response section for generating response voice on the basis of the user information and transmitting to the user terminal, an application execution request section for requesting execution of an application on the basis of the user information, a connection request reception part for receiving an operator request transmitted from the user terminal in response to a specific operation of the user to the application, a connection identifier transmission section for transmitting a connection identifier, generated in response to reception of the operator request, to the user terminal and the operator terminal 3, and a command relay part for bidirectionally relaying a command, i.e., an instruction to the application, between the user terminal connected by specifying the connection identifier and the operator terminal.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a server that remotely supports a user, a server control method, a server control program, and a remote support system.

  In a remote support system that accepts an inquiry call from a user, for example, in a call center, an automatic voice response (Interactive Voice Response, IVR) that automatically provides voice guidance in order to respond to many inquiries from the user while saving operator resources. May be used.

  With the spread of mobile terminals and the like having not only a voice call function but also a browser function, in a remote support system, in addition to the provision of voice information by IVR, the provision of visual information by a support application or the like executed on the terminal has been performed. It has become to.

  For example, in Patent Literature 1, when it is determined that the support application is available on the terminal, a voice prompting the user to start the support application is output to the terminal, and when the screen of the support application displayed on the terminal changes, the screen corresponding to the screen after the change An automatic voice response system that outputs voice to a terminal is described.

JP 2017-28346 A

  In a remote support system, a support application may provide support by voice communication with an operator in addition to providing visual information to support a user. At this time, if the remote support system only connects the user and the operator so that they can make a voice call, the operator cannot confirm the visual information provided to the user, and it is difficult to appropriately respond to the inquiry.

  Therefore, an object of the present invention is to provide a server that allows an operator to check a screen of a support application provided to a user when providing support by the operator together with the support application to the user.

  The server according to the present invention generates a response voice based on the user information and a user information storage unit that stores user information obtained by performing voice recognition processing on voice received from the user terminal, and transmits the response voice to the user terminal. An automatic voice response unit, an application execution request unit that requests the user terminal to execute the application based on the user information, and a request for a voice call with the operator transmitted from the user terminal in response to a predetermined operation of the user on the application. Connection request receiving unit for receiving an operator request to be transmitted, and connection identifier transmission for transmitting a connection identifier generated in response to the reception of the operator request for identifying the connection to the user terminal and the operator terminal used by the operator Unit and a command that causes the application to execute a predetermined process are transmitted to the user terminal connected by specifying the connection identifier. And a command relay unit that relays in both directions between the operator terminal specified to connected the connection identifier.

  Further, in the server according to the present invention, the user information includes at least a user identifier for identifying the user, the operator request includes at least the user identifier, and the server according to the present invention includes the user identifier included in the operator request. It is preferable to further include a user information transmitting unit that transmits user information including the user information to the operator terminal.

  In addition, the server according to the present invention preferably further includes a display information providing unit that provides display information for the application to display a screen on the user terminal to the user terminal and the operator terminal.

  Further, in the server according to the present invention, the command relay unit relays a command including a command for requesting a transition of a screen displayed by the application on the user terminal from the user terminal to the operator terminal, and the display information providing unit includes: It is preferable that display information for displaying the transition destination screen be provided to the operator terminal that has received the command in response to a request from the operator terminal that has received the command.

  Further, in the server according to the present invention, the command relay unit relays a command including a command for requesting shooting at the user terminal from the operator terminal to the user terminal, and the display information providing unit transmits the command from the user terminal that has received the command. It is preferable that the display information, which is an image captured by the user, is obtained, and the obtained display information is provided to the operator terminal that transmitted the command in response to a request from the operator terminal that transmitted the command.

  Further, in the server according to the present invention, the command relay unit relays a command including a command for requesting display of an image corresponding to the specified file on the user terminal from the operator terminal to the user terminal, and displays the display information providing unit. Preferably, in response to a request from the user terminal that has received the command, it is preferable to provide display information, which is a file obtained from the operator terminal that has transmitted the command, to the user terminal that has received the command.

  In the server according to the present invention, it is preferable that the command relay unit relays a report of an execution status of a predetermined process executed in accordance with the relayed command.

  A server control method according to the present invention includes: a user information storing step of storing user information obtained by performing voice recognition processing on a voice received from a user terminal; and generating a response voice based on the user information. An automatic voice response step for transmitting to the user, an application execution requesting step for requesting the user terminal to execute the application based on the user information, and a voice with the operator transmitted from the user terminal in response to a predetermined operation of the user for the application A connection request receiving step of receiving an operator request for calling, a connection identifier transmission step of transmitting a connection identifier generated in response to the reception of the operator request to a user terminal and an operator terminal used by the operator, A command that is a command to execute a predetermined process by specifying a connection identifier Including a command relay relaying bidirectionally between the operator terminal connected to a connect identifier and connection to the user terminal.

  A control program for a server according to the present invention includes a user information storing step of storing user information obtained by performing voice recognition processing on a voice received from a user terminal, and generating a response voice based on the user information. An automatic voice response step for transmitting to the user, an application execution requesting step for requesting the user terminal to execute the application based on the user information, and a voice with the operator transmitted from the user terminal in response to a predetermined operation of the user for the application A connection request receiving step of receiving an operator request for calling, a connection identifier transmission step of transmitting a connection identifier generated in response to the reception of the operator request to a user terminal and an operator terminal used by the operator, Command, which is a command for executing a predetermined process by the A command relay relaying bidirectionally between the operator terminal to the user terminal and the connection identifier is connected specified to which are connected with, to execute on the server.

  A remote support system according to the present invention is a remote support system having an operator terminal used by an operator and a server connectable to the operator terminal, wherein the server performs voice recognition processing on voice received from the user terminal. A user information storage unit that stores user information obtained by the user, an automatic voice response unit that generates a response voice based on the user information, and transmits the generated response voice to the user terminal, and requests the user terminal to execute the application based on the user information. An application execution requesting unit, a connection request receiving unit for receiving an operator request for requesting a voice call with an operator, transmitted from a user terminal in response to a predetermined operation of the user on the application, and a connection request receiving unit for receiving the operator request. Connection identifier transmission for transmitting the generated connection identifier to the user terminal and the operator terminal Command relay that bidirectionally relays a command, which is an instruction for causing an application to execute a predetermined process, between a user terminal connected by specifying a connection identifier and an operator terminal connected by specifying a connection identifier Unit.

  Further, in the remote support system according to the present invention, the command relay unit displays the number of pixels in the vertical direction and the horizontal direction of the display area used by the application for displaying the display information in the user terminal, and the application displays the display information on the user screen. When the display magnification is relayed from the user terminal to the operator terminal and the number of pixels and the display magnification are received, the operator terminal receives the received image in the shared screen area having the number of pixels in the vertical and horizontal directions according to the number of pixels. The command relay unit displays the display information at the displayed display magnification, relays a command including a command for requesting a predetermined amount of scrolling of the screen displayed by the application on the user terminal from the user terminal to the operator terminal, and receives the command. The operator terminal that has determined the predetermined amount included in the command, the number of pixels relayed and the number of pixels in the vertical and horizontal directions of the shared screen area Converted on the basis of a ratio, it is preferable to execute the command.

  Further, in the remote support system according to the present invention, the command relay unit relays the number of pixels in the vertical direction and the horizontal direction of the display area used by the application for displaying the display information in the user terminal, from the user terminal to the operator terminal, The operator terminal receiving the number, the coordinates specified as drawing target coordinates in the shared screen area having the number of pixels in the vertical and horizontal directions according to the number of pixels, the number of relayed pixels, and the vertical direction of the shared screen area A command including a command requesting drawing to the converted coordinates is generated and transmitted to a command relay unit, and the command relay unit transmits the command to the command relay unit. It is preferable to relay from the operator terminal to the user terminal.

  ADVANTAGE OF THE INVENTION According to this invention, when providing support by an operator with a support application to a user, the server which enables an operator to confirm the screen of the support application provided to the user can be provided.

It is a schematic diagram which shows the operation | movement outline | summary of a remote support system. It is a schematic diagram which shows the schematic structure of a server. It is a schematic diagram which shows the schematic structure of an operator terminal. It is a schematic diagram which shows the schematic structure of a user terminal. It is a sequence diagram of a relay start process. It is a figure showing an example of a screen displayed on a user terminal in relay start processing. It is a figure showing other examples of a screen displayed on a user terminal in relay start processing. It is a figure showing other examples of a screen displayed on a user terminal in relay start processing. It is a figure showing an example of a screen displayed on an operator terminal in relay start processing. It is a sequence diagram of a screen transition command relay process. It is a sequence diagram of a screen scroll command relay process. (A) is an example of a screen displayed on the user terminal in the screen scroll command relay processing, (b) is an example of a screen displayed on the operator terminal in the screen scroll command relay processing, and (c) is a screen scroll. It is an example of a screen displayed on a user terminal when a screen scroll command is requested in a command relay process. (A) is an example of a screen displayed on the user terminal after the execution of the screen scroll command in the screen scroll command relay processing, and (b) is an operator after receiving the screen scroll instruction at the user terminal in the screen scroll command relay processing. It is an example of the screen displayed on a terminal, and (c) is another example of the screen displayed on the operator terminal after receiving the screen scroll command in the user terminal in the screen scroll command relay processing. (A) is another example of the screen displayed on the user terminal in the screen scroll command relay processing, (b) is another example of the screen displayed on the operator terminal in the screen scroll command relay processing, (c) () Is an example of a screen displayed on the user terminal when a display magnification change command is requested in the screen scroll command relay processing. (A) is an example of a screen displayed on the user terminal after executing the display magnification change instruction in the screen scroll command relay processing, and (b) is a display magnification change instruction received by the user terminal in the screen scroll command relay processing. It is an example of the screen displayed on the operator terminal later, (c) is another example of the screen displayed on the user terminal after executing the screen scroll command in the screen scroll command relay processing, and (d) is the screen scroll command. It is another example of the screen displayed on the operator terminal after the screen scroll instruction is received by the user terminal in the relay processing. It is a sequence diagram of a photographing command relay process. It is a figure showing an example of a screen displayed on a user terminal in photography command relay processing. It is a figure showing other examples of a screen displayed on a user terminal in photography command relay processing. It is a figure showing an example of a screen displayed on an operator terminal in photography command relay processing. It is a figure showing other examples of a screen displayed on an operator terminal in photography command relay processing. It is a sequence diagram of a screen display command relay process. It is a figure showing an example of a screen displayed on an operator terminal in screen display command relay processing. It is a figure showing other examples of a screen displayed on an operator terminal in screen display command relay processing. It is a figure showing an example of a screen displayed on a user terminal in screen display command relay processing. It is a figure showing other examples of a screen displayed on a user terminal in screen display command relay processing. It is a sequence diagram of a drawing command relay process.

  Hereinafter, a server and a server control method will be described in detail with reference to the drawings. However, it should be understood that the invention is not limited to the drawings or the embodiments described below.

  FIG. 1 is a schematic diagram showing an operation outline of the remote support system.

  The remote support system 1 includes a server 2 and an operator terminal 3. The server 2 includes an IVR server 21, an application server 22, and a relay server 23. The remote support system 1 is connected to the user terminal 4 and remotely supports a user operating the user terminal 4.

  When the user operates the user terminal 4 to connect to the IVR server 21, the IVR server 21 provides the user terminal 4 with support by an automatic voice response. The IVR server 21 acquires user information about the user by receiving a voice based on an operation performed by the user on the user terminal 4 in response to the automatic voice response.

  The user terminal 4 executes the support application based on the application execution request transmitted by the IVR server 21.

  In response to the user's predetermined operation on the support application executed on the user terminal 4, the user terminal 4 transmits an operator request for requesting a call with the operator to the application server 22.

  Upon receiving the operator request, the application server 22 generates a connection identifier and transmits the connection identifier to the operator terminal 3 and the user terminal 4.

  The operator terminal 3 receiving the connection identifier requests the relay server 23 for a connection specifying the connection identifier.

  The user terminal 4 having received the connection identifier requests the relay server 23 for a connection specifying the connection identifier.

  The relay server 23 relays a command between the operator terminal 3 and the user terminal 4 connected by designating a connection identifier.

  For example, when the user causes the user terminal 4 to execute a command for changing the displayed screen to the screen X by a predetermined operation on the user terminal 4, the relay server 23 transmits the command for changing to the screen X from the user terminal 4 to the operator. Relay to terminal 3. The operator terminal 3 can display the same screen X as the screen X displayed on the user terminal 4 by executing the command for transitioning to the received screen X.

  As described above, when providing the support by the operator together with the support application to the user, the remote support system 1 allows the operator to check the screen of the support application provided to the user.

  FIG. 2 is a schematic diagram illustrating a schematic configuration of the server.

  The server 2 includes an IVR server 21, an application server 22, and a relay server 23.

  The IVR server 21 provides the user terminal 4 with support by an automatic voice response. For this purpose, the IVR server 21 includes an IVR communication unit 211, an IVR storage unit 212, and an IVR processing unit 213.

  The IVR communication unit 211 has a communication interface circuit for connecting the IVR server 21 to the operator terminal 3 and the user terminal 4. The IVR communication unit 211 connects the IVR server 21 to a network in which the connected terminals can perform voice communication with each other, for example, a public line network or the Internet. The IVR communication unit 211 supplies the data received from the user terminal 4 to the IVR processing unit 213. In addition, the IVR communication unit 211 transmits the data supplied from the IVR processing unit 213 to the user terminal 4 or the like.

  The IVR storage unit 212 has, for example, at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The IVR storage unit 212 stores a driver program, an operating system program, an application program, data, and the like used for processing by the IVR processing unit 213. For example, the IVR storage unit 212 stores a communication device driver program for controlling the IVR communication unit 211 and the like as a driver program. The various programs are stored in an IVR storage unit from a computer-readable portable recording medium such as a CD-ROM (Compact Disc Read-Only Memory) and a DVR-ROM (DVD Read-Only Memory) using a known setup program. 212 may be installed.

  The IVR processing unit 213 includes one or more processors and their peripheral circuits. The IVR processing unit 213 controls the overall operation of the IVR server 21 and is, for example, a CPU (Central Processing Unit). The IVR processing unit 213 controls operations of the IVR communication unit 211 and the like such that various processes of the IVR server 21 are executed by appropriate means based on a program or the like stored in the IVR storage unit 212. The IVR processing unit 213 executes a process based on a program (a driver program, an operating system program, an application program, etc.) stored in the IVR storage unit 212. Further, the IVR processing unit 213 can execute a plurality of programs (such as application programs) in parallel.

  The IVR processing unit 213 includes a user information storage unit 2131, an automatic voice response unit 2132, an application execution request unit 2133, a user information transmission unit 2134, and a voice relay unit 2135. Each of these units included in the IVR processing unit 213 is a functional module implemented by a program executed on a processor included in the IVR processing unit 213. Alternatively, these units included in the IVR processing unit 213 may be implemented in the IVR server 21 as an independent integrated circuit, microprocessor, or firmware.

  The application server 22 provides a user with support by visual information in cooperation with a support application executed on the user terminal 4. The application server 22 transmits and receives data according to a communication protocol such as HTTP. To this end, the application server 22 includes an application communication unit 221, an application storage unit 222, and an application processing unit 223.

  The application communication unit 221 has a communication interface circuit for connecting the application server 22 to the operator terminal 3 and the user terminal 4. The application communication unit 221 connects the application server 22 to a network in which the connected terminals perform data communication with each other, for example, the Internet. The application communication unit 221 supplies the data received from the user terminal 4 to the application processing unit 223. Further, the application communication unit 221 transmits the data supplied from the application processing unit 223 to the user terminal 4 or the like.

  The application storage unit 222 has, for example, at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The application storage unit 222 stores a driver program, an operating system program, an application program, data, and the like used for processing by the application processing unit 223. For example, the application storage unit 222 stores a communication device driver program that controls the application communication unit 221 as a driver program. The various programs are stored in an application storage unit from a computer-readable portable recording medium such as a CD-ROM (Compact Disc Read-Only Memory) and a DVR-ROM (DVD Read-Only Memory) using a known setup program. 222 may be installed.

  The application processing unit 223 includes one or more processors and their peripheral circuits. The application processing unit 223 controls the overall operation of the application server 22 as a whole, and is, for example, a CPU (Central Processing Unit). The application processing unit 223 controls operations of the application communication unit 221 and the like such that various processes of the application server 22 are executed by appropriate means based on a program or the like stored in the application storage unit 222. The application processing unit 223 executes a process based on a program (a driver program, an operating system program, an application program, and the like) stored in the application storage unit 222. Further, the application processing unit 223 can execute a plurality of programs (such as application programs) in parallel.

  The application processing unit 223 includes a display information providing unit 2231, a connection request receiving unit 2232, and a connection identifier transmitting unit 2233. Each of these units included in the application processing unit 223 is a functional module implemented by a program executed on a processor included in the application processing unit 223. Alternatively, each of these units included in the application processing unit 223 may be implemented on the application server 22 as an independent integrated circuit, microprocessor, or firmware.

  The relay server 23 bidirectionally relays a command between the operator terminal 3 and the user terminal 4 connected by the WebSocket protocol by designating a predetermined identifier. To this end, the relay server 23 includes a relay communication unit 231, a relay storage unit 232, and a relay processing unit 233.

  The relay communication unit 231 has a communication interface circuit for connecting the relay server 23 to the operator terminal 3 and the user terminal 4. The relay communication unit 231 connects the relay server 23 to a network in which the connected terminals perform data communication with each other, for example, the Internet. The relay communication unit 231 supplies the data received from the operator terminal 3 or the user terminal 4 to the relay processing unit 233. Further, the relay communication unit 231 transmits the data supplied from the relay processing unit 233 to the operator terminal 3 or the user terminal 4.

  The relay storage unit 232 has, for example, at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The relay storage unit 232 stores a driver program, an operating system program, an application program, data, and the like used for processing by the relay processing unit 233. For example, the relay storage unit 232 stores a communication device driver program that controls the relay communication unit 231 as a driver program. The various programs are transferred from a computer-readable portable recording medium such as a CD-ROM (Compact Disc Read-Only Memory) or a DVR-ROM (DVD Read-Only Memory) to a relay storage unit using a known setup program or the like. 232 may be installed.

  The relay processing unit 233 includes one or more processors and their peripheral circuits. The relay processing unit 233 controls the overall operation of the relay server 23 in an integrated manner, and is, for example, a CPU (Central Processing Unit). The relay processing unit 233 controls operations of the relay communication unit 231 and the like such that various processes of the relay server 23 are executed by appropriate means based on a program or the like stored in the relay storage unit 232. The relay processing unit 233 executes a process based on a program (a driver program, an operating system program, an application program, and the like) stored in the relay storage unit 232. Further, the relay processing unit 233 can execute a plurality of programs (such as application programs) in parallel.

  Further, the protocol for connecting to the relay server 23 is not limited to WebSocket, but may be another protocol capable of bidirectional relay between terminals connected by specifying an identifier.

  The relay processing unit 233 includes a command relay unit 2331. The command relay unit 2331 is a functional module implemented by a program executed on a processor of the relay processing unit 233. Alternatively, the command relay unit 2331 may be implemented in the relay server 23 as an independent integrated circuit, a microprocessor, or firmware.

  The server 2 of the present disclosure, by adopting the above-described hardware configuration, enables the operator to check the screen of the support application provided to the user when providing support by the operator together with the support application to the user.

  In the present embodiment, the server 2 includes an IVR server 21, an application server 22, and a relay server 23, which are independent hardware. The server 2 may be constituted by a single piece of hardware capable of simultaneously executing the functions of these servers.

  FIG. 3 is a schematic diagram showing a schematic configuration of the operator terminal.

  The operator terminal 3 connects to the server 2 and communicates with the server 2. The operator terminal 3 provides support to the user when operated by the operator. For this purpose, the operator terminal 3 includes an operator communication unit 31, an operator storage unit 32, an operator input / output unit 33, an operator display unit 34, and an operator processing unit 35. The operator terminal 3 is an information processing device capable of data communication, for example, a personal computer (PC).

  The operator communication unit 31 includes a communication interface circuit, and connects the operator terminal 3 to the server 2 via a network. The operator communication unit 31 transmits the data supplied from the operator processing unit 35 to the server 2 via a network. The operator communication unit 31 supplies the data received from the server 2 via the network to the operator processing unit 35.

  The operator storage unit 32 includes, for example, a magnetic disk device or a semiconductor memory device. The operator storage unit 32 stores an operating system program, a driver program, an application program, data, and the like used for processing in the operator processing unit 35. For example, the operator storage unit 32 stores an input device program for controlling the operator input / output unit 33 and the like as a driver program. Further, the operator storage unit 32 stores, as an application program, a program or the like for requesting the server 2 to connect by designating a connection identifier. Further, the operator storage unit 32 may temporarily store temporary data related to a predetermined process.

  The operator input / output unit 33 is a device that receives an operation of the operator on the operator terminal 3 and is, for example, a keyboard or a mouse. The operator can use the operator input / output unit 33 to input an instruction for specific coordinates on the screen, such as characters, numbers, and symbols. When operated by the operator, the operator input / output unit 33 generates a signal corresponding to the operation. Then, the generated signal is supplied to the operator processing unit 35 as an instruction of the operator.

  The operator display unit 34 is a device that displays images, images, and the like, and is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The operator display unit 34 displays an image corresponding to the video data supplied from the operator processing unit 35, an image corresponding to the image data, and the like.

  The operator processing unit 35 includes one or more processors and their peripheral circuits. The operator processing unit 35 controls the overall operation of the operator terminal 3 as a whole, and is, for example, a CPU. The operator processing unit 35 controls the operator communication unit 31 so that various processes of the operator terminal 3 are executed by appropriate means based on a program stored in the operator storage unit 32, an operation on the operator input / output unit 33, and the like. And the operation of the operator display unit 34 and the like. The operator processing unit 35 executes a process based on a program (operating system program, driver program, application program, etc.) stored in the operator storage unit 32. Further, the operator processing unit 35 can execute a plurality of programs (such as application programs) in parallel.

  The operator processing section 35 includes an operator communication section 351, an operator command generation section 352, an operator command transmission / reception section 353, and an operator command execution section 354. These units are functional modules realized by a program executed by a processor included in the operator processing unit 35. Alternatively, these units may be implemented in the operator terminal 3 as firmware.

  FIG. 4 is a schematic diagram illustrating a schematic configuration of the user terminal.

  The user terminal 4 connects to the server 2 and communicates with the server 2. The user terminal 4 is operated by a user and requests the server 2 for support. To this end, the user terminal 4 includes a user communication unit 41, a user storage unit 42, a user input / output unit 43, a user display unit 44, and a user processing unit 45. The user terminal 4 is an information processing device capable of data communication, for example, a multifunctional mobile phone (so-called “smartphone”), a mobile phone (so-called “feature phone”), a tablet terminal, a personal computer, or the like.

  The user communication unit 41 includes a communication interface circuit, and connects the user terminal 4 to the server 2 via a network. The user communication unit 41 connects the user terminal 4 to a network in which the connected terminals can make voice calls with each other, for example, a public line network or the Internet, and a network in which the connected terminals perform data communication with each other, for example, the Internet. Connect to The user communication unit 41 transmits the data supplied from the user processing unit 45 to the server 2 via a network. Further, the user communication unit 41 supplies the data received from the server 2 via the network to the user processing unit 45.

  The user storage unit 42 includes, for example, a semiconductor memory device. The user storage unit 42 stores an operating system program, a driver program, an application program, data, and the like used for processing in the user processing unit 45. For example, the user storage unit 42 stores an input device program for controlling the user input / output unit 43 and the like as a driver program. In addition, the user storage unit 42 stores, as an application program, a program for requesting the server 2 to connect by designating a connection identifier. Further, the user storage unit 42 may temporarily store temporary data related to a predetermined process.

  The user input / output unit 43 is a device that receives or outputs an input to the user terminal 4, and is, for example, a microphone, a speaker, a camera, a touch panel or a key button that receives a user operation. The user can use the user input / output unit 43 to input voice, instructions for specific coordinates on the screen, such as characters, numbers, and symbols. When operated by a user, the user input / output unit 43 generates a signal corresponding to the operation. Then, the generated signal is supplied to the user processing unit 45 as a user's instruction. Also, it outputs a sound or the like according to the signal generated by the user processing unit 45.

  The user display unit 44 is a device that reproduces sound and images, and is, for example, a speaker, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The user display unit 44 reproduces a sound corresponding to the sound data supplied from the user processing unit 45, an image corresponding to the image data, and the like.

  The user processing unit 45 includes one or more processors and their peripheral circuits. The user processing unit 45 controls the overall operation of the user terminal 4 and is, for example, a CPU. The user processing unit 45 controls the user communication unit 41 so that various processes of the user terminal 4 are executed by appropriate means based on a program stored in the user storage unit 42, an operation on the user input / output unit 43, and the like. And the operation of the user display unit 44 and the like. The user processing unit 45 performs processing based on programs (such as an operating system program, a driver program, and an application program) stored in the user storage unit 42. Further, the user processing unit 45 can execute a plurality of programs (such as application programs) in parallel.

  The user processing unit 45 includes a user communication unit 451, a user command generation unit 452, a user command transmission / reception unit 453, and a user command execution unit 454. Each of these units is a functional module realized by a program executed by a processor included in the user processing unit 45. Alternatively, these units may be implemented in the user terminal 4 as firmware.

  FIG. 5 is a sequence diagram of the relay start process.

  First, the user terminal 4 transmits a signal corresponding to the utterance of the user to the IVR server 21 (Step S101). The user communication unit 451 of the user terminal 4 calls a predetermined telephone number to connect to the automatic voice response unit 2132 of the IVR server 21 via a network in which the connected terminals can perform voice communication with each other. Then, the user communication unit 451 generates a signal based on the user's uttered voice received by the user input / output unit 43, and transmits the signal to the automatic voice response unit 2132 via the network.

  Next, the user information storage unit 2131 of the IVR server 21 performs a voice recognition process on the received voice, and stores the user information obtained by the voice recognition process (Step S102). The voice recognition processing is performed, for example, by converting a received signal into text and determining whether the converted text includes a predetermined keyword. The acquired user information is stored in the IVR storage unit 212. The user information is, for example, a question item of the user, a member number of the user, a member name, an age, a contract status and the like. The IVR server 21 acquires the member number of the user by voice (for example, a tone output in response to an operation on the keypad serving as the user input / output unit 43), refers to the member database using the acquired member number, and performs other operations. May be obtained. The member number of the user is an example of a user identifier for identifying the user. The user identifier may be a caller number of the user terminal 4 calling the IVR server 21.

  Next, the automatic voice response unit 2132 transmits to the user terminal 4 data corresponding to the response voice generated based on the recognized user information and the data stored in the IVR storage unit 212 (step S103). The user terminal 4 reproduces the response voice received by the user communication unit 41 on the user display unit 44. The user can speak a response to the response voice. That is, the processing of steps S101 to S103 may be repeated two or more times.

  Next, the application execution requesting unit 2133 of the IVR server 21 requests the user terminal 4 to execute the support application based on the user information (Step S104). The IVR communication unit 211 of the IVR server 21 transmits a message including information for executing the support application to the user terminal 4 by SMS. The IVR communication unit 211 may transmit a message to the user terminal 4 by e-mail. The transmission of the message to the user terminal 4 may be performed by a server other than the IVR server 21 that has received the application execution instruction based on the user information.

  The user terminal 4 that has received the request to execute the application executes the support application (Step S105). The support application is an application program executed by the user processing unit 45 of the user terminal. The support application may be an application program executed by the application processing unit 223 of the application server 22. In this case, the user terminal 4 receives the image generated by the application processing unit 223 from the application server 22 via the user communication unit 41, and displays the image on the user display unit 44.

  The user terminal 4 displays a message notifying that an application execution request has been received on the user display unit 44, and executes the application after accepting a user operation on the user input / output unit 43 that permits execution. It may be.

  FIG. 6 is a diagram illustrating an example of a screen displayed on the user terminal in the relay start processing.

  A screen 600 shown in FIG. 6 is displayed on the user display unit 44 when the user terminal 4 connected to the IVR server 21 and receiving support by the automatic voice response receives a request to execute an application. The screen 600 has a call time display 601, a call end icon 602, and a support application execution message 603.

  The call time display 601 indicates the time elapsed since connection with the IVR server 21. The call end icon 602 is an icon for accepting a user operation for ending a call with the IVR server 21. The support application execution message 603 is a message that notifies the user that the user terminal 4 has received a request to execute the support application. When the user input / output unit 43 receives a predetermined user operation on the support application execution message 603, for example, a tap on the display area of the support application execution message 603, the user terminal 4 executes the support application.

  Returning to FIG. 5, the user terminal 4 that executes the support application requests information to be displayed on the support application screen from the application server 22, and acquires the information from the display information providing unit 2231 of the application server 22 (Step S106).

  FIG. 7 is a diagram illustrating another example of the screen displayed on the user terminal in the relay start processing. The user terminal 4 displays a support screen on the user display unit 44 based on the information obtained from the application server 22 in step S106. A support screen 700 shown in FIG. 7 is an example of a support screen displayed on the user display unit 44 by the support application, and includes a screen title 701, a menu display icon 702, a detailed item request area 703-705, an operator request icon 706, and a top menu request. An icon 707 is provided.

  The screen title 701 indicates the title of the currently displayed support screen 700. The menu display icon 702 is an icon for receiving a predetermined user operation (for example, a tap on a display area) for displaying a menu screen on the user display unit 44. The detailed item request areas 703 to 705 are areas for receiving a predetermined user operation (for example, tapping the display area) for displaying the corresponding detailed item screen on the user display unit 44. The operator request icon 706 is an icon for receiving a predetermined user operation (for example, a tap on a display area) for transmitting an operator request for requesting telephone support by the operator to the user communication unit 41. The top menu request icon 707 is an icon for receiving a predetermined user operation (for example, tapping the display area) for displaying the top menu on the user display unit 44.

  Returning to FIG. 5, the user terminal 4 transmits an operator request for requesting telephone support by the operator to the application server 22 based on a predetermined operation of the user (step S107), and transmits the transmitted operator request to the application server 22. The connection request receiving unit 2232 receives it.

  The operator request may include a user identifier. For example, when the user identifier is a member number, the user terminal 4 reads out the member number stored in the user storage unit 42 in advance and includes it in the operator request.

  FIG. 8 is a diagram illustrating another example of the screen displayed on the user terminal in the relay start processing.

  When the user performs a predetermined operation on the operator request icon 706 on the support screen 700 in FIG. 7, the support application overlays a support method selection dialog 800 on the support screen 700. The support method selection dialog 800 has a dialog title 801, a dialog end icon 802, a chat support request area 803, and a telephone support request area 804.

  The dialog title 801 notifies that the support method selection dialog 800 requires the user to select a support method. The dialog end icon 802 is an icon for receiving a predetermined user operation (for example, a tap on the display area) for closing the support method selection dialog 800. The chat support request area 803 is an area for receiving a predetermined user operation (for example, a tap on a display area) for requesting chat support by an operator. The telephone support request area 804 is an area for receiving a predetermined user operation (for example, a tap on a display area) for requesting telephone support by an operator.

  When the user performs a predetermined operation on the operator request icon 706 on the support screen 700 in FIG. 7, the support application uses a dialog other than the support method selection dialog 800 to request telephone support by the operator. A predetermined user operation may be received. Further, when the user performs a predetermined operation on the operator request icon 706, the support application may transmit the operator request without displaying the dialog.

  Returning to FIG. 5, upon receiving the operator request, the application server 22 generates a connection identifier (Step S108). The connection identifier is an identifier for identifying the connection to the relay server 23, and is, for example, an alphanumeric character string of a predetermined length generated based on a random number.

  The connection identifier transmitting unit 2233 of the application server 22 transmits the generated connection identifier to the IVR server 21 (Step S109).

  The user information transmitting unit 2134 of the IVR server 21 that has received the connection identifier transmits the connection identifier to the operator terminal 3 together with the user information corresponding to the connection identifier stored in the IVR storage unit 212 (Step S110).

  For example, in step S102, the IVR storage unit 212 stores the user information in association with the member number, and in step S107, the user terminal 4 transmits an operator request including the member number to the application server 22 in step S107. In step S109, the application server 22 transmits the connection identifier together with the member number to the IVR server 21, and searches the IVR storage unit 212 using the member number received by the IVR server 21, thereby enabling the identification.

  Further, the application server 22 transmits the generated connection identifier to the user terminal 4 (Step S111).

  The operator terminal 3 receiving the connection identifier specifies the connection identifier and connects to the relay server 23 (step S112).

  The user terminal 4 having received the connection identifier specifies the connection identifier and connects to the relay server 23 (step S113).

  The command relay unit 2331 of the relay server 23 bidirectionally relays the command between the operator terminal 3 connected by specifying the connection identifier and the user terminal 4 connected by specifying the connection identifier (step S114). ).

  FIG. 9 is a diagram illustrating an example of a screen displayed on the operator terminal in the relay start processing.

  The operator support screen 900 displayed on the operator display unit 34 of the operator terminal 3 in which commands are bidirectionally relayed between the user terminal 4 and the user terminal 4 by the relay server 23 includes a user information display area 910 and a connection terminal display area 920. Have.

  The user information display area 910 displays a user information display column 911 that displays user information about the user such as the user's name and age, and user information about the relationship between the user and the user such as a contract relationship that the user has with the service provider. And a related information display column 912 to be displayed.

  The connected terminal display area 920 transmits a terminal information display column 921 for displaying terminal information such as a terminal name, an OS, and a connected network of the connected user terminal 4 and a command for requesting the connected user terminal to perform processing. Command display icons 9221-9225, a shared screen display area 923, and an auxiliary display area 925.

  Processing corresponding to a predetermined operation of the operator on the command transmission icons 9221-9225 will be described later.

  In the shared screen display area 923, a shared screen 924 corresponding to the screen displayed on the user display unit 44 of the connected user terminal 4 is displayed.

  Referring to such an operator support screen 900, the operator provides remote support to the user.

  Further, the voice relay unit 2135 of the IVR server 21 bidirectionally relays voice between the operator communication unit 351 of the operator terminal 3 and the user communication unit 451 of the user terminal 4 (step S115).

  Subsequently, a command relay process between the operator terminal 3 and the user terminal 4 will be specifically described.

  FIG. 10 is a sequence diagram of the screen transition command relay processing.

  First, the user command generation unit 452 of the user terminal 4 generates a screen transition command for requesting a screen transition based on a predetermined operation of the user on the user input / output unit 43 (Step S201). The screen transition is to change the screen displayed on the user display unit 44 to another screen. For example, based on a tap on the detailed item request area 703 on the support screen 700 shown in FIG. 7, a command for requesting a transition to a “charge” screen is generated.

  The user command transmitting / receiving unit 453 of the user terminal 4 transmits the generated screen transition command to the relay server 23 via the user communication unit 41, and the command relay unit 2331 of the relay server 23 receives the received screen transition command from the user terminal 4. The screen transition command is relayed to the operator terminal 3 (step S202).

  The user terminal 4 requests display information from the application server 22 based on the generated screen transition command. The display information providing unit 2231 of the application server 22 transmits the display information in response to a request from the user terminal 4, and the user terminal 4 acquires the display information (Step S203).

  The operator terminal 3 that has received the screen transition command requests display information from the application server 22 based on the received screen transition command. The display information providing unit 2231 of the application server 22 transmits the display information in response to the request of the operator terminal 3, and the operator terminal 3 acquires the display information (Step S204).

  The user command execution unit 454 of the user terminal 4 executes screen transition using the display information acquired from the application server 22 (Step S205).

  The operator command execution unit 354 of the operator terminal 3 executes screen transition using the display information acquired from the application server 22 (Step S206).

  By executing the processing as described above, the same screen transition as that performed by the user terminal 4 can be performed by the operator terminal 3.

  Further, when a screen transition is requested from the operator terminal 3 by the screen transition command transmission icon 9221 displayed on the operator support screen 900 of the operator terminal 3, the operator terminal 3 and the user terminal 4 in the above-described processing are exchanged. Similar processing can be executed.

  FIG. 11 is a sequence diagram of the screen scroll command relay processing.

  First, the user command generation unit 452 of the user terminal 4 generates a screen scroll command for requesting a screen scroll based on a predetermined operation of the user on the user input / output unit 43 (step S301). Screen scrolling refers to changing the area displayed on the user display unit 44 in the screen generated by the support application. For example, based on an upward flick on the user input / output unit 43, which is a touch panel, a request is made to change the displayed area to a lower side by a predetermined amount (for example, 50% of the vertical length of the user display unit 44). Command to be generated.

  The user command generation unit 452 of the user terminal 4 transmits the generated screen scroll command to the relay server 23 via the user communication unit 41, and the command relay unit 2331 of the relay server 23 transmits the received screen scroll command to the operator terminal. 3 (step S302).

  The user command execution unit 454 of the user terminal 4 executes screen scrolling based on the generated screen scroll command (Step S303).

  The operator command execution unit 354 of the operator terminal 3 executes screen scrolling based on the received screen scrolling command (Step S304).

  In executing the screen scroll command relay process, the command relay unit 2331 of the relay server 23 includes a vertical and horizontal pixels of a display area used by the support application for displaying display information in the user display unit 44 transmitted by the user terminal 4. The number (for example, 720 × 1280) and the display magnification (for example, 150%) when the support application displays the display information on the user terminal 4 may be relayed to the operator terminal 3.

  The operator terminal 3 that has received the number of pixels and the display magnification displays the display information at the received display magnification in the area of the shared screen 924 having the number of pixels in the vertical direction and the horizontal direction corresponding to the received number of pixels.

  The screen scroll command relay processing between the operator terminal 3 and the user terminal 4 will be described in more detail with reference to FIGS.

  FIG. 12A is an example of a screen displayed on the user terminal in the screen scroll command relay processing.

The image 1200 generated by the support application has six areas A to F. W ₁ is the number of pixels in the horizontal direction display region used to support applications in the user terminal 4 is, for example, 800 pixels, H ₁ is the number of pixels in the vertical direction is, for example, 1280 pixels. Further, the heights of the regions A to F are, for example, 320 pixels, respectively. At this time, the total height of the areas A to D is 1280 pixels, which is included in the display area used for displaying the display information by the support application on the user display unit 44. Therefore, the areas A to D are displayed as the screen 1201.

  FIG. 12B is an example of a screen displayed on the operator terminal in the screen scroll command relay processing.

The operator terminal 3 has a shared screen 924 for displaying a screen of the support application on the user terminal 4. W ₂ transverse the number of pixels of the shared screen 924 is, for example, 1600 pixels, H ₂ in the longitudinal number of pixels is, for example, 2560 pixels.

  The operator processing unit 35 generates the image 1210 by using the same display information as that used by the support application to generate the image 1200 on the user display unit. The operator processing unit 35 generates the image 1210 according to the number of pixels of the shared screen 924 to be displayed.

  The image 1210 has six regions A to F as in the image 1200. In addition, the heights of the areas A to F are each 640 pixels. At this time, the total height of the areas A to D is 2560 pixels and is included in the shared screen 924, so that the areas A to D are displayed as the screen 1211.

  FIG. 12C is an example of a screen displayed on the user terminal when a screen scroll command is requested in the screen scroll command relay processing.

  When the user input / output unit 43 detects that the user has flicked a predetermined distance upward on the screen 1201 on which the user input / output unit 43, which is a touch panel, is superimposed, the user processing unit 45 performs, based on the flicked distance, A screen scroll command is generated that requests that the displayed area be changed downward by ΔH (eg, 640 pixels).

  FIG. 13A is an example of a screen displayed on the user terminal after the execution of the screen scroll command in the screen scroll command relay processing.

  When the user processing unit 45 executes the screen scroll command, the area displayed on the user display unit 44 in the image 1200 generated by the support application is changed. That is, the areas A and B corresponding to the height of ΔH (for example, 640 pixels) move above the display area, and the areas E and F below the display screen move inside the display area. As a result, the area CF is included in the area used for displaying the support application, and the scrolled screen 1301 is displayed on the user display unit 44.

  FIG. 13B is an example of a screen displayed on the operator terminal after receiving a screen scroll command at the user terminal in the screen scroll command relay processing.

  When the operator processing unit 35 of the operator terminal 3 that has received the screen scroll command executes the screen scroll command, an area of the image 1210 displayed on the shared screen 924 is changed. That is, the area A corresponding to the height of ΔH (for example, 640 pixels) moves above the display area, and the area E below the display screen moves inside the display area. As a result, the area BE is included in the shared screen 924, and the scrolled screen 1311 is displayed on the operator display unit 34.

  FIG. 13C shows another example of the screen displayed on the operator terminal after receiving the screen scroll command at the user terminal in the screen scroll command relay processing.

  Unlike the example illustrated in FIG. 13B, in the example illustrated in FIG. 13C, the operator processing unit 35 determines ΔH that is the scroll amount included in the received command by using the pixel of the display area in the user terminal 4. The conversion is performed based on the number and the number of pixels of the display area in the operator terminal 3.

That is, when the number of pixels in the vertical direction of the display area in the user terminal 4 is 1280 pixels and the number of pixels in the vertical direction of the display area in the operator terminal 3 is 2560 pixels, the pixel ratio α _{H in the} vertical direction is 2560 ÷ 1280 = 2.

The operator processing unit 35, based on the vertical pixel ratio alpha, the ΔH × α _H = 1280, is used as the moving amount of the vertical scrolling of the operator display unit 34.

  When the operator processing unit 35 executes a screen scroll command with a movement amount of 1280 pixels, the areas A and B corresponding to a height of 1280 pixels in the image 1210 move above the display area and move below the display screen. The areas E and F on the side move to the inside of the display area. As a result, the area CF is included in the shared screen 924, and the scrolled screen 1302 is displayed on the operator display unit 34.

  FIG. 14A is another example of a screen displayed on the user terminal in the screen scroll command relay processing.

The user terminal 4 is W ₁ is the number of pixels in the horizontal direction, for example 800 pixels, H ₁ is the number of pixels in the vertical direction is, for example 1280 pixels, the region used for the display of the support application, the image 1400 A screen 1401 including the whole is displayed.

  FIG. 14B is another example of a screen displayed on the operator terminal in the screen scroll command relay processing.

Operator terminal 3 acquires display information for displaying an image 1400, a W ₂ of the number of pixels in the horizontal direction, for example, 1600 pixels, shared screen of H ₂ longitudinal number of pixels is, for example, 2560 pixels At 924, a screen 1411 including the entire image 1410 is displayed based on the display information. The operator processing unit 35 generates an image 1410 according to the number of pixels of the shared screen 924 to be displayed.

  FIG. 14C is an example of a screen displayed on the user terminal when a display magnification change command is requested in the screen scroll command relay processing.

  When the user input / output unit 43 detects a pinch-out operation by the user on the touch panel superimposed on the user display unit 44 that displays the screen 1401, the user processing unit 45 displays the image 1400 displayed on the screen 1401. A display magnification change command for changing the magnification is generated and transmitted to the relay server 23 via the user communication unit 41.

  FIG. 15A is an example of a screen displayed on the user terminal after the execution of the display magnification change command in the screen scroll command relay processing.

The user processing unit 45 performs the display magnification changing instruction generated based on the pinch-out operation of the user, and change the magnification of the image 1400, the lateral pixel number W ₃ and vertical pixel number H ₃ Image 1500 is generated. Then, a part of the image 1500 is displayed in an area used for displaying the support application.

  In FIG. 15A, the user flicks a predetermined distance upward on the screen 1501 on which the user input / output unit 43 as a touch panel is superimposed, and the user input / output unit 43 detects this and performs a user process. The unit 45 generates a screen scroll instruction requesting that the displayed area be changed downward by ΔH (for example, 640 pixels) based on the flicked distance.

  FIG. 15B is an example of a screen displayed on the operator terminal after receiving a display magnification change instruction at the user terminal in the screen scroll command relay processing.

  The operator processing unit 35 executes the display magnification change command received via the operator communication unit 31, and generates an image 1510 in which the display magnification of the image 1410 is changed. Then, a part of the image 1510 is displayed on the shared screen 924.

  At this time, since the image 1410 is generated according to the number of pixels of the shared screen 924, by setting the display magnification of the image 1410 to the magnification specified by the display magnification change command, the operator processing unit 35 The operator can display the same image as the image displayed on the operator display unit 34.

  FIG. 15C shows another example of the screen displayed on the user terminal after the execution of the screen scroll command in the screen scroll command relay processing.

  In the image 1500, the user processing unit 45 executes a screen scroll command requesting that the displayed area be changed downward by ΔH, so that the area of the image 1500 displayed on the user display unit 44 is changed. You. That is, the user processing unit 45 slides the area displayed on the screen 1502 after scrolling of the image 1500 by ΔH below the area displayed on the screen 1501 before scrolling.

FIG. 15D is another example of a screen displayed on the operator terminal after the screen scroll command is received by the user terminal in the screen scroll command relay processing.
When a screen scroll command is received via the operator communication unit 31, the operator processing unit 35 calculates ΔH, which is the scroll amount, based on the number of pixels in the display area in the user terminal 4 and the number of pixels in the display area in the operator terminal 3. And execute the scroll instruction.

  That is, the operator processing unit 35 multiplies the area displayed on the shared screen 924 of the image 1510 by ΔH by the pixel ratio α between the number of pixels in the display area in the user terminal 4 and the number of pixels in the display area in the operator terminal 3. Slide down by

  As described above, the amount of scroll movement is converted based on the number of pixels in the display area in the user terminal 4 and the number of pixels in the display area in the operator terminal 3, so that the vertical and horizontal directions of the display area in the user display unit 44 are changed. When the number of pixels of the display area does not match the number of pixels occupied by the area of the shared screen 924 in the operator terminal 3, the predetermined amount for changing the displayed area is not the ratio to the display area of the user display unit 44 but the number of pixels. Even when the operator terminal 3 receives the command to be specified, the operator terminal 3 can cause the operator display unit 34 to display a scroll screen similar to the support application screen displayed on the user display unit 44. .

  The relay of the resolution information of the user display unit transmitted by the user terminal 4 to the operator terminal 3 by the command relay unit 2331 may be executed at the start of command relay in the relay start process (step S114).

  Executing the processing as described above enables the operator terminal 3 to execute the same screen scroll as the screen scroll executed by the user terminal 4.

  Further, when a screen transition is requested from the operator terminal 3 by the screen scroll command transmission icon 9222 displayed on the operator support screen 900 of the operator terminal 3, by replacing the operator terminal 3 and the user terminal 4 in the above processing, Similar processing can be executed.

  FIG. 16 is a sequence diagram of the photographing command relay process.

  First, the operator command generation unit 352 of the operator terminal 3 generates a shooting command for requesting shooting with the user terminal 4 (Step S401). The photographing command is generated when the operator input / output unit 33 acquires a predetermined operation of the photographing command transmission icon 9223 displayed on the operator support screen 900 of the operator terminal 3 by the operator.

  The operator command transmitting / receiving unit 353 of the operator terminal 3 transmits the generated photographing command to the relay server 23 via the operator communication unit 31. The command relay unit 2331 of the relay server 23 transmits the photographing command received from the operator terminal 3. Is relayed to the user terminal 4 (step S402).

  The user terminal 4 that has received the shooting command displays a shooting permission dialog requesting the user to permit the support application to access the camera on the user display unit 44 (step S403).

  FIG. 17 is a diagram illustrating an example of a screen displayed on the user terminal in the photographing command relay processing.

  When a shooting command is received while the support screen 700 in FIG. 7 is displayed, the user terminal 4 overlays a shooting permission dialog 1700 on the support screen 700. The shooting permission dialog 1700 includes a shooting permission request 1701, a non-permission area 1702, and a permission area 1703.

  The photographing permission request 1701 displays information indicating to the user that the support application is requesting access to the camera. By performing a predetermined operation on the non-permission area 1702 or the permission area 1703 via the user input / output unit 43, the user can reject or permit access to the camera by the support application, that is, shooting.

  Returning to FIG. 16, when the user is permitted to access the camera, the user command execution unit 454 of the user terminal 4 starts the camera (step S404).

  FIG. 18 is a diagram illustrating another example of a screen displayed on the user terminal in the photographing command relay processing.

  In the user terminal 4 on which the shooting permission dialog 1700 shown in FIG. 17 is displayed, when the user performs a predetermined operation on the permission area 1703, shooting is permitted, the support application activates the camera, and shifts to the shooting mode. . A shooting screen 1800 illustrated in FIG. 18 is a screen displayed on the user display unit 44 by the support application in the shooting mode. The shooting screen 1800 has a shooting screen title 1801, a shooting preview 1802, a shooting icon 1803, and a shooting end icon 1804.

  The shooting screen title 1801 indicates to the user that the screen displayed on the user display unit 44 is a shooting screen. The shooting preview 1802 displays an image captured by a camera, which is one of the user input / output units 43. The preview may be a display of an image obtained by the camera before shooting, or a display of an image obtained based on a shooting instruction and before being stored. The shooting icon 1803 is an icon for receiving an operation of a user who performs shooting. When the user input / output unit 43 receives a user's predetermined operation on the area corresponding to the shooting icon 1803, image data corresponding to the image displayed on the shooting preview 1802 is stored in the user storage unit 42. The shooting end icon 1804 is an icon for receiving a user operation for terminating the shooting mode. When the user input / output unit 43 receives a user's predetermined operation on the area corresponding to the shooting end icon 1804, the user terminal 4 ends the shooting mode and returns to the original mode.

  Returning to FIG. 16, the user command transmission / reception unit 453 of the user terminal 4 on which the camera has been started transmits a report indicating that the camera is taking an image to the relay server 23 via the user communication unit 41 as a command execution status. Then, the command relay unit 2331 of the relay server 23 relays the received report to the operator terminal 3 (Step S405).

  The operator terminal 3 that has received the report that the user terminal 4 is capturing an image displays the execution status of the user terminal 4 on the operator display unit 34 (step S406).

  FIG. 19 is a diagram illustrating an example of a screen displayed on the operator terminal in the photographing command relay processing.

  The operator support screen 1900 illustrated in FIG. 19 is a screen obtained by partially changing the operator support screen 900 illustrated in FIG. 9, and thus common portions are denoted by the same reference numerals and description thereof is omitted.

  In the shared screen display area 1923 in the connection terminal display area 1920 of the operator support screen 1900, the content indicating that the user terminal 4 is taking an image with the camera is displayed on the shared screen 1924. The data communication amount between the user terminal 4 and the operator terminal 3 is reduced by displaying only that the photographing is being performed without relaying the photographing preview 1802 displayed on the user display unit 44 at the user terminal 4. can do.

  In the auxiliary display area 1925, a list of images received from the user terminal 4 and a screen for enlarging and displaying one image selected from these are displayed. Since the operator support screen 1900 corresponds to a state in which an image has not been received from the user terminal 4, nothing is displayed in the received image list. Such a screen for displaying the image received from the user terminal may be displayed in the auxiliary display area 1925 after the operator terminal 3 receives the image from the user terminal 4.

  Returning to FIG. 16, when the user executes photographing by performing a predetermined operation on the photographing icon 1803 (step S407), the user command executing unit 454 of the user terminal 4 photographs the photographed image on the user display unit 44. It is displayed on the preview 1802.

  Next, the user command transmitting / receiving unit 453 of the user terminal 4 transmits a report that the captured image is being previewed to the relay server 23 via the user communication unit 41 as the command execution status. The command relay unit 2331 relays the received report to the operator terminal 3 (step S408).

  The operator terminal 3 that has received the report that the user terminal 4 is previewing displays the execution status of the user terminal 4 on the operator display unit 34 (step S409).

  Next, when the user terminal 4 determines to save the image previewed by the user, the user terminal 4 transmits data of the captured image to the application server 22 via the user communication unit 41, and the display information providing unit 2231 of the application server 22 Is acquired (step S410).

  Next, the user command transmission / reception unit 453 of the user terminal 4 transmits, as the command execution status, a report indicating that the transmission of the captured image has been completed to the relay server 23 via the user communication unit 41. The command relay unit 2331 relays the received report to the operator terminal 3 (step S411).

  The operator terminal 3 that has received the report that the user terminal 4 has completed the image transmission requests the data of the captured image from the application server 22, and the display information providing unit 2231 of the application server 22 transmits the request from the operator terminal 3. Is provided to the operator terminal 3 (step S412), and the operator terminal 3 displays an image corresponding to the received data on the operator display unit 34 (step S413).

  FIG. 20 is a diagram illustrating another example of a screen displayed on the operator terminal in the photographing command relay processing.

  The operator support screen 2000 illustrated in FIG. 20 is a screen obtained by partially changing the operator support screen 900 illustrated in FIG. 9, and thus common parts are denoted by the same reference numerals and description thereof is omitted.

  In the shared screen display area 2023 in the connection terminal display area 2020 of the operator support screen 2000, the content indicating that the transmission of the image captured by the user terminal 4 has been completed is displayed on the shared screen 2024.

  In the auxiliary display area 2025, a list of images received from the user terminal 4 and a screen for enlarging and displaying one image selected from these are displayed. That is, the thumbnail of the received image is displayed in the received image list 2026, and the image corresponding to the selected thumbnail among the thumbnails displayed in the received image list 2026 is enlarged and displayed in the enlarged display area 2027.

  By executing the processing as described above, it is possible to cause the user terminal 4 to execute shooting and display the shot image on the operator terminal 3.

  FIG. 21 is a sequence diagram of the image display command relay processing.

  First, the operator command generation unit 352 of the operator terminal 3 generates an image display command requesting display of an image on the user terminal 4 (Step S501). The image display command is generated when the operator input / output unit 33 acquires a predetermined operation of the operator with respect to the image display command transmission icon 9224 displayed on the operator support screen 900 of the operator terminal 3.

  Next, the operator terminal 3 displays a screen for the user to specify an image to be displayed on the operator display unit 34 (step S502).

  FIG. 22 is a diagram illustrating an example of a screen displayed on the operator terminal in the screen display command relay processing.

  When the operator performs a predetermined operation on the image display command transmission icon 9224 on the operator support screen 2000 in FIG. 20, the operator terminal 3 displays an image file selection dialog 2200 as an overlay on the operator support screen 2000. The image file selection dialog 2200 has a file name input area 2210 and a selection decision icon 2220.

  The file name input area 2210 is an area for inputting a file name of an image file requested to be displayed on the user terminal 4. The input of the file name may be performed by direct input using a keyboard or the like, or may be performed by selecting from a list of files displayed in a selectable manner. When the operator performs a predetermined operation on the selection and determination icon 2220 in a state where the file name is appropriately input in the file name input area 2210, the file name input in the file name input area 2210 is displayed on the user terminal 4. Is determined as the file name of the image file to be copied.

  Returning to FIG. 21, the operator terminal 3 transmits the image file displayed on the user terminal to the application server, and the display information providing unit 2231 of the application server 22 receives the image file (Step S503).

  FIG. 23 is a diagram illustrating another example of a screen displayed on the operator terminal in the screen display command relay processing.

  The operator support screen 2300 shown in FIG. 23 is a screen obtained by partially changing the operator support screen 2000 shown in FIG. 20, and therefore, the common parts are denoted by the same reference numerals and description thereof is omitted.

  In the auxiliary display area 2325 of the operator support screen 2300, a screen corresponding to the screen display command is displayed. The auxiliary display area 2325 has an image file display 2326 and an image transmission icon 2327. In the image file display 2326, an image corresponding to the file determined as the image file to be displayed on the user terminal 4 is displayed. By displaying in this way, the content of the image to be displayed on the user terminal 4 can be confirmed in advance. When the operator performs a predetermined operation on the image transmission icon, the operator terminal 3 transmits the file determined as the image file displayed on the user terminal 4 via the operator communication unit 31.

  Returning to FIG. 21, the operator command transmission / reception unit 353 of the operator terminal 3 transmits the image display command to the relay server 23 via the operator communication unit 31 together with the information of the image file transmitted to the application server 22. The command relay unit 2331 relays the received shooting command to the user terminal 4 together with the information of the image file (step S504).

  The user terminal 4 that has received the image display command displays a display permission dialog requesting the user to permit image display on the user display unit 44 (step S505).

  FIG. 24 is a diagram illustrating an example of a screen displayed on the user terminal in the screen display command relay processing.

  When the image display command is received while the support screen 700 in FIG. 7 is displayed, the user terminal 4 overlays the display permission dialog 2400 on the support screen 700. The display permission dialog 2400 has a display permission request 2401, a non-permission area 2402, and a permission area 2403.

  The display permission request 2401 displays information indicating to the user that the support application requests permission to display an image sent from the operator. By performing a predetermined operation on the non-permission area 2402 or the permission area 2403 via the user input / output unit 43, the display of the received image file can be prohibited or permitted.

  Returning to FIG. 21, when the user permits the display of the received image file, the user terminal 4 requests the image file to be displayed from the application server 22 via the user communication unit 41, and the display of the application server 22 is performed. The information providing unit 2231 transmits image data to the user terminal 4 in response to a request from the user terminal 4 (Step 506).

  Next, the user command execution unit 454 of the user terminal 4 displays an image corresponding to the obtained image file on the user display unit 44 (Step S507).

  FIG. 25 is a diagram illustrating another example of a screen displayed on the user terminal in the screen display command relay processing.

  The material sharing screen 2500 illustrated in FIG. 25 includes some of the elements common to the support screen 700 illustrated in FIG. 7.

  The material sharing screen 2500 has a screen title 2501 and a shared material display area 2503. The screen title 2501 indicates the title of the currently displayed material sharing screen 2500. The shared material display area 2503 is an area where an image of an image file as a material sent from the operator is displayed.

  By executing the processing as described above, an image corresponding to the image file specified by the operator terminal 3 can be displayed on the user terminal 4.

  FIG. 26 is a sequence diagram of the drawing command relay process.

  First, the operator command generation unit 352 of the operator terminal 3 generates a drawing command requesting drawing on the user display unit 44 of the user terminal 4 (Step S601). The drawing command is generated when the operator input / output unit 33 acquires a predetermined operation of the operator with respect to the drawing command transmission icon 9225 displayed on the operator support screen 900 of the operator terminal 3.

  The operator command transmitting / receiving unit 353 of the operator terminal 3 transmits the generated drawing command to the relay server 23 via the operator communication unit 31. The command relay unit 2331 of the relay server 23 transmits the received drawing command to the user terminal 4. It relays (step S602).

  The user terminal 4 that has received the drawing command requests the user to permit drawing (step S603).

  When the drawing is permitted by the user, the user command transmitting / receiving unit 453 of the user terminal 4 transmits the drawing permission to the relay server 23 via the user communication unit 41, and the relay server 23 relays the drawing to the operator terminal 3. (Step S604).

  The operator terminal 3 that has received the drawing permission receives the operation of the operator executing the drawing through the operator input / output unit 33 (step S605). The operation of the operator executing the drawing is, for example, dragging the mouse in the area of the shared screen 924 of the operator support screen 900.

  The operator command transmitting / receiving unit 353 of the operator terminal 3 that has received the operation of the operator executing the drawing transmits the drawing command corresponding to the received operation of the operator to the relay server 23 via the operator communication unit 31, and the relay server 23 The command relay unit 2331 relays the received drawing command to the user terminal 4 (step S606).

  The user command execution unit 454 of the user terminal 4 that has received the drawing command executes drawing on the user display unit 44 corresponding to the received command.

  In executing the drawing command relay process, the command relay unit 2331 of the relay server 23 determines the number of pixels in the vertical direction and the horizontal direction of the display area used by the application for displaying display information in the user display unit 44 transmitted by the user terminal 4. , May be relayed from the user terminal 4 to the operator terminal 3. The operator processing unit 35 of the operator terminal 3 that has received the number of pixels of the display area of the user display unit 44 changes the predetermined coordinates of the shared screen 924 on the operator display unit 34 to the drawing target coordinates by operating the operator on the operator input / output unit 33. In the case where is designated as, includes a command for performing conversion based on the ratio of the received number of pixels and the number of pixels in the vertical direction and the horizontal direction of the shared screen 924 to obtain conversion coordinates, and requesting drawing to the conversion coordinates. A command is generated and transmitted to the relay server 23 via the operator communication unit 31.

That is, when the number of pixels in the vertical direction of the display area in the user terminal 4 is 1280 pixels and the number of pixels in the vertical direction of the display area in the operator terminal 3 is 2560 pixels, the pixel ratio β _{H in the} vertical direction is 25605601280. = 2. When the number of pixels in the horizontal direction of the display area in the user terminal 4 is 720 pixels and the number of pixels in the vertical direction of the display area in the operator terminal 3 is 1440 pixels, the pixel ratio β _{W in the} vertical direction is 1440 ÷ 720. = 2.

The operator processing unit 35, based on the pixel ratio beta _W and beta _H, drawn coordinates (X, Y) of the _{(X / β W, Y /} β H), included in the drawing request command transmitted.

  By operating in this manner, the operator processing unit 35 can operate the operator terminal 3 even when the screen resolution of the user display unit 44 does not match the number of pixels occupied by the area of the shared screen 924 in the operator terminal 3. A command for requesting the user terminal 4 to perform the same drawing as the drawing in can be transmitted to the relay server 23.

  By executing the processing as described above, the drawing performed by the operator on the operator terminal 3 can be displayed on the user terminal 4.

  It will be understood by those skilled in the art that various changes, substitutions and modifications can be made without departing from the spirit and scope of the invention.

DESCRIPTION OF SYMBOLS 1 Remote support system 2 Server 21 IVR server 2131 User information storage part 2132 Automatic voice response part 2133 Application execution request part 2134 User information transmission part 2135 Voice relay part 22 Application server 2231 Display information provision part 2232 Connection request reception part 2233 Connection identifier transmission Unit 23 relay server 2331 command relay unit 3 operator terminal 4 user terminal

Claims (12)

A user information storage unit that stores user information obtained by performing voice recognition processing on voice received from the user terminal;
An automatic voice response unit that generates a response voice based on the user information and transmits the generated voice to the user terminal;
An application execution request unit that requests the user terminal to execute an application based on the user information,
A connection request receiving unit that receives an operator request for requesting a voice call with an operator, transmitted from the user terminal in response to a predetermined operation of the user on the application,
A connection identifier, generated in response to the reception of the operator request, a connection identifier for identifying a connection, the user terminal, and a connection identifier transmission unit transmitting the operator terminal used by the operator.
A command, which is an instruction for causing the application to execute a predetermined process, is bidirectionally relayed between the user terminal connected by specifying the connection identifier and the operator terminal connected by specifying the connection identifier. Command relay unit to
Server comprising: The user information includes at least a user identifier for identifying a user,
The operator request includes at least the user identifier,
The server according to claim 1, further comprising: a user information transmitting unit configured to transmit the user information including the user identifier included in the operator request to the operator terminal.   The server according to claim 1, further comprising a display information providing unit configured to provide display information for the application to display a screen on the user terminal to the user terminal and the operator terminal. The command relay unit relays the command including an instruction for requesting a transition of a screen displayed by the application on the user terminal from the user terminal to the operator terminal,
The said display information provision part provides the said display information for displaying the screen of a transition destination to the said operator terminal which received the said command according to the request from the said operator terminal which received the said command. Server as described in. The command relay unit relays the command including an instruction for requesting shooting at the user terminal from the operator terminal to the user terminal,
The display information providing unit, from the user terminal that has received the command, and acquires the display information that is an image captured by the user terminal, in response to a request from the operator terminal that transmitted the command, The server according to claim 3, wherein the acquired display information is provided to the operator terminal that has transmitted the command. The command relay unit relays the command including an instruction requesting display of an image corresponding to a specified file on the user terminal, from the operator terminal to the user terminal,
The display information providing unit, in response to a request from the user terminal that has received the command, the display information that is the file obtained from the operator terminal that has transmitted the command, the user terminal that has received the command The server according to any one of claims 3 to 5, wherein the server is provided.   The server according to any one of claims 1 to 6, wherein the command relay unit relays a report of an execution status of the predetermined process executed in response to the relayed command. A user information storing step of storing user information obtained by performing voice recognition processing on voice received from the user terminal;
An automatic voice response step of generating a response voice based on the user information and transmitting the response voice to the user terminal;
An application execution requesting step of requesting the user terminal to execute an application based on the user information,
A connection request receiving step of receiving an operator request for requesting a voice call with an operator, transmitted from the user terminal in response to a predetermined operation of the user on the application,
A connection identifier transmission step of transmitting a connection identifier generated in response to the reception of the operator request to the user terminal and an operator terminal used by the operator,
A command, which is an instruction for causing the application to execute a predetermined process, is bidirectionally relayed between the user terminal connected by specifying the connection identifier and the operator terminal connected by specifying the connection identifier. Command relaying step,
And a server control method. A user information storing step of storing user information obtained by performing voice recognition processing on a voice received from the user terminal;
An automatic voice response step of generating a response voice based on the user information and transmitting the generated voice to the user terminal;
An application execution requesting step of requesting the user terminal to execute an application based on the user information,
A connection request receiving step of receiving an operator request for requesting a voice call with an operator, transmitted from the user terminal in response to a predetermined operation of the user on the application,
The connection identifier generated in response to the reception of the operator request, the user terminal, and a connection identifier transmission step of transmitting to an operator terminal used by the operator,
A command, which is an instruction for causing the application to execute a predetermined process, is bidirectionally relayed between the user terminal connected by specifying the connection identifier and the operator terminal connected by specifying the connection identifier. Command relaying step,
Server control program that causes the server to execute An operator terminal used by an operator, a remote support system having a server connectable to the operator terminal,
The server is
A user information storage unit that stores user information obtained by performing voice recognition processing on voice received from the user terminal;
An automatic voice response unit that generates a response voice based on the user information and transmits the generated voice to the user terminal;
An application execution request unit that requests the user terminal to execute an application based on the user information,
A connection request receiving unit that receives an operator request for requesting a voice call with the operator, transmitted from the user terminal in response to a predetermined operation of the user on the application,
The connection identifier generated in response to the reception of the operator request, the user terminal, and a connection identifier transmission unit that transmits to the operator terminal,
A command, which is an instruction for causing the application to execute a predetermined process, is bidirectionally relayed between the user terminal connected by specifying the connection identifier and the operator terminal connected by specifying the connection identifier. Command relay unit to
A remote support system comprising: The command relay unit, in the user terminal, the number of pixels in the vertical direction and the horizontal direction of a display area used by the application for display of display information, and a display magnification when the application displays the display information on the user terminal. Is relayed from the user terminal to the operator terminal,
The operator terminal that has received the number of pixels and the display magnification displays the display information at the received display magnification in the shared screen area having the number of pixels in the vertical and horizontal directions according to the number of pixels. ,
The command relay unit relays the command including an instruction for requesting a predetermined amount of scrolling of a screen displayed on the user terminal by the application from the user terminal to the operator terminal,
The operator terminal that has received the command converts the predetermined amount included in the command based on a ratio between the number of pixels relayed and the number of pixels in the vertical and horizontal directions of the shared screen area, The remote support system according to claim 10, wherein the remote support system executes a command. The command relay unit relays the number of pixels in a vertical direction and a horizontal direction of a display area used by the application for displaying display information in the user terminal, from the user terminal to the operator terminal,
The operator terminal receiving the number of pixels, the coordinates specified as drawing target coordinates in the shared screen area having the number of pixels in the vertical and horizontal directions according to the number of pixels, the number of pixels relayed, and the shared Including the transformation coordinates converted based on the ratio of the number of pixels in the vertical direction and the horizontal direction of the screen area, generating the command including a command requesting drawing to the transformation coordinates, transmitting the command to the command relay unit,
The remote support system according to claim 10, wherein the command relay unit relays the command from the operator terminal to the user terminal.