JP2017079385A - Intercommunication system, intercommunication method, and intercommunication program using device capable of transmission of video and voice - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an intercommunication system including a spectacles-type communication device capable of improving an interactive property, and a method.SOLUTION: An intercommunication system 1 comprises: a spectacles-type communication device 2a that is worn by a wearer and generates a visual field video signal similar to a visual field of the wearer, outputs a voice signal of the wearer, and inputs and outputs a voice signal from a remote user terminal; and a management server 3 that is communicatively connected with the spectacles-type communication device 2a and remote user terminals 4a via at least one remote user terminal 4a and a network 5. The management server 3 establishes intercommunication between the spectacles-type communication device 2a and the remote user terminals 4a.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a mutual communication system and a mutual communication method in which a field information terminal, a remote information terminal, and a management server are connected via a network, and communication between the field information terminal and the remote information terminal is achieved. Related to the mutual communication program, and in particular, a store clerk in a real store or a work site wearing a device capable of transmitting on-site video and audio, and a customer or work site outside the store via a network. In particular, the present invention relates to a mutual communication system, a mutual communication method, and a mutual communication program using a wearable small glasses-type communication device.

Currently, electronic commerce that performs two-way communication over the Internet between a net shop (virtual store) that sells various products over the Internet and a user terminal on the customer side that applies for purchase of products is at home, etc. You can purchase the desired product freely from any user terminal at any time, and the payment for the purchased product can be automatically withdrawn from the bank account, so shopping equivalent to a virtual shopping street that is a collection of virtual stores Malls are becoming increasingly popular.
In such a net shop, for example, a sales system has been proposed in which a customer who is not in the product sales floor receives advice from a staff member of a virtual store and purchases a product while checking the image of the product.

  However, the above-mentioned sales system cannot be applied to transactions of products that cannot be handled as the same product, which is a ready-made product and a mass-produced product (for example, fresh food products and handmade products). There are challenges.

  Thus, for example, an online sales system described in Patent Document 1 has been proposed. In Patent Document 1, an actual shop operation means of an online shop actually sells a product on the actual shop side with respect to information including a purchase intention transmitted from a user terminal owned by a user permitted to enter the shop. And an intention to purchase an image for packaging and sealing the product acquired from the store terminal when receiving information including an intention to purchase from the user for the actual product to be sold. The information is transmitted to the user terminal that has transmitted the information including.

JP 2005-202665 A

However, in the configuration of Patent Document 1, the actual store side shoots the state of packaging the product with a photographing device, and only transmits the product packaging image from the store terminal to the user terminal via the network.
That is, the product packaging image is transmitted unilaterally from the actual store side, which is not sufficient from the viewpoint of bidirectional communication between the actual store side and the user. For example, when a user wants to confirm various products displayed in a real store, the user must make a determination based on images displayed in a virtual store (net shop), and no consideration is given to interactivity.
Accordingly, an object of the present invention is that a store clerk (wearer) of a real store wearing a small eyeglass-type communication device or a worker at a work site transmits video and audio within his / her field of view to a remote user terminal. In addition, an object of the present invention is to provide an intercommunication system having an eyeglass-type communication device that can improve interactivity by executing a real-time change of visual field in response to a request from a user.

An intercommunication system according to the present invention is an intercommunication system in which a field information terminal, a remote information terminal, and a management server are connected via a network and communication between the field information terminal and the remote information terminal is achieved. There,
The field information terminal is equipped with a field information input / output unit that is worn by a specific wearer and inputs / outputs visual field image information similar to the field of view of the wearer and a voice signal of the wearer.
The remote information terminal comprises field information output means for outputting information transmitted from the field information terminal, and voice input / output means for inputting and outputting at least the voice of the remote information terminal user,
The management server is communicably connected to the field information terminal and the remote information terminal via the network,
The management server has a function of establishing mutual communication between the site information terminal and the remote information terminal via the management server.

Furthermore, in the mutual communication system of the present invention, the field information terminal is disposed at the site, and the field information input / output means of the field information terminal is a glasses-type communication device worn by a wearer at the field,
When the user of the remote information terminal accesses the site information terminal at the site from a remote location and performs shopping in the actual store that is the site, or when communicating with the work site,
Reservation registration means for managing a reservation of mutual communication date and time between the spectacles type communication device as the field information input / output means and the remote information terminal, and date and time for mutual communication enabling the mutual communication By having password issuing means for issuing a limited password, a reservation for mutual communication between the glasses-type communication device and the remote information terminal is established in advance.

Furthermore, in the mutual communication system provided with the glasses-type communication device of the present invention,
The management server includes an authentication processing unit that authenticates the date-and-time-limited password that identifies the glasses-type communication device, and performs mutual communication between the glasses-type communication device and the remote information terminal according to an authentication result by the authentication processing unit. It is characterized by establishing.

Furthermore, in the mutual communication system provided with the glasses-type communication device of the present invention, a plurality of glasses-type communication devices are installed at the site,
The reservation registration unit of the management server is installed in the actual store when reserving the date and time of mutual communication between the spectacles type communication device as the field information input / output unit and the remote information terminal. One eyeglass-type communication device among a plurality of eyeglass-type communication devices is reserved.

Furthermore, in the mutual communication system provided with the glasses-type communication device of the present invention,
The management server includes a database of performance data storing the visual field video signal and the audio signal of the wearer and the audio signal from the remote information terminal during a mutual communication period between the glasses-type communication device and the remote information terminal. It is characterized by having.

Furthermore, in the mutual communication system provided with the glasses-type communication device of the present invention,
A schedule management database is provided for storing a plurality of pieces of reservation information on the date and time of mutual communication between the glasses-type communication device and the remote information terminal registered by the reservation registration unit.

An intercommunication method according to the present invention includes an intercommunication system in which a field information terminal, a remote information terminal, and a management server are connected via a network, and communication between the field information terminal and the remote information terminal is achieved. The mutual communication method used,
The on-site information terminal is attached to a specific wearer, and the on-site information input / output means inputs / outputs visual field video information similar to the wearer's field of view and the audio signal of the wearer,
The remote information terminal outputs information transmitted from the field information terminal by the field information input / output means, and inputs / outputs at least the voice of the remote information terminal user by the voice input / output means,
The management server establishes mutual communication between the site information terminal and the remote information terminal via the management server.

Furthermore, in the mutual communication method of the present invention,
The on-site information terminal is arranged at the site, and the on-site information input / output means of the on-site information terminal is a glasses-type communication device worn by a wearer at the site,
When the user of the remote information terminal accesses the on-site information terminal arranged at the site from a remote place and takes communication,
The management server manages the reservation of the date and time of mutual communication between the spectacles-type communication device that is the field information input / output means and the remote information terminal by the reservation registration means, and the date and time limitation that enables the mutual communication By having password issuing means for issuing a password, a reservation for mutual communication between the glasses-type communication device and the remote information terminal is established in advance.

Furthermore, in the mutual communication method of the present invention, when a plurality of eyeglass-type communication devices are installed at the site,
The reservation registration unit of the management server is installed in the actual store when reserving the date and time of mutual communication between the spectacles type communication device as the field information input / output unit and the remote information terminal. One eyeglass communication device is reserved among a plurality of eyeglass communication devices.

An intercommunication system comprising a communication device on the actual store side, a user terminal on the user side, a management server that manages data between the communication device and the user terminal, and a network that connects the communication device to the user terminal and the management server Because
The communication device has a communication control unit,
The communication control unit
Including an input unit, a display unit, an output unit, a communication interface unit, an image processing unit, a voice processing unit, a synthesis processing unit, an operation recognition unit,
The image processing unit receives a field of view video signal and data including a line-of-sight direction input from the camera of the input unit, obtains a field of view video signal corresponding to the field of view of the wearer wearing the communication device, and the combination processing unit To supply
The audio processing unit supplies an audio signal input from a microphone of the input unit to the synthesis processing unit, and supplies an audio signal transmitted from the user terminal to a speaker of the output device,
The motion recognition unit receives the input data, calculates data including the line-of-sight direction based on the data, and supplies the data to the synthesis processing unit.
The synthesis processing unit supplies the visual field video signal and the audio signal to the communication interface unit of the display unit and the output unit,
The communication interface unit transmits the visual field video signal and the audio signal to a corresponding user terminal via the network,
The communication control unit further includes one or more of a monitor display processing unit, a communication quality display processing unit, and a bit rate / frame rate adjustment function processing unit that process a display function on the monitor of the display unit,
The monitor display processing unit
Overlaying any one or more of a viewing mark, a reserved time, and a horizontal confirmation grid on the video display obtained by photographing the visual field video signal and the audio signal generated by the synthesis processing unit. Alternatively, a hidden menu screen is displayed by operating the communication device,
The quality display function processing unit displays, as an indicator, quality information indicating a guideline of a communication speed calculated from a result of periodic signal exchange between the communication device and the management server. Similarly, the user terminal and the management server Display the quality information indicating the standard of the communication speed calculated from the results of regular signal exchanges on the display unit on the user terminal side,
The bit rate / frame rate adjustment function processing unit changes the number of frames and / or the number of pixels of video communicated with the user terminal.

A mutual store comprising a communication device on the actual store / site side, a user terminal on the user side, a management server for managing data between the communication device and the user terminal, and a network for connecting the communication device to the user terminal and the management server The mutual communication method in the communication system is
The communication device has a communication control unit,
The communication control unit
Receiving data including the visual field video signal and the line-of-sight direction input from the camera of the input unit, obtaining a visual field video signal corresponding to the field of view of the wearer wearing the communication device, and supplying the visual field video signal to the synthesis processing unit,
Supplying an audio signal input from the microphone of the input unit to the synthesis processing unit, and supplying an audio signal transmitted from the user terminal to a speaker of the output unit;
Receiving posture data of the input unit, calculating data including the line-of-sight direction based on the posture data and supplying the data to the synthesis processing unit;
Transmitting the visual field video signal and the audio signal to a corresponding user terminal via a speaker of the output unit, the communication interface unit, and a network;
Further, one or more of a viewing mark, a reservation remaining time, and a horizontal confirmation grid are overlaid on the video display obtained by photographing the visual field video signal and the audio signal generated by the synthesis processing unit. Display or display a hidden menu screen by operating the communication device,
Displaying quality information indicating a guideline of communication speed calculated from the result of regular signal exchange between the communication device and the management server as an indicator, and regular signal exchange between the user terminal and the management server Display quality information indicating a guideline of communication speed calculated from the result of the above on the display unit of the user terminal,
The number of frames and / or the number of pixels of video to be communicated with the user terminal is changed.

According to the present invention, a wearer in an actual store wearing a small glasses-type communication device or a worker at a work site or the like transmits video and audio within his / her field of view to a remote user terminal and from the user. It is possible to provide an intercommunication system having an eyeglass-type communication device that can improve interactivity by executing a visual field change or the like in real time in response to the request.
For example, if a wearer in a real store picks up and explains a product according to a request from a user at a remote location via a glasses-type communication device, whether the user at the remote location is in the real store You can act with a sense of realism and realize a virtual visit to the store that is even more realistic.
In addition, since the video (image within the field of view of the worker) of the worker who works at the construction site such as a building site is transmitted to the terminal of the remote construction company via the glasses-type communication device. It becomes possible to easily check the work procedure and the progress of the construction period.
Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

1 is an overall schematic configuration diagram of an intercommunication system according to an embodiment of the present invention. It is an external appearance perspective view of the spectacles type communication apparatus shown in FIG. It is a functional block diagram of the communication control part which comprises the spectacles type communication apparatus shown in FIG. It is another external appearance perspective view of the eyeglass type communication apparatus shown in FIG. It is a functional block diagram of the communication control part which comprises the spectacles type communication apparatus shown in FIG. It is a block diagram of the bone-conduction microphone shown in FIG. It is a block diagram of the bone conduction speaker shown in FIG. It is a functional block diagram of the real shop terminal shown in FIG. It is a functional block diagram of the management server shown in FIG. It is a schematic sequence diagram of a mutual communication system. It is a flowchart which shows the processing flow of the management server shown in FIG. It is a flowchart which shows the processing flow of the real shop terminal shown in FIG. 8, and a communication control part. It is a flowchart which shows the processing flow of the user terminal shown in FIG. It is a display example of a reservation screen by the management server. It is a display example of a reservation screen by the management server. It is a display example of a reservation screen by the management server. It is an example of a screen display at the time of reservation registration. It is an example of a display screen at the time of reservation registration completion and date / time limited password transmission. It is an example of a screen display at the time of reservation search. It is an example of a screen display at the time of reservation search. It is an example of a screen display at the time of reservation change. It is an example of a screen display at the time of reservation change completion. It is an example of a screen display at the time of reservation deletion. It is an example of a screen display at the time of reservation deletion. It is an example of a screen display at the time of completion of reservation deletion. It is an example of a screen display of a user terminal at the time of mutual communication. It is a whole schematic block diagram of the mutual communication system which concerns on the other Example of this invention. It is an example of a screen display of a user terminal at the time of mutual communication. It is a whole schematic block diagram of the mutual communication system which concerns on the other Example of this invention. It is a figure which shows an example of the monitor screen of a display part. It is a flowchart which shows the processing flow of the communication control part shown in FIG.

  In this specification, the wearer who wears the eyeglass-type communication device at the actual store and the worker who wears the eyeglass-type communication device at the work site may be collectively referred to as a wearer. Further, in this specification, “actual store” means not only a store that handles merchandise to be traded but also an exhibition hall that exhibits merchandise to be traded such as a showroom or a product that is difficult to handle as the same (for example, (Including used cars and used properties) and auction locations. Furthermore, it may be referred to as “site” including a work site such as a case where an operation is explained while referring to an actual screen such as a call center.

In the following embodiment, a spectacle-type communication device will be described as an example of a small communication device arranged in an actual store, but a small communication device including a small camera and an audio microphone may be used. good.
Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall schematic configuration diagram of an intercommunication system according to an embodiment of the present invention.
As shown in FIG. 1, the mutual communication system 1 includes a real store 2, a management server 3, a user terminal 4 a that is a personal computer (hereinafter referred to as a PC) owned by a user A, and a user that is a tablet owned by a user B. It is comprised from the terminal 4b, the user terminal 4c which is a smart phone (mobile phone) which the user C owns, and the network 5 which connects these so that two-way communication is possible.
The actual store 2 includes an eyeglass-type communication device 2a, a router 2b, and an actual store terminal 2c that is a PC.
The glasses-type communication device 2a and the actual store terminal 2c are connected to the network 5 via the router 2b.
For example, Wi-Fi (registered trademark of Softbank BB Corp.) or Bluetooth (Bluetooth SIG, Inc .: registered trademark of Bluetooth SIG, Inc.) is connected between the glasses-type communication device 2a and the router 2b. ing.
Note that the number of user terminals connected to the network 5 is not limited to three as shown in FIG. 1, and a plurality of user terminals can be connected. In addition, all user terminals may be smartphones or tablets. Moreover, the real store terminal 2c installed in the real store 2 may be a smartphone or a tablet instead of the PC.

(Glasses type communication device)
2 is an external perspective view of the eyeglass-type communication device shown in FIG.
As shown in FIG. 2, a small spectacles-type communication device 2 a that can be worn and carried by a wearer includes a spectacle body 24 and a communication control unit 20. The spectacle body 24 mainly includes a left temple 24a, a right temple 24b, a lens frame 24c, and a lens or glass 24d. The communication control unit 20 is detachably attached to the right temple 24b. FIG. 2 shows an example in which the right eye is generally effective, and thus the example in which the communication control unit 20 is attached to the right temple 24b is shown, but of course, the communication control unit 20 may be attached to the left temple 24a. .

The communication control unit 20 includes an imaging unit 21, a voice input / output unit 22, and a support unit 23 that supports them.
The length of the support unit 23 can be adjusted, and for each store clerk wearing the glasses-type communication device 2a, the gap between the lens or glass 24d and the imaging unit 21 and the relative relationship between the lens or glass 24d and the imaging unit 21 are determined. Positioning is possible.
The imaging unit 21 includes a camera 21a on the front surface (surface opposite to the lens or glass 24d) and an attitude sensor 21b on the side surface.
The imaging unit 21 includes a thin small display 21c on the rear surface (the surface facing the lens or glass 24d).
Here, for example, a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display is used as the small display 21c. As the camera 21a, for example, a CCD or a moving image full HD camera is used.
The voice input / output unit 22 includes a microphone 22a and a speaker 22b on the surface facing the right temple 24b. The voice input / output unit 22 includes a volume adjustment button, a menu selection button, and a determination button, which are not shown.

In FIG. 3, the functional block diagram of the communication control part 20 which comprises the spectacles type communication apparatus 2a is shown.
The communication control unit 20 includes an input I / F 201, an output I / F 202, a video acquisition unit 203, an audio acquisition unit 204, an image processing unit 205, an audio processing unit 206, a synthesis processing unit 207, an operation recognition unit 208, and a communication I / F 209. , The GPS device 211, and an internal bus 210 that connects them to each other.
Here, the image processing unit 205, the sound processing unit 206, the synthesis processing unit 207, the motion recognition unit 208, and the GPS device 211 temporarily store, for example, a ROM that stores various programs (not shown) and data of calculation processes. This is realized by a memory such as a RAM and a processor such as a CPU that reads and executes a program stored in the ROM.

The input I / F 201 outputs a video signal input from the camera 21 a to the video acquisition unit 203 via the internal bus 210 and also transmits a wearer's voice signal (conversation) input from the microphone 22 a to the internal bus 210. To the voice acquisition unit 204.
Further, the input I / F 201 outputs posture data (data indicating the posture of the glasses-type communication device 2a) input from the posture sensor 21b to the motion recognition unit 208 via the internal bus 210.

The video acquisition unit 203 transfers the acquired video signal to the image processing unit 205 via the internal bus 210.
In addition, the sound acquisition unit 204 transfers the acquired sound signal to the sound processing unit 206 via the internal bus 210.
The motion recognition unit 208 calculates data including at least the line-of-sight direction of the store clerk wearing the glasses-type communication device 2a based on the input posture data, and outputs the data to the image processing unit 205 via the internal bus 210.
The image processing unit 205 obtains a visual field video signal corresponding to the field of view (in the visual field) of the store clerk wearing the glasses-type communication device 2a based on the input video signal and data including the line-of-sight direction, and via the internal bus 210. The data is output to the composition processing unit 207.
The audio processing unit 206 removes noise from the input audio signal and outputs the audio signal after noise removal to the synthesis processing unit 207 via the internal bus 210. The synthesis processing unit 207 superimposes or synchronizes the input audio signal after noise removal and the visual field video signal, and executes processing such as compression for communication, and the router 2b via the internal bus 210 and the communication I / F 209. Send to.
Thus, the audio signal and the visual field video signal processed by the synthesis processing unit 207 are transmitted to the corresponding user terminal via the router 2b and the network 5. Also, the voice processing unit 206 removes noise from the user's voice signal (conversation) received from the user terminal and outputs the noise signal after noise removal to the speaker 22b via the internal bus 210 and the output I / F 202. To do.
The GPS device 211 acquires its position information by simultaneously acquiring distances from four or more GPS satellites. In other words, the GPS device 211 measures the glasses-type communication device 2a and outputs it to the router 2b via the internal bus 210 and the communication I / F 209 as necessary. This makes it possible to know the position information of the store clerk who wears the glasses-type communication device 2a. In particular, when one eyeglass-type communication device 2a is shared by a plurality of store employees, the glasses-type communication device 2a is currently used. It is possible to easily grasp where is used or stored.
Here, the communication control unit 20 may further include a monitor display processing unit, a communication quality display function control unit, and a bit rate / frame rate adjustment function processing unit. Examples including such a configuration will be described later.

FIG. 4 is an external perspective view of a modified example of the glasses-type communication device.
As shown in FIG. 4, the glasses-type communication device 2a ′ includes a glasses body 24, a camera 21a ′, a microphone 22a ′, a speaker 22b ′, and a communication control unit 20 ′. The camera 21a ′ is provided at a connecting portion that connects the left and right lens frames 24c. For example, a bone conduction speaker is used as the speaker 22b ′ provided in the left temple 24a. For example, a bone conduction microphone is used as the microphone 22a ′ provided in the right temple 24b.

FIG. 5 shows a functional block diagram of the communication control unit 20 ′ constituting the glasses-type communication device 2a ′.
The communication control unit 20 ′ includes an input I / F 201, an output I / F 202, a video acquisition unit 203, an audio acquisition unit 204, an image processing unit 205, an audio processing unit 206, a synthesis processing unit 207, a communication I / F 209, and these. An internal bus 210 is connected to each other.
Here, the image processing unit 205, the sound processing unit 206, and the synthesis processing unit 207 include, for example, a ROM that stores various programs (not shown), a memory such as a RAM that temporarily stores calculation process data, and the like, and the ROM This is realized by a processor such as a CPU that reads and executes the program stored in the program.

  The input I / F 201 outputs the video signal input from the camera 21a ′ to the video acquisition unit 203 via the internal bus 210, and also transmits the audio signal (conversation) of the wearer input from the microphone 22a ′ to the internal The sound is output to the sound acquisition unit 204 via the bus 210.

The video acquisition unit 203 transfers the acquired video signal to the image processing unit 205 via the internal bus 210.
In addition, the sound acquisition unit 204 transfers the acquired sound signal to the sound processing unit 206 via the internal bus 210.
The image processing unit 205 outputs the input video signal to the synthesis processing unit 207 via the internal bus 210 as a visual field video signal corresponding to the field of view (inside the visual field) of the store clerk wearing the glasses-type communication device 2a.
The audio processing unit 206 removes noise from the input audio signal and outputs the audio signal after noise removal to the synthesis processing unit 207 via the internal bus 210.
The synthesis processing unit 207 superimposes or synchronizes the input audio signal after noise removal and the visual field video signal, and executes processing such as compression for communication, and the router 2b via the internal bus 210 and the communication I / F 209. Send to.
Thus, the audio signal and the visual field video signal processed by the synthesis processing unit 207 are transmitted to the corresponding user terminal via the router 2b and the network 5. Also, the voice processing unit 206 removes noise from the user's voice signal (conversation) received from the user terminal, and sends the noise signal after the noise removal to the speaker 22b ′ via the internal bus 210 and the output I / F 202. Output.

Hereinafter, the bone conduction microphone and the bone conduction speaker used as the microphone 22a ′ and the speaker 22b ′ will be described.
FIG. 6 is a configuration diagram of the bone conduction microphone 22a ′.
As shown in FIG. 6, the bone conduction microphone 22 a ′ includes a vibration sensor 221, an amplifier 222, and an A / D converter 223.
The vibration sensor 221 acquires the vibration of the vocal cords that transmits the bone and outputs it to the amplifier 222.
The A / D converter 223 outputs the audio signal of the store clerk wearing the glasses-type communication device 2a ′ amplified by the amplifier 222 to the input I / F 201 constituting the communication control unit 20 ′.
FIG. 7 shows a configuration diagram of the bone conduction speaker 22b ′.
The bone conduction speaker 22 b ′ includes a bone conduction speaker 224, an amplifier 225, and a D / A converter 226.
The D / A converter 226 converts the voice signal of the user owning the user terminal input from the output I / F 202 constituting the communication control unit 20 ′ into an analog signal and outputs the analog signal to the amplifier 225. The user's voice signal amplified by the amplifier 225 is output to the bone conduction speaker 224.
Here, the bone conduction speaker 224 includes, for example, a vibration plate, a magnet, a yoke, and a voice coil, vibrates the vibration plate in accordance with an audio signal, and transmits the vibration to the bone.

(Actual store terminal)
FIG. 8 is a functional block diagram of the actual store terminal 2c.
As shown in FIG. 8, the actual store terminal 2c includes an input unit 231, a display unit 232, an input I / F 233, an output I / F 234, a calculation unit 235, a login authentication unit 236, a communication I / F 239, and these. An internal bus 240 to be connected is provided.
The actual store terminal 2c includes a battery unit 238 including a power source and a battery.
Here, the calculation unit 235 and the login authentication unit 236 read out, for example, a ROM that stores various programs (not shown), a memory such as a RAM that temporarily stores calculation process data, and the like, and a program stored in the ROM. This is realized by a processor such as a CPU to be executed.

The input unit 231 includes, for example, a keyboard and a mouse, and outputs input information such as a user name, a password, and other parameters input when logging into the actual store terminal 2 c to the input I / F 233.
The calculation unit 235 reads and executes a desired application program from the ROM (not shown) for the input information input from the input unit 231, and displays the calculation result via the internal bus 240 and the output I / F 234. The function to output to.
The computing unit 235 also has a function of accessing a management server 3 described later. The login authentication unit 236 determines whether or not the user name and password input from the input unit 231 are registered in advance, and if the login authentication result is “OK”, the actual store terminal by the subsequent user Allow the use of 2c.
On the other hand, when the result of the login authentication is “non-”, the subsequent access to the actual store terminal 2c by the user is prohibited.
The communication I / F 239 transmits the input information input from the input unit 231 and the calculation result by the calculation unit 235 to the router 2b.

  The functional block configurations of the user terminal 4a, the user terminal 4b, and the user terminal 4c are also the same as the configuration of the actual store terminal 2c shown in FIG.

(Management server)
FIG. 9 is a functional block diagram of the management server 3.
As shown in FIG. 9, the management server 3 includes an input unit 31, a display unit 32, an input I / F 33, an output I / F 34, a reservation registration / change unit 35, a reservation search unit 36, a reservation deletion unit 37, and a password issue unit. 38, an authentication processing unit 39, a schedule management DB (database) 40, a password management DB 41, a performance data DB 42, a communication I / F 43, and an internal bus 44 for interconnecting them.
Here, the reservation registration / change unit 35, the reservation search unit 36, the reservation deletion unit 37, the password issuance unit 38, and the authentication processing unit 39 temporarily store, for example, a ROM for storing various programs (not shown) and data of calculation processes. This is realized by a memory such as a RAM for storing the memory and a processor such as a CPU for reading and executing the program stored in the ROM.

The schedule management DB 40 stores a use schedule of the mutual communication system 1 to be described later in, for example, a calendar format and stores reservation information.
The password management DB stores a date and time limited password for executing the mutual communication issued when the use reservation registration of the mutual communication system 1 from the store terminal 2c described later is possible.
Here, the stored date and time limited password is stored in association with each actual store using the mutual communication system 1.
The date / time limited password is, for example, a date / time limited password registered for reservation, and even if the date / time limited password is transmitted from the store terminal 2c after the reservation registration date / time has elapsed, it is not accepted. In other words, such a date-only password is invalidated when the reservation registration date has passed. Further, in the performance data DB 42, a voice signal and a visual field video signal of a store clerk wearing the above-described glasses-type communication device 2a and a user's voice signal received from the user terminal, which are transmitted from the glasses-type communication device 2a described later. An audio signal is stored as performance data.

The reservation registration / change unit 35 stores or already stores reservation registration information in the schedule management DB 40 via the internal bus 44 in response to reservation registration and reservation change from the actual store terminal 2c received via the communication I / F 43. Change and update the registered reservation information.
The reservation search unit 36 accesses the schedule management DB 40 via the internal bus 44 and searches for specified reservation information.
The reservation deletion unit 37 accesses the schedule management DB 40 via the internal bus 44 and deletes the specified reservation information.
The password issuing unit 38 transmits the above-mentioned date and time limited password via the internal bus 44, the communication I / F 43, and the network 5 to the actual store terminal 2 c where the use reservation of the mutual communication system 1 is registered.
The authentication processing unit 39 executes authentication processing of the date-only password from the actual store terminal 2c received via the communication I / F 43.

Next, the overall operation of the mutual communication system 1 will be described. In addition, below, the case where the user A who owns the user terminal 4a and the real shop 2 use the mutual communication system 1 as an example is demonstrated.
FIG. 10 is a schematic sequence diagram of the mutual communication system 1.
First, as a preparation stage, a time for relaying using the mutual communication system 1 between the actual store 2 and the user A who owns the user terminal 4a is determined in advance (step S101). If the glasses-type communication device 2a in the actual store 2 is empty, it can be used or reserved immediately.
The actual store terminal 2c in the actual store 2 transmits a reservation request for the use date and time of the glasses-type communication device 2a, that is, the use date and time of the mutual communication system 1, to the management server via the router 2b and the network 5. Then, the management screen of the management server 3 is accessed and a desired use date and time is input (step S102). When the input use date and time can be registered, the management server 3 transmits the password “issue date and time” via the communication I / F 43 and the network 5 from the password issuing unit 38 to the actual store terminal 2c. “Limited password” is transmitted (step S103).

Next, the actual shop terminal 2c transmits the “relay mutual communication screen URL” and the “date and time limited password” to the user terminal 4a via the router 2b and the network 5 (step S104).
When the reservation use date and time comes, the store clerk at the actual store 2 side wears the glasses-type communication device 2a, starts up, and enters a standby state.
On the other hand, on the user terminal 4a side, the mutual communication screen is accessed by the WEB browser, and the “date and time limited password” is input.
The management server 3 authenticates the “date and time limited password” input from the user terminal 4a, and relays between the glasses-type communication device 2a and the user terminal 4a via the management server 3 when the authentication result is “permitted”. Then, the relay video is viewed and a call is made (step S105).
Here, the visual field video signal and the voice signal due to the call during the relay period are all stored in the result data DB 42 in the management server 3.
When there is a reproduction request for the relayed visual field video signal and conversation (audio signal) from the actual store terminal 2c to the management server 3, the management screen of the management server 3 is accessed and stored in the corresponding performance data DB 42. A signal and a conversation (audio signal) are reproduced (step S106).

(Management server processing flow)
FIG. 11 is a flowchart showing the processing flow of the management server 3.
First, before describing the processing flow of the management server 3, the display form of the management screen when the actual store terminal 2 c accesses the management screen of the management server 3 via the network 5 will be described.
14 to 16 show display examples of the reservation screen by the management server 3, that is, display forms of the management screen.
As shown in FIG. 14, the management screen of the management server is called a reservation management system screen, and displays a first display area displayed as a calendar, a reservation ID and a date / time limited password, and “reservation disclosure date / time” and “reservation date / time”. A second display area for receiving the “end date and time”, a “reservation registration” button, a “reservation change” button, a “reservation search” button, and a third display area in which a “reservation deletion” button is displayed.
In the upper right column of the first display area, there is an area that can designate switching of the display range in units of “month”, “week”, and “day”. In the example shown in FIG. 14, the “month” unit is specified, and only the “month” area is highlighted. In the upper left view of the first display area, a button is provided that allows the display month to be switched. In the calendar display in the first display area shown in FIG. 14, the reservation status within the period from October 1, 2015 to October 7, 2015 is displayed as a “reservation” icon. It is shown that reservations are registered on October 2 (Friday) and October 4 (Sunday).

FIG. 15 shows a display state in which “week” unit display is selected, and only “week” in the upper right column of the first display area is highlighted. In the calendar display in the first display area, within the period from October 1, 2015 to October 7, 2015, October 2 (Friday) 12: 00 to 13:00 and October 4 ( It shows that the reservation is registered at 15: 00-16: 00.
FIG. 16 shows a display state in which “day” unit display is selected, and only “day” in the upper right column of the first display area is highlighted. In the calendar display in the first display area, it is shown that the reservation is registered on Friday, October 2, 2015 from 12:00 to 13:00.

As shown in FIG. 11, in step S111, when a reservation request for use date and time of the glasses-type communication device 2a is received from the actual store terminal 2c, the reservation registration / change unit 35 constituting the management server 3 newly registers the reservation. It is determined whether or not. If it is a new registration, the process proceeds to step S112. If it is not a new registration, the process proceeds to step S113.
Subsequently, in step S112, the actual store terminal 2c accesses the management screen of the management server 3, and registers the reservation start date and time and the reservation end date and time (registers the start and end date). FIG. 17 shows a screen display example at the time of reservation registration, that is, a management screen.
The wearer of the actual store 2 switches the first display area shown in FIG. 17 to “week” unit display, and in the calendar display in the first display area, “October 6th 18: Move the cursor to the column “00 to 19:00” (hatched area in the figure) and click.
Then, “2015/10/6 18:00” is automatically displayed in the “Reservation start date and time” column in the second display area, and “2015/10/6 19:00” is automatically displayed in the “Reservation end date and time” column. Input. Thereafter, the cursor is moved to and clicked on the “reservation registration” button in the third display area. The reservation registration / change unit 35 receives the registered start / end date and time.

  In step S114, the reservation registration / change unit 35 accesses the schedule management DB 40 and confirms whether or not there is a registration that overlaps with the start / end date / time received in step S112. As a result of the confirmation, if a duplicate registration already exists, the process returns to step S112 to prompt registration of the start / end date and time by the actual store terminal 2c. On the other hand, if there is no overlapping registration as a result of the confirmation, the process proceeds to step S115.

  In step S115, the password issuing unit 38 transmits a password (date and time limited password) for limiting the date and time to the actual store terminal 2c via the network 5, and specifies the date and time limited password and the transmission destination actual store terminal 2c. And store it in the password management DB 41. The display form of the management screen at this time is shown in FIG. As shown in FIG. 18, a dialog box “The reservation has been registered. Please tell the customer the date and time limited password. 123456” is displayed on the screen. At the same time, “999999999999” is displayed in the “Reservation ID” field of the second display area in the management screen, and “123456” is naturally displayed in the “Date and time limited password” field. Here, the “reservation ID” is automatically generated by the management server 3.

  In step S113, the reservation search unit 36 displays a dialog box that prompts input of a reservation ID on the management screen. FIG. 19 shows a state in which this dialog box is displayed. As shown in FIG. 19, the dialog box includes a message “Reservation search: Search from reservation ID”, an area for receiving an input of a reservation ID, a “Search” button, and a “Cancel” button. included. When “999999999999” is input in the area for receiving the input of the reservation ID and the cursor is moved to and clicked on the “search” button, the reservation search unit 36 accesses the schedule management DB 40 and extracts the corresponding reservation information. The reservation search unit 36 displays the extracted reservation information on the management screen as shown in FIG. Specifically, during the calendar display of the first display area, a “reservation” icon hatched in the column of October 12 (Monday) 10: 00 to 11:00 is displayed, and in the second display area , “999999999999” previously input is displayed in the “reservation ID” field, and hatched in the “2015/10/12 10:00” and “reservation end date and time” fields hatched in the “reservation start date and time” field “2015/10/12 11:00” is displayed.

  In step S116, the reservation registration / change unit 35 determines whether a change or delete instruction is input to the reservation information extracted in step S113. If a reservation change instruction is input, the process proceeds to step S117. If a reservation deletion instruction is input, the process proceeds to step S119.

  In step S117, as shown in FIG. 21, the reservation registration / change unit 35, when the reservation information to be changed in the calendar display in the first display area on the management screen is designated by clicking, A “reservation” icon corresponding to the designated reservation information is hatched. In this state, the changed date and time are entered in the “reservation start date and time” and “reservation end date and time” fields in the second display area, respectively, and the “change reservation” button in the third display area is displayed. When the instruction input (click) is accepted, the reservation registration / change unit 35 takes in the reservation start / end date and time after the change.

  In step S118, the reservation registration / change unit 36 accesses the schedule management DB 40 and confirms whether or not there is reservation information that overlaps the changed reservation start / end date and time acquired in step S117. As a result of confirmation, if duplicate reservation information has already been stored, the process returns to step S117 to prompt the user to input the changed reservation start / end date and time again. On the other hand, if there is no overlapping reservation information, as shown in FIG. 22, a dialog box including a message “Reservation has been changed” is displayed on the management screen, and the process proceeds to step S120.

  In step S119, the reservation deletion unit 37 designates the “reservation” icon to be deleted in the calendar display of the first display area by clicking on the management screen shown in FIG. 23, and “reservation deletion” in the third display area. When an instruction input to the button is received, the schedule management DB 40 is accessed to extract the reservation information to be deleted, and a dialog box for confirming execution of the reservation information deletion is displayed on the management screen as shown in FIG. To do. When a delete instruction is received in the dialog box, the extracted reservation information to be deleted is deleted from the schedule management DB 40, and a dialog box informing that the reservation has been deleted is displayed on the management screen as shown in FIG. The process proceeds to the next step S120.

  The processing from step S111 to step S119 described above is a processing flow of the reservation registration, reservation change, and reservation deletion functions executed by the management server 3.

  Next, when the management server 3 receives the date / time limited password from the user terminal 4a via the network 5 (step S120), the received date / time limited password is stored in the communication I / F 43 and the internal bus constituting the management server 3. The data is input to the authentication processing unit 39 via 44.

  In step S121, the authentication processing unit 39 accesses the password management DB 41 and executes the authentication process by determining whether or not the input date / time limited password is stored in the password management DB 41. If the result of the authentication process is “Yes”, that is, if a date / time limited password that matches the input date / time limited password is stored in the password management DB 41, the process proceeds to step S 122, and the user terminal is connected via the management server 3. The relay between 4a and the glasses-type communication device 2a is established. After the end of the relay, the process proceeds to the next step S124. On the other hand, when the result of the authentication processing is “non-”, that is, when the date-and-time-limited password that matches the input date-and-time-limited password is not stored in the password management DB 41, the authentication processing unit 39 performs the communication I / F 43 and An invalid message or a retry request is transmitted to the user terminal 4a via the network 5 (step S123), and the process ends.

  In step S124, the management server 3 monitors whether or not there is a request for downloading the visual field video signal and the audio signal during the relay period from the actual store terminal 2c via the network 5. If there is no download request, the process ends. On the other hand, when there is the download request, the management server 3 refers to the result data DB 42 and transmits a visual field video signal and an audio signal corresponding to the download request to the actual store terminal 2c via the network 5 ( Step S125), the process is terminated.

(Processing flow of actual store terminal and communication control unit)
FIG. 12 shows a processing flow of the communication control unit 20 constituting the actual store terminal 2c and the glasses-type communication device 2a.
Below, the case where the spectacles type communication apparatus 2a shown in FIG.2 and FIG.3 is used is demonstrated to an example.

  In step S131, the login password is input to the login screen displayed on the display unit 232 via the input unit 231 configuring the actual store terminal 2c. The login screen has an area for accepting input of “user name” and “password”. When the input of the login password is received in the “password” input field, the input login password is input to the login authentication unit 236 via the input I / F 233 and the internal bus 240. The login authentication unit 236 determines whether or not the input login password is valid, that is, executes login authentication processing (step S132). If the result of the login authentication process is “non”, the process returns to step S131. If the result of the login authentication process is “permitted”, the process proceeds to step S133.

  In step S133, the actual store terminal 2c accesses the management screen of the management server 3 via the communication I / F 239, the router 2b, and the network 5. In step S134, the actual store terminal 2c executes the use of the glasses-type communication device 2a, that is, the reservation of the use date and time of the mutual communication system 1 on the management screen described in the processing flow of the management server 3 described above.

In step S135, when the date-and-time-limited password is received from the management server 3 via the network 5, the actual shop terminal 2c sends the “date-and-time-limited password” and “intercommunication screen” to the user terminal 4a via the router 2b and the network 5. URL "is transmitted (step S136).
Thereafter, when the reserved use date / time is reached, the glasses-type communication device 2a is activated and enters a standby state (step S137).
When the relay between the user terminal 4a and the eyeglass-type communication device 2a is established via the management server 3 (step S138), the process proceeds to step S139.

In step 139, a visual field image signal similar to the field of view of the store clerk wearing the glasses-type communication device 2a is generated by the image processing unit 205 constituting the communication control unit 20, and the store clerk wearing the glasses-type communication device 2a. The sound processing unit 206 generates a sound signal such as a description of the product.
The field-of-view video signal and the audio signal are superimposed by the synthesis processing unit 207, and the compressed field-of-view video signal and audio signal are transmitted to the user terminal 4a via the router 2b and the network 5. Also, an audio signal requesting introduction of another product from the user A who is the owner of the user terminal 4a is received via the communication I / F 209, the audio processing unit 206, and the speaker 22b. The store clerk wearing the glasses-type communication device 2a picks up another product in response to the request of the user A, and transmits the visual image signal and the audio signal related to the description of the product via the network 5 in the same manner as described above. To the user terminal 4a. In this way, the store clerk of the actual store 2 can pick up and explain the product as desired by the user A via the glasses-type communication device 2a, thereby realizing a virtual store that is even more realistic.

In step S140, the glasses-type communication device 2a is stopped.
In step S141, the actual shop terminal 2c sends a download request for the visual field video signal during the relay period, the voice signal of the clerk wearing the glasses-type communication device 2a, and the voice signal of the user A who is the owner of the user terminal 4a to the network. 5 to the management server 3.
In step S142, the actual shop end 2c receives the visual field video signal and the audio signal corresponding to the download request stored in the result data DB 42 from the management server 3 via the network 5, and the process is terminated.

(User terminal processing flow)
FIG. 13 shows a processing flow of the user terminal. In addition, although the process flow of the user terminal 4a is demonstrated here, the remote user terminal 4b and the user terminal 4c which exist in a remote place are also the same. As described above, the functional block configuration of the user terminal 4a is the same as the functional block configuration of the actual store terminal 2c shown in FIG. Therefore, the following description will be made using each functional block shown in FIG.

  As shown in FIG. 13, in step S153, the user terminal 4a receives the “date and time limited password” and the “mutual communication screen URL” from the actual store terminal 2c via the network 5, using the communication I / F 239.

  In step S154, using the WEB browser, the “intercommunication screen URL” is input from the input unit 231 to access the mutual communication screen, and the “date and time limited password” is input from the input unit 231.

  In step S157, when the result of the authentication processing by the management server 3 (authentication processing unit 39) received via the network 5 is “non-”, the user terminal 4a receives an invalid message from the management server 3 or a retry request. Is received (step S156), and the process returns to step S154. On the other hand, when the result of the authentication process by the management server 3 is “OK”, the relay between the user terminal 4a and the glasses-type communication device 2a is established via the management server 3 (step S157). move on.

In step S158, a visual field image signal similar to the field of view of the store clerk wearing the glasses-type communication device 2a is displayed on the mutual communication screen of the user terminal 4a, and a description of the product by the store clerk wearing the glasses-type communication device 2a, etc. Is output from a speaker (not shown) of the user terminal 4a. FIG. 26 shows a screen display example of the user terminal 4a during mutual communication.
As shown in FIG. 26, the blouse picked up by the store clerk wearing the glasses-type communication device 2a is displayed in the approximate center of the screen. If the user A who is the owner of the user terminal 4a wants to obtain confirmation and explanation of other products from the mutual communication screen and the audio signal of the store clerk, the network 5 Via the clerk wearing the glasses-type communication device 2a. For example, other clothes displayed on the shelves in FIG. 26 can be viewed in the same manner, and explanations of the products can be heard. Thereby, the user A who is the owner of the user terminal 4a can check and purchase the product as if he / she visited the actual store 2.

  In step S159, the relay is completed and the process is completed by the viewing end operation from the user terminal 4a.

  In the present embodiment, the business operator who owns the management server 3 and operates the mutual communication system 1 having the glasses-type communication device 2a concludes a usage contract with the manager of the actual store 2 in advance, Contracts that receive monthly usage fees as consideration, and contracts that allow mutual communication system 1 to be used regardless of the number of monthly usages at a fixed monthly fee, or contracts that charge monthly fees in addition to basic charges May be exchanged. In this case, since the management server 3 has the performance data DB 42, it is possible to easily calculate a charge in a metered amount system and charge appropriately.

  As described above, according to the present embodiment, a wearer who is a store clerk wearing a small eyeglass-type communication device or a worker at a work site can send video and audio within his field of view to a remote user. It is possible to provide an intercommunication system having an eyeglass-type communication device that can improve interactivity by transmitting to a terminal and executing a change in visual field in real time in response to a request from a user.

  In addition, according to the present embodiment, a virtual store visitor can realize a virtual visit to the store more realistically by having the wearer in the actual store pick up and explain the product as desired by the user via the glasses-type communication device. .

FIG. 27 is an overall schematic configuration diagram of an intercommunication system according to another embodiment of the present invention.
In the first embodiment, the actual store 2 includes the glasses-type communication device 2a, the router 2b, and the actual store terminal 2c that is a PC. In the present embodiment, the construction company terminal 2c ' The eyeglass-type communication device 2a worn by the worker, and the visual field image signal and the audio signal obtained by the eyeglass-type communication device 2a are provided with a mobile terminal 2b 'such as a smartphone, and the visual field image signal and the audio signal are managed by the network 5 The difference is that the transmission is made to the construction company terminal 2c ′ or the remote user terminal via the server 3. Other points or configurations are the same as those in the first embodiment, and redundant description is omitted below.

  As shown in FIG. 27, the intercommunication system 1 of this embodiment includes a construction company terminal 2c ′, a management server 3, a user terminal 4a that is a PC owned by a user A, and a user terminal that is a router bullet owned by a user B. 4b, a user terminal 4c that is a smartphone (mobile phone) owned by the user C, a glasses-type communication device 2a that can be worn by a worker at a work site, a terminal 2b ′ such as a smartphone that is held by the worker, It is comprised from the network 5 connected so that communication is possible.

The management server 3 has the functional block configuration shown in FIG. 9 as in the first embodiment, and the processing flow of the management server 3 is the processing flow shown in FIG. 11 as in the first embodiment.
The functional block configurations of the construction company terminal 2c ′, the user terminal 4a, the user terminal 4b, and the user terminal 4c are the same as the functional block configuration shown in FIG. In addition, these processing flows are step S158 in FIG. 13, for example, when the user A who owns the user terminal 4a is the owner of the building under construction, the mutual communication screen of the user terminal 4a or the construction company terminal 2c ′. A visual field image signal similar to the field of view of the worker wearing the glasses-type communication device 2a is displayed on the mutual communication screen, and the explanation of the progress of the construction period by the worker wearing the glasses-type communication device 2a, etc. A signal is output from a speaker (not shown) of the user terminal 4a or the construction company terminal 2c ′.
FIG. 28 shows a screen display example of the user terminal 4a or the construction company terminal 2c ′ during mutual communication. As shown in FIG. 28, the worker wearing the glasses-type communication device 2a displays the status of the work site such as a construction site in the approximate center of the screen. For example, a skilled employee who operates the construction company terminal 2c ′ uses a mutual communication screen and a voice signal of the worker to temporarily confirm the work procedure and consult about the progress of the construction period via a microphone (not shown) via the network 5. It is transmitted to the worker wearing the glasses-type communication device 2a.
This makes it possible for a skilled worker of a remote construction company to easily check the work procedure and know the progress of the work schedule as if he were at the work site. The product can be confirmed and purchased.

In addition, the processing flow of the communication control unit 20 constituting the mobile terminal 2b ′ such as a smartphone and the glasses-type communication device 2a is illustrated in FIG. 12 in the above-described first embodiment via the router 2b and the network 5 via the management server 3 or the network 5. In this embodiment, the information transmitted to the user terminal 4a is transmitted from the portable terminal 2b ′ to the user terminal 4a or the construction company terminal 2c ′ via the network 5.
According to the present embodiment, a wearer in an actual store wearing a small eyeglass-type communication device or a worker at a work site or the like transmits video and audio within his field of view to a remote user terminal and It is possible to provide an intercommunication system having an eyeglass-type communication device that can improve interactivity by executing a visual field change or the like in real time in response to a request from the user.

  In addition, according to the present embodiment, an image that remains in the line of sight of a worker who works at a work site such as a building site via a glasses-type communication device (image within the worker's field of view) Since it is transmitted to the terminal, it is possible to easily check the work procedure and the progress of the construction period.

In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. In the above-described embodiment, the site and the glasses-type communication device are described as one-to-one. However, when a plurality of glasses-type communication devices are installed on the site, the site server uses the reservation registration unit of the management server to When reserving the date and time of mutual communication between the spectacle-type communication device that is the input / output means and the remote information terminal, one spectacle-type communication device of a plurality of spectacle-type communication devices installed in an actual store is used. It is configured to make a reservation.
Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace the configurations of other embodiments with respect to a part of the configurations of the embodiments.

  FIG. 29 is a diagram illustrating a configuration example of a communication control unit in the mutual communication system 1 according to another embodiment of the present invention. In this embodiment, a monitor display processing unit 212, a communication quality display processing unit 213, and a bit rate / frame rate adjustment processing unit 214 are added to the embodiment shown in FIG. The same reference numerals are given and description thereof is omitted.

The communication control unit 20 includes a monitor display processing unit 212 for performing monitor display, a communication quality display processing unit 213 for performing communication quality display, and a bit rate / for changing the number of frames / pixels of video to be communicated. A frame rate adjustment processing unit 214.
The monitor display processing unit 212 displays an image captured by the camera 1a of the input unit on the monitor 21c of the display unit. The video imaged by the camera 1c is a visual field image (angle of view), and on the viewer's side, the field of view video (angle of view) that can confirm which range the relay by the camera 1c relays. It is. On the monitor 1c of the display unit, the next information is displayed / hidden by overlaying on the video display.
(1) Viewing mark (2) Reserving remaining time (3) Horizontal confirmation grid (like a crosshair displayed during camera shooting)
The display / non-display of information is switched by operating an operation button (switch) attached to the glasses-type communication device 2a, for example.
The communication quality display processing unit 213 displays an indicator that indicates a guideline of the communication speed on the display screen of the remote information terminal (user terminal 4) that views the relay. For example, a signal indicating an indicator is periodically transmitted from the glasses-type communication device 2a to the management server 3 side, the communication speed is calculated on the management server 3 side, and the average communication speed (uplink speed) for a certain period is calculated. Originally, the communication quality is displayed in three stages (see FIG. 30).
The bit rate / frame rate adjustment processing unit 214 changes the number of frames / number of pixels of video to be communicated.
30A, 30B, and 30C are diagrams illustrating examples of the monitor screen of the display unit 2c. FIG. 30A is an example in which a viewing mark 21c1, a reservation remaining time 21c2, a quality confirmation indicator 21c3, a grid line 21c4, and the like are overlaid and displayed on the monitor screen 21c0 on the visual field image being captured.
As shown in FIG. 30B, the display / non-display of these items is switched as follows: ON / OFF button 21c11 for viewing mark displayed on monitor screen 21c0, ON / OFF button 21c21 for remaining reservation time, ON / OFF of indicator The OFF button 21c3, the grid line 21c4 reservation remaining time 21c2, the quality confirmation indicator 21c31, and the grid line ON / OFF button 21c41 are operated by operation. The button operation is performed by the user of the communication device.

FIG. 31 is a flowchart illustrating a processing flow of the communication control unit 20.
In step S <b> 202, the video acquisition unit 203 acquires a video signal (step S <b> 201) input from the camera 21 a of the input unit and supplies it to the image processing unit 205.
In step S <b> 203, the audio acquisition unit 204 acquires an audio signal (step S <b> 201) input from the microphone 22 a of the input unit using the input I / F 201 and supplies the acquired audio signal to the audio processing unit 206.
In step S <b> 204, the image processing unit 205 performs predetermined image processing on the video signal supplied from the input I / F 201 and supplies the video signal to the operation recognition unit 208.
In step S <b> 205, the voice processing unit 206 performs predetermined voice processing on the voice signal supplied from the input I / F 201 and supplies the voice signal to the motion recognition unit 208.
In step S206, the motion recognition unit 208 receives the posture data of the posture sensor of the input unit, calculates the motion recognition data including at least the line-of-sight direction of the wearer used by the glasses-type communication device 2a, and sends it to the composition processing unit 207. Supply.
In step S207, the synthesis processing unit 207 performs a synthesis process on the video signal and the audio signal supplied from the image processing unit 205 and the audio processing unit 206, and the data supplied from the motion recognition unit 208, and the monitor display processing unit 212, The data is supplied to the communication quality display processing unit 213 and the bit rate / frame rate adjustment processing unit 214.

In step S208, when displaying the visual field video signal input from the camera 1a on the monitor 21c of the display unit, the monitor display processing unit 212 views, for example, a video in addition to the video displayed by the camera 1a. A process of enabling overlaying (overlapping) any one or more of the viewing mark 21c1 indicating that the reservation is in progress, the reservation remaining time 21c2 indicating the remaining reservation time, and the horizontal confirmation grid 21c4 is executed.
In step S209, the communication quality display processing unit 213 executes a process of enabling an indicator 21c3 serving as a guideline for the communication speed to be displayed on the display screen of the remote information terminal that views the relay by the camera 1a. The indicator 21c3 displays the quality in three stages, for example.
In step S210, the output I / F 202 outputs the video signal and audio signal processed in steps S208 and S209 to the display unit 21c and the speaker 22b.
In step S211, the display unit 21c monitors the video signal output from the output I / F 202.
In step S212, when communicating the video signals received from the bit rate / frame rate adjustment processing unit 214 and the composition processing unit 207 to the user terminal, the number of video frames / number of images is changed (bit rate / frame rate adjustment). It outputs to communication I / F209.
In step S213, the communication I / F 209 communicates the video signal adjusted in bit rate / frame rate in step S212 to the user terminals 4a to 4c via the network.

Thereby, the user terminal can catch the video signal obtained by photographing on the communication device 2a side from the same viewpoint as the communication device 2a side.
Further, according to the present embodiment, the correlation between the image of the object camera explained by the presenter and the sound of the object explanation by the microphone is achieved, and the image and sound of the object are transmitted to the remote user terminal. By transmitting, for example, even if the explainer of the communication device freely moves and replaces the target object to be described, the visual field image and sound of the target object captured by the presenter can be misunderstood to the remote user side. I can tell you exactly.

DESCRIPTION OF SYMBOLS 1 ... Mutual communication system, 2 ... Real shop, 2a, 2a '... Glasses type communication apparatus, 2b ... Router, 2b' ... Portable terminal (smart phone), 2c ... Real shop terminal, 2c '... Construction company terminal, 3 ... Management Server, 4a ... user terminal a, 4b ... user terminal b, 4c ... user terminal c, 5 ... network, 20, 20 '... communication control unit, 21 ... imaging unit, 21a, 21a' ... camera, 21b ... posture sensor, 21c ... small display, 22 ... voice input / output unit, 22a, 22a '... microphone, 22b, 22b' ... speaker, 23 ... support unit, 24 ... glass body, 24a ... left temple, 24b ... right temple, 24c ... lens frame 24d ... lens or glass, 201 ... input I / F, 202 ... output I / F, 203 ... video acquisition unit, 204 ... audio Acquiring unit, 205 ... Image processing unit, 206 ... Audio processing unit, 207 ... Composition processing unit, 208 ... Motion recognition unit, 209 ... Communication I / F, 210 ... Internal bus, 211 ... GPS device, 212 ... Monitor display processing unit 213: Communication quality display processing unit, 214: Bit rate / frame rate adjustment processing unit, 221 ... Vibration sensor, 222 ... Amplifier, 223 ... A / D converter, 224 ... Bone conduction speaker, 225 ... Amplifier, 226 ... D / A converter, 231 ... input unit, 232 ... display unit, 233 ... input I / F, 234 ... output I / F, 235 ... calculation unit, 238 ... battery unit, 239 ... communication I / F, 240 ... internal bus 31 ... Input unit, 32 ... Display unit, 33 ... Input I / F, 34 ... Output I / F, 35 ... Reservation registration / change unit, 36 ... Reservation search unit, 37 ... Reservation deletion unit, 38 ... Password issue unit 39 ... authentication unit, 40 ... scheduling DB, 41 ... password management DB, 42 ... actual data DB, 43 ... communication I / F, 44 ... internal bus

Claims (14)

An intercommunication system in which an on-site information terminal, a remote information terminal, and a management server are connected via a network to achieve mutual communication between the on-site information terminal and the remote information terminal,
The field information terminal is equipped with a field information input / output unit that is worn by a specific wearer and inputs / outputs visual field image information similar to the field of view of the wearer and a voice signal of the wearer.
The remote information terminal comprises field information input / output means for inputting / outputting information transmitted from the field information terminal, and voice input / output means for inputting / outputting at least the voice of the remote information terminal user,
The management server is communicably connected to the field information terminal and the remote information terminal via the network,
The management server has a function of establishing mutual communication between the field information terminal and the remote information terminal via the management server. The mutual communication system according to claim 1,
The site information terminal is disposed at a site, and the site information input / output means of the site information terminal is a glasses-type communication device worn by a wearer at the site,
When the user of the remote information terminal communicates with the site by accessing the site information terminal at the site from a remote location,
The management server issues reservation registration means for managing reservations for mutual communication date and time between the spectacles type communication device as the field information input / output means and the remote information terminal, and a date and time limited password enabling the mutual communication. A mutual communication system characterized in that a reservation for mutual communication between the spectacles-type communication device and the remote information terminal is established in advance by having a password issuing means. The mutual communication system according to claim 2,
The management server includes an authentication processing unit that authenticates the date-and-time-limited password that identifies the glasses-type communication device, and performs mutual communication between the glasses-type communication device and the remote information terminal according to an authentication result by the authentication processing unit. Mutual communication system characterized by establishing. In the mutual communication system according to claim 3, when a plurality of eyeglass-type communication devices are installed in a store on site,
When the reservation registration unit of the management server reserves the date and time of mutual communication between the spectacles type communication device as the site information input / output unit and the remote information terminal, a plurality of glasses installed at the site Communication system characterized in that one eyeglass-type communication device is reserved among the communication devices. In the mutual communication system according to claim 3 or claim 4,
The management server has a record data database for storing the wearer's visual field video signal and audio signal and an audio signal from the remote information terminal during a mutual communication period between the glasses-type communication device and the remote information terminal. Mutual communication system characterized by The intercommunication system according to any one of claims 3 to 5,
A mutual communication system comprising a schedule management database for storing a plurality of pieces of reservation information on the date and time of mutual communication between the glasses-type communication device registered by the reservation registration means and the remote information terminal. An intercommunication method using an intercommunication system in which an on-site information terminal, a remote information terminal, and a management server are connected via a network and communication between the on-site information terminal and the remote information terminal is achieved. ,
The on-site information terminal is attached to a specific wearer, and the on-site information input / output means inputs / outputs visual field video information similar to the wearer's field of view and the audio signal of the wearer,
The remote information terminal outputs information transmitted from the field information terminal by the field information input / output means, and inputs / outputs at least the voice of the remote information terminal user by the voice input / output means,
The management server establishes mutual communication between the field information terminal and the remote information terminal via the management server. The mutual communication method according to claim 7,
The site information terminal is disposed at a site, and the site information input / output means of the site information terminal is a glasses-type communication device worn by a wearer at the site,
When the user of the remote information terminal communicates with the site by accessing the site information terminal at the site from a remote location,
The management server manages the reservation of the date and time of mutual communication between the spectacles-type communication device that is the field information input / output means and the remote information terminal by the reservation registration means, and the date and time limitation that enables the mutual communication An intercommunication method characterized in that a reservation for mutual communication between the glasses-type communication device and the remote information terminal is established in advance by having a password issuing means for issuing a password. The mutual communication method according to claim 8, wherein a plurality of glasses-type communication devices are installed at the site,
When the reservation registration unit of the management server reserves the date and time of mutual communication between the spectacles type communication device as the site information input / output unit and the remote information terminal, a plurality of glasses installed at the site An intercommunication method characterized by reserving one eyeglass-type communication device among the communication devices. A communication device on the actual store / site side, a user terminal on the user side, a management server that manages data between the communication device and the user terminal, and a network that connects the communication device to the user terminal and the management server In mutual communication system,
The communication device has a communication control unit,
The communication control unit
Including an input unit, a display unit, an output unit, a communication interface unit, an image processing unit, a voice processing unit, a synthesis processing unit, an operation recognition unit,
The image processing unit receives a visual field video signal and data including a line-of-sight direction input from the input unit, obtains a visual field video signal corresponding to a field of view of a wearer wearing the communication device, and supplies the visual field video signal to the synthesis processing unit And
The audio processing unit supplies an audio signal input from the input unit to the synthesis processing unit, and supplies an audio signal transmitted from the user terminal to the audio output unit,
The motion recognition unit receives posture data of the input unit, calculates data including the line-of-sight direction based on the posture data, and supplies the data to the synthesis processing unit.
The synthesis processing unit supplies the visual field video signal and the audio signal to the output unit and the communication interface unit,
The communication interface unit transmits the visual field video signal and the audio signal to a corresponding user terminal via the network,
The communication control unit further includes one or more of a monitor display processing unit, a communication quality display processing unit, and a bit rate / frame rate adjustment processing unit that process a display function on the monitor of the display unit,
The monitor display processing unit overlays and displays any one or more of a viewing mark, a remaining reservation time, and a horizontal confirmation grid on the video display based on the visual field video signal captured by the input unit or the communication device To display the hidden menu screen,
The communication quality display function processing unit displays, as an indicator, quality information indicating a guideline of communication speed calculated from a result of periodic signal exchange between the communication device and the management server. Similarly, the user terminal and the management server Display quality information indicating a guideline of communication speed calculated from the result of periodic signal exchange with the user terminal,
The bit rate / frame rate adjustment processing unit changes the number of video frames and / or the number of pixels communicated with the user terminal. A mutual store comprising a communication device on the actual store / site side, a user terminal on the user side, a management server for managing data between the communication device and the user terminal, and a network for connecting the communication device to the user terminal and the management server In the mutual communication method in the communication system,
The communication device has a communication control unit,
The communication control unit
Receiving data including a visual field video signal and a line-of-sight direction input from the input unit, obtaining a visual field video signal corresponding to the visual field of the wearer wearing the communication device, and supplying the visual field video signal to the synthesis processing unit,
Supplying the audio signal input from the input unit to the synthesis processing unit, and supplying the audio signal transmitted from the user terminal to the output unit;
Receiving posture data of the input unit, calculating data including the line-of-sight direction based on the posture data and supplying the data to the synthesis processing unit;
Transmitting the visual field video signal and the audio signal to the corresponding user terminal via the output unit, the communication interface unit, and a network;
Further, on the video display based on the captured visual field video signal, one or more of a viewing mark, a remaining reservation time, and a horizontal confirmation grid are overlaid or displayed or a non-display menu image is displayed by operating the communication device And
Displaying quality information indicating a guideline of communication speed calculated from the result of regular signal exchange between the communication device and the management server as an indicator, and regular signal exchange between the user terminal and the management server Display quality information indicating a guideline of communication speed calculated from the result of the above on the display unit of the user terminal,
An intercommunication method characterized by changing the number of frames and / or the number of pixels of video communicated with the user terminal. An intercommunication program installed in an intercommunication device that receives a video signal of a camera on the field side, an audio signal of a microphone, and attitude data of an attitude sensor, and communicates the video signal and the audio signal with a user terminal,
Receiving a visual field video signal and data including a line-of-sight direction input from the camera and the posture sensor, obtaining a visual field video signal corresponding to a field of view of a wearer wearing the communication device, and supplying the visual field video signal to a synthesis processing unit;
Supplying an audio signal input from the microphone to the synthesis processing unit, and supplying an audio signal transmitted from the user terminal to a speaker of an output unit;
Transmitting the visual field video signal and the audio signal to the corresponding user terminal through the communication interface and network with the speaker;
A video display based on the captured visual field video signal is displayed by overlaying one or more of a viewing mark, a remaining reservation time, and a horizontal confirmation grid, or a non-display menu image is displayed by operating the communication device. Step,
Displaying quality information indicating a standard of communication speed of the visual field video signal on a display unit of the user terminal;
An intercommunication program for causing a computer to execute a step of changing the number of frames and / or the number of pixels communicated between the user terminal and the intercommunication device. In an intercommunication system mounted on an intercommunication device that receives a video signal of a camera on the site side, an audio signal of a microphone, and communicates the video signal and the audio signal with a remote user terminal,
The communication device includes:
The eyeglasses that can be worn and used by the instructor explaining the object to the user at the remote location,
An attitude sensor indicating the attitude of the communication device;
Means for receiving data including a visual field video signal and a line-of-sight direction input from the camera and the posture sensor, obtaining a visual field video signal corresponding to a field of view of a wearer wearing the communication device, and supplying the visual field video signal to a synthesis processing unit;
Means for supplying an audio signal input from the microphone to the synthesis processing unit, and supplying an audio signal transmitted from the user terminal to a speaker of an output unit;
Means for aiming a correlation between the video of the object synthesized by the synthesis processing unit and the audio of the object explained by the microphone, and transmitting the video and audio of the object to the user terminal;
A mutual communication system characterized by the provision of The mutual communication system according to claim 13 further includes:
The video display based on the visual field video signal captured by the camera is displayed by overlaying any one or more of a viewing mark, a reservation remaining time, and a horizontal confirmation grid, or a menu screen that is not displayed by operating the communication device. Means for displaying,
Means for displaying quality information indicating an indication of a communication speed of the visual field video signal on a display unit of the user terminal;
Means for changing the number of video frames and / or the number of pixels communicated between the user terminal and the mutual communication device;
An intercommunication system characterized by comprising: