JP6547467B2 - Communication support device, communication support method, program and communication support system - Google Patents

Description

  The present invention relates to a communication support apparatus, a communication support method, a program and a communication support system.

  The most direct communication is face-to-face communication such as conversations and meetings. However, when the communication partner is in a remote office, etc., telecommunication such as fixed telephone, mobile phone, electronic mail, video conference system, chat system, electronic bulletin board, etc. is used. At this time, when starting direct communication, first look at the other side of the communication partner and understand the situation of the communication partner (for example, after confirming the presence / absence of the communication partner by visual observation etc.) ), Can start communication. On the other hand, when telecommunications are started, it may be difficult to understand the situation because there is a distance gap with the communication partner.

  In recent years, a presence display system capable of automatically collecting a user's destination and presence / absence and viewing / sharing with a plurality of users by estimating the presence status by using a PC (personal computer) usage status or a seating sensor etc. Are also becoming popular (eg, Microsoft Lync®, etc.). Furthermore, in addition to the use situation of the PC and the situation estimation by various sensors, etc., the situation of the communication partner can be confirmed from the camera image (Patent Document 1), or the prior adjustment before communication can be simplified (Patent Document 2) There is disclosed a technique of predicting in advance the means of selectable telecommunications, performing grouping (Patent Document 3), and automatically displaying (Conventional Document 4).

  In Patent Document 1, when a person appearing in a captured image on the display screen of a captured image obtained from a camera is touched, the person is identified from the captured image by face recognition, and the telephone number of the person is acquired. A technique for performing telecommunications by telephone to the person is disclosed. Further, Patent Document 2 discloses a technology capable of automatically responding to communication start request in response to a request for starting communication. Further, Patent Document 3 focuses on what type of communication means (voice, text, etc.) is desired to communicate, rather than which communication means is used, and displays a plurality of communication means in groups. Technology is disclosed. Patent Document 4 discloses a technique for searching for information of a communication partner presumed to be necessary and automatically displaying the information according to the context of the window opened by the user.

  From such a background, even when using telecommunications, it is possible to start telecommunications while understanding the situation of the communication partner to some extent.

Unexamined-Japanese-Patent No. 2007-208863 JP 2012-48708 A JP, 2011-151608, A JP 2005-122726 A

  However, for example, in the case of work that does not use a PC or work in a room, work in various places, work while performing another communication, etc., the above-mentioned technology may not be sufficient. For example, even if the other party is in the presence of a communication partner, there is a communication means that is required for paper-based work without a PC and work while using a PC, even while using a PC. It is different. It is also conceivable to obtain and infer to the user all possible conceivable action patterns like this with a wide variety of sensors, but it is possible for the user himself or herself to select one or more communication means best suited to those behavior patterns. It is not easy to do.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a person, a device, or the like in a captured image obtained from a camera at a remote site. Supports selection of communication simply and intuitively so that the user who views the display screen can interact and at the same time can confirm the situation (context) in the captured image and establish optimal telecommunications Providing a communication selection support system that makes it possible to

  As a result, it is possible to establish a realistic communication in which it is possible to change the way of talking (communication means) while looking at the situation as it is near the base even if it is a telecommunications with a remote base. In addition, it is possible to create an active communication environment, such as recalling the desire for communication or recalling that it is necessary to contact someone from the captured image.

In order to solve the above problems, according to one aspect of the present invention, a video analysis unit that analyzes a video captured by an imaging device to obtain a video analysis result including presence or absence of a predetermined object related to the user; And a communication means determination unit for determining a communication means with the user based on the video analysis result and the presence information , wherein the communication means is operated When selected by the body, an application corresponding to the communication means is activated, the object includes the body of the user or the device of the user, and presence information indicating the state of the user is in use by a fixed telephone Information indicating whether or not there is an appointment for the user Information indicating the presence or absence of the user, location information of the user, information indicating whether the PC is online, information indicating whether the user is temporarily leaving, video telephone application Includes information indicating whether the input interface is in use, information indicating whether the audio interface and the video interface are in use, or information indicating whether the time during which the input interface is not used has exceeded a predetermined time. , A communication support device is provided.

  The communication means determination unit may identify a context indicating the situation of the user based on the video analysis result and the presence information, and may determine the communication means based on the context.

  The communication means determination unit may dynamically update the communication means when at least one of the video analysis result and the presence information is changed.

The communication support device may include a communication unit that notifies the communication support terminal of the communication means.
Further, according to another aspect of the present invention, there is provided a video analysis unit for analyzing a video captured by an imaging device to obtain a video analysis result including presence or absence of a predetermined object related to a user, and presence information indicating a state of the user , A communication means determination unit for determining communication means with the user based on the video analysis result and the presence information, and designation in the video based on a designated position for the video An area specifying unit for specifying an area, wherein the image analysis unit obtains the image analysis result based on the specified area, and the object includes the body of the user or the device of the user, Presence information indicating the status of the user includes information indicating whether or not a fixed telephone is in use; Information indicating whether or not the user has entered, presence information of the user, location estimation information of the user, information indicating whether the PC is online, information indicating whether the user is temporarily leaving, Information indicating whether the video telephony application is in use, information indicating whether the audio interface and video interface are in use, or indicating whether the time during which the input interface is not in use has exceeded a predetermined time A communication support device is provided that includes information.

Further, according to another aspect of the present invention, it is possible to analyze an image captured by one or a plurality of imaging devices to obtain an image analysis result including presence or absence of a predetermined object related to the user, and to indicate the state of the user. a presence information acquisition unit for acquiring presence information, the said image analysis result based on the presence information, and determining a means of communication with the user, only contains the communication means is selected by the operating body In the case, an application corresponding to the communication means is activated, the object includes the body of the user or the device of the user, and presence information indicating the state of the user indicates whether a landline is in use or not Information indicating, information indicating whether the user's schedule is included, the user Presence information, location estimation information of the user, information indicating whether the PC is online, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use A communication support method is provided , including information indicating whether an audio interface and a video interface are in use, or information indicating whether a time when an input interface is not used exceeds a predetermined time .

Further, according to another aspect of the present invention, there is provided a video analysis unit that analyzes a video captured by one or a plurality of imaging devices and obtains a video analysis result including presence or absence of a predetermined object related to a user. comprises a presence information acquisition unit for acquiring presence information indicating a state of the user, the said image analysis result based on the presence information, and a communication unit determining unit that determines a means of communication with the user, said communication means Is selected by the operating body, an application corresponding to the communication means is activated, the object includes the body of the user or the device of the user, and the presence information indicating the state of the user is used by a fixed telephone Information indicating whether or not the user Information indicating whether a schedule has been entered, presence information of the user, location estimation information of the user, information indicating whether a PC is online, information indicating whether the user is temporarily leaving, Information indicating whether the video telephony application is in use, information indicating whether the audio interface and video interface are in use, or indicating whether the time during which the input interface is not in use has exceeded a predetermined time A program is provided that includes information and functions as a communication support device.

Further, according to another aspect of the present invention, there is provided a communication support system having a communication support device and a communication support terminal, wherein the communication support device analyzes an image captured by one or more imaging devices. Based on the video analysis result and the presence information, a video analysis unit for obtaining a video analysis result including presence or absence of a predetermined object related to the user, a presence information acquisition unit for obtaining presence information indicating the state of the user, and A communication means determination unit for determining a communication means with the user, the communication support terminal, a communication unit for receiving a notification of the communication means, and an application for activating an application corresponding to the communication means Comprising a tio emission start portion, when the communication means is selected by the operating body, the application corresponding to the communication means is activated, the object comprises a body or the user of the device of the user, the user The presence information indicating the state of the information includes information indicating whether the fixed telephone is in use, information indicating whether the user's schedule is included, the user's presence information, the user's location estimation information, Information indicating whether the PC is online, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, an audio interface and a video interface are in use Information indicating whether the input interface has not been used for a predetermined time It includes information indicating whether, communication support system is provided.

  The communication means determination unit may identify a context indicating the situation of the user based on the video analysis result and the presence information, and may determine the communication means based on the context.

  The communication support terminal may include an application activation unit that activates an application corresponding to the communication unit.

  The communication support terminal may include a menu control unit that generates a selection menu for selecting any of the plurality of communication units determined by the communication unit determination unit.

  The menu control unit may dynamically update the selection menu based on the change when a change occurs in the determination of the plurality of communication units by the communication unit determination unit.

  The menu control unit may control the selection menu based on the priority associated with each of the plurality of communication means.

The menu control unit may control the selection menu based on the answer when an answer to a predetermined question is input.
Further, according to another aspect of the present invention, there is provided a communication support system having a communication support device and a communication support terminal, wherein the communication support device analyzes an image captured by one or more imaging devices. Based on the video analysis result and the presence information, a video analysis unit for obtaining a video analysis result including presence or absence of a predetermined object related to the user, a presence information acquisition unit for obtaining presence information indicating the state of the user, and A communication means determination unit for determining a communication means with the user, the communication support terminal, a communication unit for receiving a notification of the communication means, and an application for activating an application corresponding to the communication means A communication start unit, an output unit for displaying the video, and an input unit for inputting a designated position with respect to the video, and the communication means determination unit determines the designated area according to the designated position and the video analysis result The communication means is determined based on the information and the presence information, the object includes the body of the user or the device of the user, and the presence information indicating the state of the user is in use by a fixed telephone Information indicating whether or not the user's schedule is included, the user's presence information, the user's location estimation information, information indicating whether the PC is online, the user's temporary Information indicating whether you are away, information indicating whether the video telephony application is in use, audio interface and video interface Esu information indicating whether in use or, an input interface is time that is not in use includes information indicating whether exceeds a predetermined time, the communication support system is provided.

  As described above, according to the present invention, a user who views the display screen of the captured image can interact with a person, a device, or the like in the captured image obtained from a camera at a remote site, at the same time Provided is a communication selection support system that can support selection of communication simply and intuitively so that optimum telecommunications can be established while confirming the situation (context) in a captured image. It is possible.

BRIEF DESCRIPTION OF THE DRAWINGS It is a functional block diagram which shows the structural example of the communication assistance system which concerns on the 1st Embodiment of this invention. It is a figure showing an example of a display of a picture. It is a figure for demonstrating the 1st determination method of a specific communication means. It is a figure for demonstrating the 2nd determination method of a specific communication means. It is a figure for demonstrating the 3rd determination method of a specific communication means. It is a figure for demonstrating the 4th determination method of a specific communication means. It is a figure which shows the example of context information. It is a flowchart which shows the flow of operation | movement of the communication assistance system which concerns on the 1st Embodiment of this invention. It is a functional block diagram which shows the structural example of the communication assistance terminal which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating notionally the function of a menu control part. It is a figure which shows the example of the selection menu displayed when 1 user is selected. It is a figure which shows the example of the selection menu displayed when another user is selected. It is a flowchart which shows the flow of operation | movement of the communication assistance terminal which concerns on the 3rd Embodiment of this invention. It is a functional block diagram which shows the structural example of the communication assistance terminal which concerns on the 3rd Embodiment of this invention. It is a figure which shows the example of the question displayed when predetermined conditions are satisfy | filled. It is a figure which shows the expansion example of a question. It is a flowchart which shows the flow of operation | movement of the communication assistance terminal which concerns on the 3rd Embodiment of this invention.

  The present invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. In the present specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals and redundant description will be omitted.

  Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numerals after the same reference numerals. In addition, similar components in different embodiments are distinguished by attaching different alphabets after the same reference numerals. However, in the case where it is not necessary to distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numeral is given.

(1. First Embodiment)
First, a first embodiment of the present invention will be described. A configuration example of a communication support system according to a first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a functional configuration diagram showing a configuration example of a communication support system 1A according to a first embodiment of the present invention. As shown in FIG. 1, the communication support system 1A according to the first embodiment of the present invention includes a communication support device 10 and one or more communication support terminals 20A-1 to 20-j (j is a natural number). , And two imaging devices 50.

  The communication support device 10 is configured to be able to communicate with the two imaging devices 50 via the communication network 40. Here, although the case where two imaging devices 50 exist is demonstrated, one imaging device 50 may be sufficient and multiple other than two may be sufficient. Such two imaging devices 50 may be installed in various places such as places in an office or public facilities. The communication support device 10 is configured to be able to communicate with the communication support terminals 20A-1 to 20-j via the communication network 40.

  Further, as shown in FIG. 1, the communication support apparatus 10 includes the video receiving unit 110, the communication unit 120, the area specifying unit 130, the video analyzing unit 140, the communication means determining unit 150, and the presence information acquiring unit. 160, a camera information storage unit 170, an object information storage unit 180, and a context information storage unit 190. In addition, an application is stored outside the communication support device 10. The application may be stored inside the communication support device 10.

  The communication support terminal 20A includes an input unit 210, a communication unit 220, an output unit 230, an application information storage unit 240, and an application activation unit 250. The communication support terminal 20A may be a smartphone, a game console, a tablet PC, a PC or the like. In addition, an application is stored outside the communication support terminal 20A. The application may be stored inside the communication support terminal 20A. The following description will be made using the communication support terminal 20A-1 used by the user U1.

  The input unit 210 has a function of inputting an operation from the user. In the following, it is mainly assumed that the input unit 210 is configured by a touch panel stacked on the output unit 230, but the configuration of the input unit 210 is not limited to this configuration. For example, the input unit 210 may be a mouse. In such a case, the input unit 210 can input a click operation, a drag operation, and the like. The input unit 210 may also be an infrared camera. In such a case, the input unit 210 can input by gesture. Alternatively, the input unit 210 may be able to input an operation from the user using a plurality of input methods.

  The output unit 230 has a function of outputting various types of information. In the following, it is mainly assumed that the output unit 230 is configured by a display unit (for example, a liquid crystal display) that displays information in a form that can be viewed by the user, but the configuration of the output unit 230 is limited to this configuration I will not. For example, the output unit 230 may further have another configuration (for example, an audio output device, a vibrator, and the like) other than the display unit.

  The imaging device 50 has a function of capturing a video, and a function of transmitting a video to the communication support apparatus 10 via the communication network 40. Here, as shown in FIG. 1, it is assumed that two imaging devices 50 are installed at various positions in the office. However, the place where the two imaging devices 50 are installed is not particularly limited. In addition, camera information of the imaging device 50 (for example, predetermined parameters related to the imaging device 50 such as pan / tilt / zoom, whether or not an image is selected as a display target, etc.) is appropriately changed according to the user's operation. It is also good.

  The image receiving unit 110 receives an image captured by each of the two imaging devices 50 via the communication network 40. At this time, the video reception unit 110 adds unique identification information (for example, a camera number or the like) for uniquely identifying the imaging device 50 that has captured each video, to each of the received two videos. Good. Then, it is possible to identify the imaging device 50 that has captured an image based on the assigned identification information. Of the two videos received by the video receiving unit 110, the video selected as the display target is transmitted to the communication support terminal 20A-1 via the communication unit 120.

  In the communication support terminal 20A-1, the communication unit 220 receives an image from the communication support device 10. Then, the output unit 230 displays the video received by the communication unit 220. FIG. 2 is a diagram showing a display example of a video. As shown in FIG. 2, the output unit 230 displays an image selected as a display target. The video displayed by the output unit 230 shows that the user U2 and the user U3 are having a conversation.

  The object information storage unit 131 stores information on an object (hereinafter, also referred to as “object information”). In the following, the object is described mainly by taking a person as an example, but the object is not limited to a person, and may be a device such as a fixed telephone or the like. The object information has the position and size of an object, the relationship between objects, and the like for each display video source for specifying a video.

  The camera information storage unit 170 stores, as camera information, predetermined parameters related to the imaging device 50, such as pan / tilt / zoom and whether or not an image is selected as a display target. Subsequently, when the user U1 performs an operation of specifying a position in the video, the position specified by the user U1 is input by the input unit 210 as a specified position.

  The area designation unit 130 designates a designated area in the video based on the designated position with respect to the video. Although the designated area may be designated in any way, for example, an area including designated positions from a plurality of areas preset for each seat in the image may be designated as the designated area. At this time, the area designation unit 130 may designate the designated area based on the object information and the camera information in addition to the designated position for the video.

  The video analysis unit 140 analyzes the video to obtain a video analysis result including presence or absence of a predetermined object (for example, the user's body, the user's device, etc.) regarding the user. For example, the video analysis unit 140 may obtain whether a video analysis result is obtained by analyzing whether or not a predetermined object related to the user exists in a designated area in the video. For example, in the example shown in FIG. 2, the video analysis result may be information that one person (= user U2) exists in the A seat.

  The presence information acquisition unit 160 acquires presence information indicating the state of the user U2. Presence information may be obtained from an application being executed by a device such as a landline phone. For example, information indicating whether a fixed telephone is in use may be acquired as presence information from the fixed telephone. Alternatively, information indicating whether a schedule (such as a meeting attendance schedule) is included may be acquired as presence information from the calendar management application. Alternatively, presence information and user location estimation information held by a presence tool such as Microsoft Lync (registered trademark) may be acquired as presence information.

  Alternatively, information such as whether the PC is online or not temporarily leaving may be acquired as presence information from the communication status management application. Alternatively, information indicating whether the video telephony application is in use may be obtained as presence information from the video telephony application. Alternatively, information indicating whether the audio interface and the video interface are in use may be acquired as presence information from the audio interface and the video interface. Alternatively, it may be acquired as presence information whether the time when the input interface is not used exceeds a predetermined time.

  The context information storage unit 190 stores context information. FIG. 7 is a diagram illustrating an example of context information. As shown in FIG. 7, the context information includes information related to the presence or absence of the object analyzed from the video (for example, the number of detected seats and persons shown in FIG. 7) and the presence information (for example, shown in FIG. 7) Whether the landline phone is in use, whether it has a schedule, whether the PC is online, whether it is temporarily out, whether the video phone application is in use, the voice interface is Information as to whether or not it is in use, communication means (for example, “fixed-line phone”, “mail”, “chat”, “in-house mobile phone”, “mobile phone”, “voice mail”, etc. shown in FIG. 7), May be associated information.

  Furthermore, as shown in FIG. 7, a context indicating the user's situation may be associated with these pieces of information. The communication means determination unit 150 determines a communication means with the user U1 based on the video analysis result and the presence information. For example, the communication means determination unit 150 may obtain, from the context information storage unit 190, the communication means associated with the information (image analysis result) regarding the presence or absence of the object analyzed from the image and the presence information.

  At this time, the communication means determination unit 150 may specify the context based on the information (image analysis result) regarding the presence or absence of the object analyzed from the video and the presence information, and may determine the communication means based on the context. For example, the communication means determination unit 150 acquires, from the context information storage unit 190, the context associated with the information (image analysis result) regarding the presence or absence of the object analyzed from the video and the presence information, and the communication means associated with the context. May be acquired from the context information storage unit 190.

  A specific method of determining communication means will be described. FIG. 3 is a diagram for explaining a first determination method of the specific communication means. Referring to FIG. 3, an image is displayed by the output unit 230 of the communication support terminal 20A-1. At this time, it is assumed that a video analysis result that one user U2 is photographed is obtained from the video. In addition, it is assumed that the fixed telephone C2-2 is not used, the video telephone application is not used, and the presence information that the voice interface is not used is obtained. In such a case, the communication means determination unit 150 may determine a fixed telephone as the communication means (No. 1 in FIG. 7).

  FIG. 4 is a diagram for explaining a second determination method of the specific communication means. Referring to FIG. 4, an image is displayed by the output unit 230 of the communication support terminal 20A-1. At this time, it is assumed that a video analysis result is obtained from the video that two users U2 and U3 are shown. It is expected that the user U2 and the user U3 are in conversation. Further, it is assumed that the presence information that the device C2-1 (PC) is online is obtained. In such a case, the communication means determination unit 150 may determine an e-mail as the communication means (No. 4 in FIG. 7).

  FIG. 5 is a diagram for explaining a third determination method of the specific communication means. Referring to FIG. 5, an image is displayed by the output unit 230 of the communication support terminal 20A-1. At this time, it is assumed that a video analysis result that one user U2 is photographed is obtained from the video. Also assume that presence information is obtained that a voice interface is being used but a video telephony application is not being used. In such a case, the communication means determination unit 150 may determine a chat as the communication means (No. 8 in FIG. 7).

  FIG. 6 is a diagram for explaining a fourth determination method of the specific communication means. Referring to FIG. 6, an image is displayed by the output unit 230 of the communication support terminal 20A-1. At this time, it is assumed that a video analysis result that one user U2 is photographed is obtained from the video. In addition, it is assumed that the voice interface is also used, and the presence information that the video telephone application is also used is obtained. In such a case, the communication means determination unit 150 may determine an e-mail as the communication means (No. 9 in FIG. 7).

  The communication means determined by the communication means determination unit 150 is notified to the communication support terminal 20A-1 by the communication unit 120. In the communication support terminal 20-1, the communication unit 220 receives the notification of the communication means. The application information storage unit 240 stores application information in which communication means and information related to the application (for example, application name, application storage location, etc.) are associated.

  The application information may be different for each communication support terminal 20A. In addition, application information may be editable such as addition and deletion. The application activation unit 250 refers to the application information and activates an application corresponding to the notified communication means.

  The communication means determined by the communication means determination unit 150 may not be changed once determined, but may be changed dynamically. That is, the communication means determination unit 150 may dynamically update the communication means when at least one of the video analysis result and the presence information is changed. Alternatively, the communication means determination unit 150 may dynamically change the communication means every predetermined time. The communication means after the change may be similarly notified to the communication support terminal 20A-1.

  As described above, according to the communication support system 1A according to the first embodiment of the present invention, the display of the captured image is performed on a person, a device, or the like in the captured image obtained from the camera of the remote site. It supports selection of communication simply and intuitively so that users who look at the screen can interact and at the same time can confirm optimal situations (contexts) in the captured image while establishing optimal telecommunications. It becomes possible.

  Subsequently, the flow of the operation of the communication support system 1A according to the first embodiment of the present invention will be described. FIG. 8 is a flowchart showing the flow of the operation of the communication support system 1A according to the first embodiment of the present invention. As shown in FIG. 8, the communication support device 10 reads various information (for example, camera information, object information, context information, etc.) (S101). Subsequently, the presence information acquisition unit 160 acquires the presence information (S106), and the video reception unit 110 receives the video from the imaging device 50 (S102).

  When a video is received from the imaging device 50 by the video receiving unit 110, the communication unit 120 distributes the video to the communication support terminal 20-1 (S103). When the designated position by the user U1 is input by the input unit 210, the area designation unit 130 selects a designated area in the video based on the designated position, the camera information, and the object information. If the designated area is not selected by the area designation unit 130 ("No" in S104), the communication support device 10 returns to S102. On the other hand, when the designated area is selected by the area designation unit 130 ("Yes" in S104), the video analysis unit 140 executes video analysis based on the designated area (S105).

  Subsequently, the communication means determination unit 150 determines the communication means based on the video analysis result and the presence information (S107). The communication unit 120 notifies the communication support terminal 20A-1 of the determined communication means (S108). In the communication support terminal 20-1, the communication unit 220 receives the notification of the communication means. The application activation unit 250 activates an application corresponding to the notified communication means.

  When the application is started on the communication support terminal 20A-1 and communication is started ("Yes" in S109), the communication support device 10 ends the operation because it is not necessary to determine the communication means. On the other hand, when the communication support terminal 20A-1 does not activate the application and the communication is not started ("No" in S109), the communication support device 10 returns to S101 and continues the determination of the communication means. Do.

(2. Second embodiment)
Subsequently, a second embodiment of the present invention will be described. The communication support system 1B according to the second embodiment of the present invention differs from the communication support system 1A according to the first embodiment of the present invention in the configuration of the communication support terminal 20B. Therefore, in the second embodiment of the present invention, the description of the communication support device 10 is omitted, and the communication support terminal 20B will be mainly described. First, a configuration example of a communication support terminal 20B according to the second embodiment of the present invention will be described with reference to FIG. FIG. 9 is a functional configuration diagram showing a configuration example of a communication support terminal 20B according to the second embodiment of the present invention.

  As shown in FIG. 9, the communication support terminal 20B according to the second embodiment of the present invention has a menu control unit 260 in comparison with the communication support terminal 20A according to the first embodiment of the present invention. The point is different. Therefore, in the second embodiment of the present invention, among the components included in the communication support terminal 20B, the menu control unit 260 will be mainly described.

  FIG. 10 is a diagram for conceptually explaining the function of the menu control unit 260. As shown in FIG. As described in the first embodiment of the present invention, communication means is notified from the communication support device 10 to the communication support terminal 20A-1. At this time, it is also assumed that a plurality of communication means are notified. For example, it is assumed that a fixed telephone, an email, a chat, a mobile phone, and an answering machine are notified as communication means.

  In such a case, the menu control unit 260 may generate a selection menu for selecting any of the plurality of communication means. At this time, the menu control unit 260 may use the application name corresponding to the communication unit as a menu with reference to the application information stored by the application information storage unit 240. Then, the menu control unit 260 may cause the output unit 230 to output the selection menu. The selection menu shown in FIG. 10 includes a fixed telephone menu Z0, an e-mail menu Z1, a chat menu Z2, a mobile telephone menu Z3 and an answering menu Z4. In this example, each menu is arranged inside each sector having a center at the position E of the designated area.

  The display of the selection menu may be performed in any way. For example, the priority may be considered in displaying the selection menu. That is, the menu control unit 260 may control the selection menu based on the priorities associated with each of the plurality of communication means. For example, the menu control unit 260 may arrange the menu inside each of the equally divided fan shapes, or each fan whose central angle is an angle proportional to the priority associated with each of the plurality of communication means. A menu may be placed inside the shape.

  A specific example of the screen on which the selection menu is arranged will be described. FIG. 11 is a diagram showing an example of a selection menu displayed when the user U2 is selected. In the example shown in FIG. 11, it is assumed that chats and video calls are notified as communication means, but landlines are not notified. At this time, the menu control unit 260 may generate a selection menu including the chat menu M2 and the video telephone application menu M3, and the output unit 230 may display the selection menu around the user U2.

  Further, the menu control unit 260 may include the fixed telephone menu M1 in the selection menu and may add a mark (for example, x mark) indicating that the fixed telephone menu M1 has not been notified as communication means to the fixed telephone menu M1. Alternatively, menu control unit 260 may not include fixed telephone menu M1 in the selection menu. When the operation object Op of the user U1 performs an operation to select a menu, the input unit 210 inputs the selection operation, and the application activation unit 250 activates an application corresponding to the selected menu.

  FIG. 12 is a diagram showing an example of a selection menu displayed when the user U5 different from the user U2 is selected. In the example shown in FIG. 12, it is assumed that all fixed telephones, chats and video telephones are notified as communication means. At this time, the menu control unit 260 may generate a selection menu including the fixed telephone menu M1, the chat menu M2 and the video telephone application menu M3, and the output unit 230 may display the selection menu around the user U5.

  The display of such a selection menu may not be changed, but may be changed dynamically. For example, when a change occurs in the determination of the plurality of communication means by the communication means determination unit 150, the menu control unit 260 may dynamically update the selection menu based on the change. Specifically, when a change occurs in the communication means notified by the communication unit 120 in the communication support device 10 and received by the communication unit 220 in the communication support terminal 20B, the selection menu is updated based on the changed communication means You may

  It is also assumed that a plurality of communication means are associated with the same context (for example, No. 3 and No. 13 in FIG. 7). In such a case, the menu control unit 260 may include each of a plurality of communication means for the same context as a menu in the selection menu. No. The communication means "answer message recording" in 13 may be a function of the private branch exchange to leave a message message when the user selected from the video is using a fixed telephone.

  As described above, according to the communication support system 1B according to the second embodiment of the present invention, in addition to designating a specific area in the video through the user's input to the video, A communication menu is displayed around a specific area according to the change and the form of communication desired by the user. This enables the user to select a communication method while respecting the user's intention for more optimal communication. In addition, this makes it possible to realize a communication environment in which it is possible to change the way of speaking while looking at the situation even if it is a telecommunications with a remote base.

  Subsequently, the flow of the operation of the communication support terminal 20B according to the second embodiment of the present invention will be described. FIG. 13 is a flowchart showing the flow of the operation of the communication support terminal 20B according to the second embodiment of the present invention. As shown in FIG. 13, in the communication support terminal 20B, the communication unit 220 receives a notification of communication means from the communication support device 10 (S201). Subsequently, the menu control unit 260 generates a selection menu based on the notified communication means, and the output unit 230 displays the selection menu (S202).

  Subsequently, when the communication means (menu) is not selected from the selection menu ("No" in S203), the communication support terminal 20B returns to S201. As a result, when there is a change in the notified communication means while the selection menu is being displayed, the display content is also changed according to the change of the communication means. On the other hand, when communication means (menu) is selected from the selection menu ("Yes" in S203), the communication support terminal 20B starts the application corresponding to the selected communication means ("Yes" in S203). S204). When the application is started, communication is started (S205).

(3. Third Embodiment)
Subsequently, a third embodiment of the present invention will be described. A communication support system 1C according to the third embodiment of the present invention differs from the communication support system 1B according to the second embodiment of the present invention in the configuration of the communication support terminal 20C. Therefore, in the third embodiment of the present invention, the description of the communication support device 10 is omitted, and the communication support terminal 20C will be mainly described. First, a configuration example of a communication support terminal 20C according to a third embodiment of the present invention will be described with reference to FIG. FIG. 14 is a functional block diagram showing a configuration example of a communication support terminal 20C according to the third embodiment of the present invention.

  As shown in FIG. 14, the communication support terminal 20C according to the third embodiment of the present invention has a question selecting unit 270 in comparison with the communication support terminal 20B according to the second embodiment of the present invention. The point is different. Therefore, in the third embodiment of the present invention, among the components included in the communication support terminal 20C, the question selecting unit 270 will be mainly described.

  As described in the second embodiment of the present invention, the menu control unit 260 may generate a selection menu. Here, the question selecting unit 270 may select a predetermined question when the predetermined condition is satisfied. Then, the output unit 230 displays the selected question, and the menu control unit 260 performs an operation to input an answer to the question by the user, and when the answer is input by the input unit 210, selection is made based on the answer. You may control the menu. By controlling the selection menu, it is possible to capture the subtleties of the user that can not be followed just by estimating the context, and to support selection of a more realistic communication means.

  Note that a plurality of questions may be selected. Here, a specific example of the screen on which the question is displayed will be described. FIG. 15 is a diagram showing an example of a question displayed when a predetermined condition is satisfied. In FIG. 15, when the condition that the user U2 present in the designated area is in conversation is satisfied, the question selecting unit 270 selects the question Y1 such as “Do you want to participate in the conversation? And an example in which the output unit 230 displays this question.

  Here, when the menu control unit 260 selects to participate in the conversation (Yes), a menu corresponding to communication means capable of participating in the conversation (for example, a video telephone application menu, a telephone menu in the speaker state, etc.) May be included in the selection menu, and a menu (e.g., a mail menu, a chat menu, etc.) corresponding to communication means that can not participate in a conversation may be deleted from the selection menu. In addition, when participation in a conversation is selected (No), the selection menu may not be updated.

  In addition, the correspondence of a predetermined condition and a question is not limited to this example. For example, the predetermined condition may be a condition that the identified context is in conversation. In such a case, depending on the person in the conversation, it may be assumed that it is desired to participate in the videophone. Therefore, in such a case, the question selection unit 270 may select a question such as "Do you participate in a video call?"

  Further, the predetermined condition may be a condition that the user existing in the designated area has already started another communication. In such a case, it is also assumed that participation in the other communication is desired. Therefore, in such a case, the question selection unit 270 may select a question such as “Do you participate in another communication that has already been started?”.

  The predetermined condition may be a condition that a predetermined device exists in the designated area. In such a case, it is also assumed that the user is requested to use a predetermined device (for example, a headphone, etc.) if he / she does not want to hear the conversation content with the user present in the predetermined area. Therefore, in such a case, the question selecting unit 270 may select a question such as "Can my voice be heard around?"

  Subsequently, the flow of the operation of the communication support terminal 20C according to the third embodiment of the present invention will be described. FIG. 17 is a flowchart showing the flow of the operation of the communication support terminal 20C according to the third embodiment of the present invention. As shown in FIG. 17, in the communication support terminal 20C, the communication unit 220 receives a notification of communication means from the communication support device 10. Then, when the predetermined condition is not satisfied (“No” in S301), the menu control unit 260 generates a selection menu based on the notified communication means (S302).

  On the other hand, if the predetermined condition is satisfied (“Yes” in S301), the question selecting unit 270 selects a question, and the menu control unit 260 selects the question based on the selected question and the notified communication means. To generate a selection menu (S302). Subsequently, the output unit 230 displays a selection menu (S202). Subsequently, the menu control unit 260 proceeds to S306 if the answer to the question is not input ("No" in S304), and if the answer to the question is input ("Yes" in S304), The selected menu is updated according to the answer (S305).

  Subsequently, when the communication means (menu) is not selected from the selection menu ("No" in S306), the communication support terminal 20C returns to S301. On the other hand, when the communication means (menu) is selected from the selection menu ("Yes" in S306), the communication support terminal 20C starts the application corresponding to the selected communication means. When the application is activated, communication is started (S307).

  Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that those skilled in the art to which the present invention belongs can conceive of various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also fall within the technical scope of the present invention.

  For example, in the example described above, the menu is arranged in each of the equally divided fan shapes, and each menu in each fan shape whose central angle is an angle proportional to the priority associated with each of the plurality of communication means Described an example where is placed. However, the arrangement of the menu is not limited to such an example. For example, the menu control unit 260 may arrange the menu in a predetermined direction (for example, from upper left to lower right). In addition, each menu to be displayed may be accompanied by an animation such as dissolve. Furthermore, in addition to each menu, names of surrounding persons, presence information, etc. may be displayed together.

  Also, for example, a technique for improving the accuracy of video analysis for analyzing a designated area in a video may be used. More specifically, if the three-dimensional position information of each imaging device 50, the camera information, and the coordinate system of the base to be imaged are stored by the camera information storage unit 170, according to the designated position input by the input unit 210. Although the designated area is two-dimensional information, it is also possible to convert the designated area into three-dimensional information. Therefore, it is possible to analyze a certain area using videos from a plurality of angles of view, and it is possible to improve the accuracy of video analysis.

  Also, for example, row items and column items in context information (table) stored by the context information storage unit 190 may be editable from the user. This is to cope with addition / editing of a new communication device or application, and also to change in accordance with the user's preference (for example, prioritizing communication by e-mail). From this, when the user is constant to a certain extent, different tables may be held for each user.

  Also, it has been described above that the embodiment of the present invention can be effectively applied when bases are dispersed at a plurality of points in a general office environment. However, the embodiments of the present invention are also applicable to financial institutions, public institutions, general stores, and the like. For example, in the case where the head office remotely monitors each branch office with a video at a financial institution having a plurality of stores, etc., when the administrator discovers an abnormality, the case where communication is taken immediately or communication while paying attention to the customer etc. The embodiment of the present invention can be effectively applied even when selecting means. Thus, the embodiments of the present invention can be effectively applied even when the user does not directly know the person in the image.

1 (1A to 1C) Communication Support System 10 Communication Support Device 20 (20A to 20C) Communication Support Terminal 40 Communication Network 50 Imaging Device 110 Image Reception Unit 120 Communication Unit 130 Area Designation Unit 131 Object Information Storage Unit 140 Image Analysis Unit 150 Communication Means determination unit 160 Presence information acquisition unit 170 Camera information storage unit 180 Object information storage unit 190 Context information storage unit 210 Input unit 220 Communication unit 230 Output unit 240 Application information storage unit 250 Application activation unit 260 Menu control unit 270 Question selection unit Y1 Question

Claims (15)

A video analysis unit that analyzes a video captured by the imaging device to obtain a video analysis result including presence or absence of a predetermined object related to the user;
A presence information acquisition unit that acquires presence information indicating the state of the user;
A communication means determination unit that determines communication means with the user based on the video analysis result and the presence information;
Equipped with
When the communication means is selected by the operating body, an application corresponding to the communication means is activated,
The object includes the body of the user or the device of the user,
The presence information indicating the state of the user includes information indicating whether a fixed telephone is in use, information indicating whether a schedule of the user is included, presence information of the user, location estimation of the user Information, information indicating whether the PC is online or not, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, audio interface and video interface are in use Information indicating whether the input interface is not in use, or information indicating whether the time when the input interface has not been used exceeds a predetermined time,
Communication support device. The communication means determination unit specifies a context indicating the situation of the user based on the video analysis result and the presence information, and determines the communication means based on the context.
The communication support device according to claim 1. The communication means determination unit dynamically updates the communication means when at least one of the video analysis result and the presence information is changed.
The communication support device according to claim 1. The communication support device is
A communication unit for notifying the communication support terminal of the communication means;
The communication support device according to claim 1 . A video analysis unit that analyzes a video captured by the imaging device to obtain a video analysis result including presence or absence of a predetermined object related to the user;
A presence information acquisition unit that acquires presence information indicating the state of the user;
A communication means determination unit that determines communication means with the user based on the video analysis result and the presence information;
An area designation unit for designating a designated area in the video based on a designated position with respect to the video;
The video analysis unit obtains the video analysis result based on the designated area, and
The object includes the body of the user or the device of the user,
The presence information indicating the state of the user includes information indicating whether a fixed telephone is in use, information indicating whether a schedule of the user is included, presence information of the user, location estimation of the user Information, information indicating whether the PC is online or not, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, audio interface and video interface are in use Information indicating whether the input interface is not in use, or information indicating whether the time when the input interface has not been used exceeds a predetermined time,
Communication support device. Analyzing an image captured by one or more imaging devices to obtain an image analysis result including presence or absence of a predetermined object related to a user;
A presence information acquisition unit that acquires presence information indicating the state of the user;
Determining communication means with the user based on the video analysis result and the presence information;
Only including,
When the communication means is selected by the operating body, an application corresponding to the communication means is activated,
The object includes the body of the user or the device of the user,
The presence information indicating the state of the user includes information indicating whether a fixed telephone is in use, information indicating whether a schedule of the user is included, presence information of the user, location estimation of the user Information, information indicating whether the PC is online or not, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, audio interface and video interface are in use Information indicating whether the input interface is not in use, or information indicating whether the time when the input interface has not been used exceeds a predetermined time,
Communication support method. Computer,
A video analysis unit that analyzes video captured by one or a plurality of imaging devices and obtains a video analysis result including presence or absence of a predetermined object related to a user;
A presence information acquisition unit that acquires presence information indicating the state of the user;
A communication means determination unit that determines communication means with the user based on the video analysis result and the presence information;
Equipped with
When the communication means is selected by the operating body, an application corresponding to the communication means is activated,
The object includes the body of the user or the device of the user,
The presence information indicating the state of the user includes information indicating whether a fixed telephone is in use, information indicating whether a schedule of the user is included, presence information of the user, location estimation of the user Information, information indicating whether the PC is online or not, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, audio interface and video interface are in use Information indicating whether the input interface is not in use, or information indicating whether the time when the input interface has not been used exceeds a predetermined time,
A program for functioning as a communication support device. A communication support system comprising a communication support device and a communication support terminal, wherein
The communication support device is
A video analysis unit that analyzes video captured by one or a plurality of imaging devices and obtains a video analysis result including presence or absence of a predetermined object related to a user;
A presence information acquisition unit that acquires presence information indicating the state of the user;
A communication means determination unit that determines communication means with the user based on the video analysis result and the presence information;
The communication support terminal is
A communication unit that receives a notification of the communication means;
An application activation unit for activating an application corresponding to the communication means;
Equipped with
When the communication means is selected by the operating body, an application corresponding to the communication means is activated,
The object includes the body of the user or the device of the user,
The presence information indicating the state of the user includes information indicating whether a fixed telephone is in use, information indicating whether a schedule of the user is included, presence information of the user, location estimation of the user Information, information indicating whether the PC is online or not, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, audio interface and video interface are in use Information indicating whether the input interface is not in use, or information indicating whether the time when the input interface has not been used exceeds a predetermined time,
Communication support system The communication means determination unit specifies a context indicating the situation of the user based on the video analysis result and the presence information, and determines the communication means based on the context.
The communication support system according to claim 8. The communication support terminal includes an application activation unit that activates an application corresponding to the communication unit.
The communication support system according to claim 8. The communication support terminal includes a menu control unit that generates a selection menu for selecting any of the plurality of communication means determined by the communication means determination unit.
The communication support system according to claim 8. The menu control unit dynamically updates the selection menu based on the change when a change occurs in the determination of the plurality of communication units by the communication unit determination unit.
A communication support system according to claim 11 . The menu control unit controls the selection menu based on the priority associated with each of the plurality of communication means.
A communication support system according to claim 11 . The menu control unit controls the selection menu based on the answer when an answer to a predetermined question is input.
A communication support system according to claim 11 .   A communication support system comprising a communication support device and a communication support terminal, wherein
  The communication support device is
  A video analysis unit that analyzes video captured by one or a plurality of imaging devices and obtains a video analysis result including presence or absence of a predetermined object related to a user;
  A presence information acquisition unit that acquires presence information indicating the state of the user;
  A communication means determination unit that determines communication means with the user based on the video analysis result and the presence information;
  The communication support terminal is
  A communication unit that receives a notification of the communication means;
  An application activation unit for activating an application corresponding to the communication means;
  An output unit for displaying the video;
  An input unit for inputting a designated position for the video;
  Equipped with
  The communication means determination unit determines the communication means based on a designated area corresponding to the designated position, the video analysis result, and the presence information.
  The object includes the body of the user or the device of the user,
  The presence information indicating the state of the user includes information indicating whether a fixed telephone is in use, information indicating whether a schedule of the user is included, presence information of the user, location estimation of the user Information, information indicating whether the PC is online or not, information indicating whether the user is temporarily leaving, information indicating whether the video telephone application is in use, audio interface and video interface are in use Information indicating whether the input interface is not in use, or information indicating whether the time when the input interface has not been used exceeds a predetermined time,
  Communication support system.

Images