JP2018174435A - Relay apparatus, transmission system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that a video conference terminal can display each image of the plurality of parties on a plurality respective display regions of one screen when there are a plurality of parties of a call, however, there is present an image of the other party not placed on one screen when there are parties exceeding the number of display areas of the screen, and therefore, it is difficult to realize a state close to a conversation at the same place, particularly, when an image of a partner side who is wished to know a situation is not included in the screen in images of the plurality of parties.SOLUTION: A relay device 30 receives priority allocation request information indicating a request for an image to be preferentially allocated on one screen from a terminal 10 (S86). Then, the relay device 30 performs processing of arranging an image related to image data sent from the other terminal on the screen on the basis of priority arrangement request information (S102). Therefore, it is possible to display an image of a party who is wished to know the situation in particular.SELECTED DRAWING: Figure 13

Description

  The present invention relates to a relay device, a transmission system, an image processing method, and a program.

  Video conferencing systems for conducting video conferences (video conferences) with remote locations have become widespread in response to requests for reductions in business trip expenses and business trip times. In such a video conference system, image data and sound data are transmitted and received between a plurality of video conference terminals, thereby realizing a state close to a conversation at the same place.

  In addition, when there are a plurality of call partners, the video conference terminal can display each image of the plurality of opponents in each of a plurality of display areas on one screen (see Patent Document 1).

  However, when calling with a number of opponents exceeding the number of display areas on one screen, there is an image of the other party that is not arranged on one screen. For this reason, it is difficult to realize a state close to having a conversation in the same place, especially when the image of the other party who wants to know the situation is not included in one screen among the images of the other party. There arises the problem of being.

  According to the first aspect of the present invention, specific image data is created by arranging each image related to each image data transmitted from each of a plurality of transmission terminals on one screen, and the specific transmission data is specified for the specific transmission terminal. A relay device that transmits the image data of the image data, the transmission source of the image data related to the priority image that is the image to be preferentially arranged on the one screen among the images related to the image data from the specific transmission terminal First receiving means for receiving transmission source identification information for identifying each of the plurality of transmission terminals, second receiving means for receiving each image data transmitted from each of the plurality of transmission terminals, and the second receiving means The transmission source identification received by the first receiving means when the number of transmission sources of the respective image data received by the computer exceeds the upper limit of the number of images that can be arranged on the one screen. Indicated by information An arrangement unit that preferentially arranges the priority image related to the image data transmitted from the transmission source in a predetermined display area of the one screen, and the arrangement unit arranges the priority image with respect to the specific transmission terminal. And a transmission means for transmitting the specific image data created in (1).

  As described above, according to the present invention, by transmitting specific image data including a priority image, which is an image to be preferentially arranged on one screen, to a specific transmission terminal, a conversation can be performed at the same place. There is an effect that it is possible to realize a state close to performing.

1 is a schematic diagram of a transmission system. It is the conceptual diagram which showed the transmission / reception state of the image data in a transmission system, sound data, and various management information. It is a conceptual diagram explaining the image quality of image data. It is an external view of the terminal which concerns on this embodiment. It is a hardware block diagram of the terminal which concerns on this embodiment. It is a hardware block diagram of the management system, relay apparatus, or program provision system which concerns on this embodiment. It is a functional block diagram of the transmission terminal, relay apparatus, and transmission management system which comprise the transmission system which concerns on this embodiment. It is a conceptual diagram which shows an arrangement | positioning management table. It is a conceptual diagram which shows an authentication management table. It is a conceptual diagram which shows a session management table. It is the sequence diagram which showed the participation process of a meeting. It is a figure showing an example of a reservation list screen. It is the sequence figure which showed the processing of priority arrangement request. It is the sequence diagram which showed the process of relay. It is the flowchart which showed the process of image arrangement | positioning. FIG. 6 is a sequence diagram illustrating a process for preferentially arranging images. It is the figure which showed an example of the call screen before priority arrangement | positioning. It is the figure which showed an example of the call screen after priority arrangement | positioning.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
First, a first embodiment of the present invention will be described.
<< Overall Configuration of Embodiment >>
FIG. 1 is a schematic diagram of a transmission system 1 according to this embodiment of the present invention. FIG. 2 is a conceptual diagram showing a state of transmission / reception of image data, sound data, and various management information in the transmission system. FIG. 3 is a conceptual diagram illustrating the image quality of image data.

  The transmission system includes a data providing system that transmits content data in one direction from one transmission terminal to the other transmission terminal via the transmission management system, and information between a plurality of transmission terminals via the transmission management system. Includes a communication system that communicates emotions and the like. This communication system is a system for mutually transmitting information, emotions and the like between a plurality of communication terminals via a communication management system, and examples thereof include a video conference system and a video phone system.

  In the present embodiment, assuming a video conference system as an example of a communication system, a video conference management system as an example of a communication management system, and a video conference terminal as an example of a communication terminal, a transmission system, a transmission management system, The transmission terminal will be described. That is, this does not mean that the transmission system, transmission management system, and transmission terminal of the present invention are limited to this embodiment.

  First, the transmission system 1 shown in FIG. 1 includes a plurality of transmission terminals (10aa, 10ab,...), A display (120aa, 120ab,...) For each transmission terminal (10aa, 10ab,. The apparatus (30a, 30b, 30c, 30d), the transmission management system 50, the program providing system 90, and the maintenance system 100 are constructed.

  The plurality of terminals 10 transmit and receive image data and sound data as an example of content data.

  In the following, “transmission terminal” is simply represented as “terminal”, and “transmission management system” is simply represented as “management system”. Further, an arbitrary terminal among the plurality of terminals (10aa, 10ab,...) Is represented as “terminal 10”, and an arbitrary display among the plurality of displays (120aa, 120ab,...) Is represented as “display 120”. An arbitrary relay device among the plurality of relay devices (30a, 30b, 30c, 30d) is represented as “relay device 30”. Further, a terminal as a request source that requests the start of a video conference is represented as a “request source terminal”, and a terminal as a request destination (relay destination) is represented as a “destination terminal”.

  As shown in FIG. 2, in the transmission system 1, a management information session sei for transmitting and receiving various types of management information is transmitted between the request source terminal and the destination terminal via the management system 50. Established. Further, between the request source terminal and the destination terminal, four pieces of data of high resolution image data, medium resolution image data, low resolution image data, and sound data are transmitted and received via the relay device 30. Four sessions are established. Here, these four sessions are collectively shown as an image / sound data session sed. Note that the image / sound data session sed does not necessarily have to be four sessions, and may have a smaller or larger number of sessions than the four sessions.

  Here, the resolution of the image data handled in the present embodiment will be described. As shown in FIG. 3 (a), the horizontal image is composed of 160 pixels and the vertical image is 120 pixels, and a low-resolution image serving as a base image and a horizontal image as illustrated in FIG. 3 (b). There are a medium resolution image consisting of 320 pixels and 240 pixels vertically, and a high resolution image consisting of 640 pixels horizontally and 480 pixels vertically as shown in FIG. Among these, when passing through a narrow band route, low-quality image data consisting only of low-resolution image data serving as a base image is relayed. When the band is relatively wide, low-resolution image data serving as a base image and medium-quality image data composed of medium-resolution image data are relayed. When the bandwidth is very wide, high-resolution image data composed of low-resolution image data serving as base image quality, intermediate-resolution image data, and high-resolution image data is relayed.

  The relay device 30 shown in FIG. 1 relays content data among a plurality of terminals 10. The management system 50 centrally manages the login authentication from the terminal 10, the management of the call status of the terminal 10, the management of the destination list, and the communication status of the relay device 30. The image of the image data may be a moving image or a still image, or both a moving image and a still image.

  A plurality of routers (70a, 70b, 70c, 70d, 70ab, 70cd) select an optimum route for image data and sound data. In the following, an arbitrary router among the routers (70a, 70b, 70c, 70d, 70ab, 70cd) is represented as “router 70”.

  The program providing system 90 includes an HD (Hard Disk) 204 described below, and stores a terminal program for causing the terminal 10 to realize various functions (or to cause the terminal 10 to function as various means). The terminal program can be transmitted to The HD 204 of the program providing system 90 also stores a relay device program for causing the relay device 30 to realize various functions (or to cause the relay device 30 to function as various means). The relay device program can be transmitted. Further, the HD 204 of the program providing system 90 also stores a transmission management program for realizing various functions in the management system 50 (or causing the management system 50 to function as various means). A transmission management program can be transmitted.

  The maintenance system 100 is a computer for performing maintenance, management, or maintenance of at least one of the terminal 10, the relay device 30, the management system 50, and the program providing system 90. For example, when the maintenance system 100 is installed in the country and the terminal 10, the relay device 30, the management system 50, or the program providing system 90 is installed outside the country, the maintenance system 100 is remotely connected via the communication network 2. Maintenance such as maintenance, management, and maintenance of at least one of the terminal 10, the relay device 30, the management system 50, and the program providing system 90 is performed. In addition, the maintenance system 100 does not go through the communication network 2, but includes a model number, a manufacturing number, a sales destination, a maintenance check, at least one of the terminal 10, the relay device 30, the management system 50, and the program providing system 90. Or, maintenance such as failure history management is performed.

  By the way, the terminals (10aa, 10ab, 10ac,...), The relay device 30a, and the router 70a are communicably connected via the LAN 2a. The terminals (10ba, 10bb, 10bc,...), The relay device 30b, and the router 70b are communicably connected via a LAN 2b. The LAN 2a and the LAN 2b are communicably connected by a dedicated line 2ab including a router 70ab, and are constructed in a predetermined area A. For example, the region A is Japan, the LAN 2a is constructed in the Tokyo office, and the LAN 2b is constructed in the Osaka office.

  On the other hand, the terminals (10ca, 10cb, 10cc,...), The relay device 30c, and the router 70c are communicably connected via a LAN 2c. The terminal 10d (a, 10db, 10dc,...), The relay device 30d, and the router 70d are communicably connected via a LAN 2d. The LAN 2c and the LAN 2d are communicably connected via a dedicated line 2cd including the router 70cd, and are constructed in a predetermined area B. For example, region B is the United States of America, LAN 2c is built in a New York office, and LAN 2d is Washington D.C. C. Is built in the office. Area A and area B are connected to each other via routers (70ab, 70cd) via the Internet 2i.

  In addition, the management system 50 and the program providing system 90 are communicably connected to the terminal 10 and the relay device 30 via the Internet 2i. The management system 50 and the program providing system 90 may be installed in the region A or the region B, or may be installed in a region other than these.

  In the present embodiment, the communication network 2 of the present embodiment is constructed by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d. The communication network 2 may have a place where wireless communication such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark) is performed in addition to wired communication. Further, in FIG. 1, four sets of numbers shown below each terminal 10, each relay device 30, management system 50, each router 70, and program providing system 90 are simplified IP addresses in general IPv4. Is shown. For example, the IP address of the terminal 10aa is “1.2.1.3”. In addition, IPv6 may be used instead of IPv4, but IPv4 will be used for the sake of simplicity.

  In addition, each terminal 10 is not only a call between a plurality of business establishments or a call between different rooms in the same business establishment, but a call within the same room or a call between the outdoors and indoors or outdoors and outdoors. May be used. When each terminal 10 is used outdoors, wireless communication such as a cellular phone communication network is performed.

<< Hardware Configuration of Embodiment >>
Next, the hardware configuration of this embodiment will be described. FIG. 4 is an external view of the terminal 10 according to the present embodiment. As shown in FIG. 4, the terminal 10 includes a housing 1100, an arm 1200, and a camera housing 1300. Among these, the front wall surface 1110 of the housing 1100 is provided with an air intake surface formed by a plurality of air intake holes, and the rear wall surface 1120 of the housing 1100 is provided with an air exhaust surface formed with a plurality of air exhaust holes. 1121 is provided. Thus, by driving a cooling fan built in the housing 1100, the outside air behind the transmission terminal 10 can be taken in via the intake surface and exhausted to the rear of the transmission terminal 10 via the exhaust surface 1121. A sound collecting hole 1131 is formed in the right side wall surface 1130 of the housing 1100, and sounds such as sound, sound, noise, and heartbeat can be collected by a built-in microphone 114 described later.

  An operation panel 1150 is formed on the right wall surface 1130 side of the housing 1100. The operation panel 1150 is provided with a plurality of operation buttons (108a to 108e) to be described later, a power switch 109 to be described later, and an alarm lamp 119 to be described later, and outputs sound from a built-in speaker 115 to be described later. A sound output surface 1151 formed by a plurality of sound output holes for passing through is formed. Further, a housing 1160 as a recess for housing the arm 1200 and the camera housing 1300 is formed on the left wall surface 1140 side of the housing 1100. The right side wall surface 1130 of the housing 1100 is provided with a plurality of connection ports (1132a to 1132c) for electrically connecting cables to an external device connection I / F 118 described later. On the other hand, the left wall surface 1140 of the housing 1100 is provided with a connection port (not shown) for electrically connecting the cable 120c for the display 120 to an external device connection I / F 118 described later.

  In the following description, “operation button 108” is used when an arbitrary operation button is indicated among the operation buttons (108a to 108e), and “an arbitrary connection port is indicated among the connection ports (1132a to 1132c)”. This will be described using the connection port 1132 ”.

  Next, the arm 1200 is attached to the housing 1100 via a torque hinge 1210, and the arm 1200 is configured to be able to rotate in the vertical direction with respect to the housing 1100 within a tilt angle θ1 of 135 degrees. ing. FIG. 4 shows a state where the tilt angle θ1 is 90 degrees. The camera housing 1300 is provided with a built-in camera 112, which will be described later, and can capture images of users, documents, rooms, and the like. A torque hinge 1310 is formed in the camera housing 1300. The camera housing 1300 is attached to the arm 1200 via a torque hinge 1310. The camera housing 1300 is attached to the arm 1200 via a torque hinge 1310. The pan angle θ2 of ± 180 degrees is assumed with respect to the arm 1200, with the state shown in FIG. And a tilt angle θ3 of ± 45 degrees can be rotated in the vertical and horizontal directions.

  Note that the external view of FIG. 4 is merely an example and is not limited to this external view. The transmission terminal 10 may be, for example, a general-purpose PC, a smartphone, or a tablet terminal. The camera and microphone do not necessarily have to be built-in and may be externally attached. Further, since the transmission management system 50 has the same appearance as that of a general server computer, description of the appearance is omitted.

  FIG. 5 is a hardware configuration diagram of the terminal 10 according to this embodiment of the present invention. As shown in FIG. 5, the terminal 10 of the present embodiment has programs used for driving the CPU 101 such as a CPU (Central Processing Unit) 101 and IPL (Initial Program Loader) that control the operation of the entire terminal 10. ROM (Read Only Memory) 102, RAM (Random Access Memory) 103 used as a work area for the CPU 101, flash memory 104 for storing various data such as terminal programs, image data, and sound data, and control of the CPU 101 The SSD (Solid State Drive) 105 that controls the reading or writing of various data to the flash memory 104 according to the above, the media drive 107 that controls the reading or writing (storage) of data to the recording medium 106 such as the flash memory, and the destination To be operated when selecting A button 108, a power switch 109 for switching on / off the power of the terminal 10, and a network I / F (Interface) 111 for data transmission using the communication network 2 are provided.

  The terminal 10 also includes a built-in camera 112 that captures an image of a subject under the control of the CPU 101 to obtain image data, an image sensor I / F 113 that controls driving of the camera 112, a built-in microphone 114 that inputs sound, a sound The built-in speaker 115 that outputs the sound, the sound input / output I / F 116 that processes the input / output of the sound signal between the microphone 114 and the speaker 115 according to the control of the CPU 101, and the image data on the external display 120 according to the control of the CPU 101 A display I / F 117 for transmission, an external device connection I / F 118 for connecting various external devices, and an address bus and a data bus for electrically connecting the above components as shown in FIG. The bus line 110 is provided.

  The display 120 is a display unit configured by a liquid crystal or an organic EL that displays an image of a subject, an operation icon, and the like. The display 120 is connected to the display I / F 117 by a cable 120c. The cable 120c may be an analog RGB (VGA) signal cable, a component video cable, or an HDMI (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal. It may be a cable.

  The camera 112 includes a lens and a solid-state image sensor that converts light into electric charges and digitizes a subject image (video). As the solid-state image sensor, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) is used. Etc. are used.

  External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 118 by a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven in preference to the built-in camera 112 under the control of the CPU 101. Similarly, when an external microphone is connected or when an external speaker is connected, each of the external microphones and the built-in speaker 115 is given priority over the internal microphone 114 and the internal speaker 115 according to the control of the CPU 101. An external speaker is driven.

  The recording medium 106 is detachable from the terminal 10. Further, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 101, not only the flash memory 104 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

  Further, the terminal program may be recorded in a computer-readable recording medium such as the recording medium 106 and distributed as a file in an installable or executable format. The terminal program may be stored in the ROM 102 instead of the flash memory 104.

  FIG. 6 is a hardware configuration diagram of the management system according to this embodiment of the present invention. The management system 50 stores various data such as a CPU 201 that controls the operation of the entire management system 50, a ROM 202 that stores programs used to drive the CPU 201 such as an IPL, a RAM 203 that is used as a work area for the CPU 201, and a transmission management program. The HD 204 to be stored, an HDD (Hard Disk Drive) 205 that controls reading or writing of various data to the HD 204 according to the control of the CPU 201, and a media drive 207 that controls reading or writing (storage) of data to a recording medium 206 such as a flash memory. , A display 208 that displays various information such as a cursor, menu, window, character, or image, a network I / F 209 for transmitting data using the communication network 2, characters, numerical values, A keyboard 211 having a plurality of keys for inputting seed instructions, a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc., and a CD-ROM as an example of a removable recording medium (Compact Disc Read Only Memory) 213, a CD-ROM drive 214 for controlling the reading or writing of various data, and an address bus for electrically connecting the above components as shown in FIG. A bus line 210 such as a data bus is provided.

  The transmission management program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213. . The transmission management program may be stored in the ROM 202 instead of the HD 204.

  Further, since the relay device 30 has the same hardware configuration as that of the management system 50, the description thereof is omitted. However, a relay device program for controlling the relay device 30 is recorded in the HD 204. Also in this case, the relay device program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good. The relay device program may be stored in the ROM 202 instead of the HD 204.

  Further, since the program providing system 90 and the maintenance system 100 have the same hardware configuration as the management system 50, the description thereof is omitted. However, a program providing program for controlling the program providing system 90 is recorded in the HD 204. Also in this case, the program providing program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Good. The program providing system program may be stored in the ROM 202 instead of the HD 204.

  As another example of the detachable recording medium, the recording medium is provided by being recorded on a computer-readable recording medium such as a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), or a Blu-ray Disc. May be.

<< Functional Configuration of Embodiment >>
Next, the functional configuration of this embodiment will be described. FIG. 7 is a functional block diagram of a transmission terminal, a relay device, and a transmission management system that constitute the transmission system according to the present embodiment. In FIG. 7, the terminal 10, the relay device 30, and the management system 50 are connected so that data communication can be performed via the communication network 2. Further, the program providing system 90 and the maintenance system 100 shown in FIG. 1 are omitted in FIG. 7 because they are not directly related to the video conference communication.

<Functional configuration of terminal>
The terminal 10 includes a transmission / reception unit 11, a reception unit 12, an imaging unit 14, a sound input unit 15a, a sound output unit 15b, a display control unit 17, a delay detection unit 18, and a storage / read processing unit 19. Each of these units is a function realized by any one of the constituent elements shown in FIG. 5 being operated by a command from the CPU 101 according to the terminal program developed from the flash memory 104 onto the RAM 103, or It is a means to function. Further, the terminal 10 has a storage unit 1000 realized by the RAM 103 shown in FIG. 5 and the flash memory 104 shown in FIG.

(Functional configuration of terminal)
Next, each functional configuration of the terminal 10 will be described in detail with reference to FIGS. 5 and 7. In the following, in describing each functional configuration of the terminal 10, among the components illustrated in FIG. 5, a relationship with main components for realizing each functional configuration of the terminal 10 will also be described. .

  The transmission / reception unit 11 of the terminal 10 shown in FIG. 7 is realized by a command from the CPU 101 shown in FIG. 5 and the network I / F 111 shown in FIG. Sends and receives various data (information) to and from the terminal, device or system.

  The accepting unit 12 is realized by the command from the CPU 101 shown in FIG. 5 and the operation buttons 108 and the power switch 109 shown in FIG. 5, and accepts various inputs by the user. For example, when the user turns on the power switch 109 shown in FIG. 5, the accepting unit 12 shown in FIG. 7 accepts the power on and turns on the power.

  The imaging unit 14 is realized by a command from the CPU 101 illustrated in FIG. 5 and the camera 112 and the image sensor I / F 113 illustrated in FIG. 5. Output data.

  The sound input unit 15a is realized by the instruction from the CPU 101 shown in FIG. 5 and the sound input / output I / F 116 shown in FIG. 5, and the user's sound is converted into a sound signal by the microphone 140. Thereafter, sound data related to the sound signal is input. The sound output unit 15b is realized by a command from the CPU 101 shown in FIG. 5 and the sound input / output I / F 116 shown in FIG. 5, and outputs a sound signal related to the sound data to the speaker 150. Sound is output from.

  The display control unit 17 is realized by the instruction from the CPU 101 shown in FIG. 5 and the display I / F 117 shown in FIG. 5. As will be described later, the received image data having different resolutions are combined. Control for transmitting the combined image data to the display 120 is performed.

  The storage / reading processing unit 19 is executed by the instruction from the CPU 101 shown in FIG. 5 and the instruction from the SSD 105 shown in FIG. 5 or realized by the instruction from the CPU 101, and stores various data in the storage unit 1000. A process of reading various data stored in the unit 1000 is performed. The storage unit 1000 stores a terminal ID (Identification) for identifying the own terminal (terminal 10), a password, and the like.

  Further, the storage unit 1000 overwrites and stores image data and sound data received when a call with the destination terminal is made. Among these, an image is displayed on the display 120 by the image data before being overwritten, and a sound is output from the speaker 150 by the sound data before being overwritten.

  The terminal ID and the relay device ID and session ID, which will be described later, are identification information such as language, characters, symbols, and various signs used to uniquely identify the terminal 10, the relay device 30, and the communication session, respectively. Indicates. The terminal ID, the relay device ID, and the session ID may be identification information in which at least two of the language, characters, symbols, and various indicia are combined.

  Further, instead of the terminal ID, a user ID for identifying the user of the terminal 10 may be used. The terminal ID and user ID are examples of participation target identification information. The terminal ID, user ID, relay device ID, and session ID are examples of terminal identification information, user identification information, relay device identification information, and session identification information, respectively.

<Functional configuration of relay device>
The relay device 30 includes a transmission / reception unit 31, a determination unit 36, an arrangement unit 37, and a storage / read processing unit 39. Each of these units functions or functions realized by any one of the constituent elements shown in FIG. 6 operating according to a command from the CPU 201 according to the relay device program expanded from the HD 204 onto the RAM 203. Means. In addition, the relay device 30 includes a storage unit 3000 constructed by the RAM 203 shown in FIG. 6 and / or the HD 204 shown in FIG.

(Placement management table)
Furthermore, an arrangement management DB 3001 configured by an arrangement management table as shown in FIG. 8 is constructed in the storage unit 3000. In this arrangement management table, for each session ID, the terminal ID of the request source terminal, the terminal ID of the transmission source terminal, and presence / absence of priority arrangement are associated and managed. Among these, the session ID is an example of session identification information for identifying a communication session for content data such as image data and sound data transmitted and received between terminals. The terminal ID of the request source terminal is an example of request source identification information for identifying a request side terminal that requests to place a specific image preferentially on one screen (also referred to as “one image”). Hereinafter, an image that is preferentially arranged is also referred to as a priority image. The terminal ID of the transmission source terminal is an example of transmission source identification information for identifying the transmission source terminal of the image data related to the priority image requested to be preferentially arranged by the request source terminal. Presence / absence of priority placement is information indicating, for each transmission source terminal of image data related to the priority image, whether or not the priority image is preferentially placed on one screen.

  For example, in the first record of the arrangement management table shown in FIG. 8, based on a request from a request source terminal with a terminal ID “01aa” that communicates in a communication session with a session ID “se01”, the terminal An image related to the image data from the transmission source terminal whose ID is “01ab” is indicated as “◯” as a priority placement target. On the other hand, the second record relates to the image data from the transmission source terminal whose terminal ID is “01ac” based on the request from the same request source terminal that communicates in the same communication session as the first record. The image indicates that “x” is not a priority placement target.

(Each functional configuration of the relay device)
Next, each functional configuration of the relay device 30 will be described in detail. In the following, in describing each functional configuration of the relay device 30, among the components illustrated in FIG. 6, the relationship with the main components for realizing each functional configuration of the relay device 30 is also described. explain.

  The transmission / reception unit 31 of the relay device 30 shown in FIG. 7 is realized by the command from the CPU 201 shown in FIG. 6 and the network I / F 209 shown in FIG. Various data (information) is transmitted / received to / from other terminals, devices, or systems.

  The determination unit 36 is realized by a command from the CPU 201 shown in FIG. 6 and makes various determinations. For example, the determination unit 36 determines whether the number of participation in the meeting exceeds the upper limit of the number of images that can be arranged on one screen. Further, the determination unit 36 determines whether there is a priority arrangement request from the terminal 10.

  The arrangement unit 37 is realized by a command from the CPU 201 illustrated in FIG. 6, and the number of transmission sources of each image data received by the transmission / reception unit 31 is the upper limit of the number of images that can be arranged on one screen. Is exceeded, the priority image related to the image data transmitted from the transmission source indicated by the terminal ID received by the transmission / reception unit 31 is preferentially arranged in a predetermined display area of one screen.

  The storage / reading processing unit 39 is realized by an instruction from the CPU 201 illustrated in FIG. 6 and the HDD 205 illustrated in FIG. 6. The storage / reading processing unit 39 stores various data in the storage unit 3000 or is stored in the storage unit 3000. A process of reading various data is performed.

<Functional configuration of management system>
The management system 50 includes a transmission / reception unit 51, an authentication unit 52, a selection unit 56, a creation unit 57, and a storage / read processing unit 59. Each of these units functions or functions realized by any one of the constituent elements shown in FIG. 6 operating according to a command from the CPU 201 according to the management system program expanded from the HD 204 onto the RAM 203. Means. Further, the management system 50 has a storage unit 5000 constructed by the HD 204 shown in FIG.

(Authentication management table)
Furthermore, in the storage unit 5000, an authentication management DB 5001 configured by an authentication management table as shown in FIG. In this authentication management table, passwords, terminal names, terminal user e-mail addresses, and terminal IP addresses are managed in association with each terminal ID of all terminals 10 managed by the management system 50. . For example, in the authentication management table shown in FIG. 9, the terminal ID of the terminal 10aa is “01aa” and the password is “aaaa”.

(Session management table)
In the storage unit 5000, a session management DB 5002 configured by a session management table as shown in FIG. 10 is constructed. In this session management table, for each reservation ID described above, a session ID for identifying a communication session for content data between terminals, a relay device ID of the relay device 30 used for relaying content data, and participation in a call A terminal ID for identifying the terminal inside is associated and managed.

(Functional configuration of the management system)
Next, each functional configuration of the management system 50 will be described in detail. In the following, in describing each functional configuration of the management system 50, among the components shown in FIG. 6, the relationship with the main components for realizing each functional configuration of the management system 50 is also described. explain.

  The transmission / reception unit 51 is executed by the command from the CPU 201 shown in FIG. 6 and the network I / F 209 shown in FIG. Information).

  The authentication unit 52 is realized by a command from the CPU 201 shown in FIG. 6, and uses the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys. Terminal authentication is performed by searching the authentication management DB 5001 and determining whether the same terminal ID and password are managed in the authentication management DB 5001.

  The selection unit 56 is realized by a command from the CPU 201 illustrated in FIG. 6, and selects a predetermined relay device from the plurality of relay devices 30. For example, the selection unit 56 selects the relay device 30 whose IP address is close to the terminal 10 that performs communication, or the relay device 30 that is installed in the same country as the terminal 10 that performs communication.

  The creation unit 57 is realized by a command from the CPU 201 illustrated in FIG. 6, and creates a session ID for identifying a communication session between the plurality of terminals 10.

  The storage / reading processing unit 59 is executed by the instruction from the CPU 201 shown in FIG. 6 and the HDD 205 shown in FIG. 6, and stores various data in the storage unit 5000 or stored in the storage unit 5000. A process of reading various data is performed.

<< Processing or Operation of Embodiment >>
Next, a processing method in the transmission system 1 according to the present embodiment will be described using FIGS. 11 to 18. First, the process related to the start of the conference will be described with reference to FIGS. 11, 12, and 17. FIG. 11 is a sequence diagram illustrating a conference participation process. FIG. 12 is a diagram showing an example of a reservation list screen. FIG. 17 is a diagram showing an example of a call screen before priority placement.

  The reservation list screen is a screen displayed on the display 120 by the display control unit 17. On the reservation list screen, reservation content information is displayed for each conference. The reservation content information includes reservation content information including a conference name (such as “HR decision meeting”), a terminal ID (“01aa”, etc.) of a terminal that is reserved for participation, and a scheduled date and time of the conference. Each reservation content information includes a “participation” button for actually participating in the conference.

  First, when the user presses the “participation” button in the reservation content information indicating the conference to be joined in the reservation list screen, the accepting unit 12 accepts participation in the conference (step S61). Then, the transmission / reception unit 11 transmits participation request information indicating a participation request to the management system 50 (step S62). This participation request information includes a terminal ID and password for identifying the terminal 10 that is the requesting terminal, and a reservation ID. As long as the user presses the “join” button, the transmission / reception unit 11 automatically transmits the terminal ID, password, and reservation ID without inputting the terminal ID, password, and reservation ID. This reservation ID is the ID received in association with (associated with) the reservation content information in step S31 described above. Thereby, the transmission / reception part 51 of the management system 50 receives participation request information.

  Next, the authentication unit 52 of the management system 50 searches the authentication management DB 5001 (see FIG. 9) using the requesting terminal ID and password received by the transmission / reception unit 51 as search keys, and the same terminal in the authentication management DB 5001 Authentication is performed by determining whether a set of ID and password is managed (step 63).

  Next, the creating unit 57 generates a session ID for identifying a communication session for communicating content data such as image data and sound data (step S64). Then, the selection unit 56 selects a predetermined relay device 30 among the plurality of relay devices 30a to 30d illustrated in FIG. 1 (step S65). The selection unit 56 selects, for example, a relay device having an IP address close to (similar to) the IP address of the requesting terminal 10.

  Next, the storage / reading processing unit 59 associates the session ID created in step S64, the relay device ID for identifying the relay device selected in step S65, and the request source terminal ID received in step S62. Then, a new record is created, registered and managed in the session management DB 5002 (step S66). If another terminal has already started the conference, the storage / reading processing unit 59 has received the “terminal ID of participating terminal” field portion of the record already created in step S62. The terminal ID of the request source is additionally registered.

  Next, the transmission / reception unit 51 transmits the session ID and relay device ID registered in step S66 to the terminal 10 (step S67). Thereby, the transmission / reception unit 11 of the terminal 10 receives the session ID and the relay device ID.

  As described above, a communication session for content data for transmitting / receiving content data is established between the transmission / reception unit 11 of the terminal 10 and the transmission / reception unit 31 of the relay device 30 selected in step S65 (step S68). When the communication session of each terminal 10 scheduled to participate is established, the display control unit 17 causes the display 120 to display a call screen as shown in FIG. FIG. 17A shows an image related to image data from the terminal 10 at the 8 bases among the terminals 10 at the 10 bases. Also, the image at the bottom right is the image of the local site. At this time, the management system 50 gives priority to its own base and randomly arranges the remaining images on one screen.

  Next, priority placement processing will be described with reference to FIGS. FIG. 13 is a sequence diagram showing the priority placement request process.

  As described above, after the communication session of each terminal 10 is established, the storage / read processing unit 59 of the management system 50 is associated with a predetermined session ID from the session management DB 5002 as shown in FIG. All the terminal IDs of the terminals participating in the call are read (step S81). Further, the storage / reading processing unit 59 reads the corresponding terminal name by searching the authentication management DB 5001 using each terminal ID read in step S81 as a search key (step S82). Then, the transmission / reception unit 51 associates each terminal ID read in step S81 with each terminal name read in step S82, and transmits all the sets of terminal IDs and terminal names to the terminal 10. (Step S83). Thereby, the transmission / reception unit 11 of the terminal 10 receives all the sets of the terminal ID and the terminal name.

  Next, in the terminal 10, when the user presses the “participation list” button b <b> 1 shown in FIG. 17A, the receiving unit 12 receives the pressing, and the display control unit 17 controls the display 120. A participation list r1 as shown in FIG. 17B is pulled up (step S84). In the participation list r1, a check box for selecting a terminal that is a transmission source of image data related to an image to be preferentially arranged is displayed for each terminal name of a participating terminal. Here, when the user checks the check box of the terminal that is the transmission source of the image data related to the image to be preferentially arranged, that is, the image to be preferentially displayed on the own terminal, and presses the “confirm” button b3, The accepting unit 12 accepts selection of priority placement (step S85). When the “select all” button b2 is pressed, all the check boxes are checked.

  Next, the transmission / reception unit 11 transmits priority placement request information to the relay device 30 (step S86). In this priority allocation request information, the session ID of the communication session in communication, the terminal ID of the request source terminal (own terminal) of the priority allocation, and the terminal IDs of the transmission source terminals related to all the terminal names displayed in the participation list b1 , And the presence / absence of priority display of each terminal ID. Thereby, the transmission / reception unit 31 of the relay device 30 receives the priority arrangement request information. Then, the storage / reading processing unit 39 of the relay device 30 stores and manages the priority arrangement request information received in step S86 in the arrangement management DB 3001 (step S87). As a result, a record is created for each terminal ID of the transmission source terminal in the arrangement management table constituting the arrangement management DB 3001, and each information included in the priority arrangement request information is managed as shown in FIG. Is done.

  Next, the relay process will be described with reference to FIGS. 14 to 16 and FIG. In FIG. 14, relaying of content data between the terminal 10 aa and the terminal 10 ab among the terminals 10 will be described.

  As illustrated in FIG. 14, the terminal 10ab transmits content data to the relay device 30 (step S101). The content data includes the terminal ID “01ab” of the terminal 10ab that is the transmission source. Thereby, the transmission / reception unit 31 of the relay device 30 receives the content data.

  Next, the relay device 30 performs image arrangement processing (step S102). Here, the image layout process will be described in detail with reference to FIGS. 15 and 16. FIG. 15 is a flowchart showing image arrangement processing. FIG. 16 is a sequence diagram illustrating processing for preferential placement of images.

  As shown in FIG. 15, the transmission / reception unit 31 of the relay device 30 makes a search request to the management system 50 via the communication network 2 by using the session ID received in step S86 as a search key. 51, the terminal ID of the participating terminal 10 corresponding in the session management DB 5002 is received (step S201). In this case, in the management system 50, the storage / read processing unit 59 reads all corresponding terminal IDs by searching the session management DB 5002 using the session ID received by the transmission / reception unit 51 as a search key, and transmits / receives Send to part 51.

  Next, the determination unit 36 of the relay device 30 determines whether the number of terminal IDs (participation number) received in step S201 exceeds the upper limit of the number of images that can be arranged on one screen (step S202). . If the number of images exceeds the upper limit (YES), the determination unit 36 in the arrangement management DB 3001 “priority arrangement” column associated with the session ID transmitted in step S201 and the received terminal ID. In consideration of the above, it is determined whether there is a request for priority placement (step S204). If there is a request for priority placement (YES), the relay device 30 performs priority placement processing (step S205). Here, the priority placement process will be described in more detail with reference to FIG.

  As shown in FIG. 16, the storage / read processing unit 39 of the relay device 30 transmits, as a continuation of the above-described step S <b> 203, a transmission registered as “◯” having a priority placement request from the placement management DB 3001. The original terminal ID is read (step S301).

  Next, the arrangement unit 37 preferentially arranges the priority image in each display area of one screen (step S302). After the priority placement, the determination unit 36 determines whether there is still a display area available in one screen (step S303). If there is a vacancy (YES), the arrangement unit 37 randomly arranges the remaining images in the vacant display area up to the upper limit of the display area (step S304). Thus, the priority placement process ends. On the other hand, in the above-described step S303, when there is no space (NO), the process of step S304 is omitted, and the priority placement process ends.

  Subsequently, returning to FIG. 15, when the process of step S205 is completed, the process returns to step S201 again. In step S202 described above, if the upper limit of the number of images is not exceeded (NO), the placement unit 37 randomly places the images on one screen (step S206). Then, the process returns to step S201. Similarly, if there is no priority placement request in step S204 (NO), the process proceeds to step S206, and the process returns to step S201.

  Returning to FIG. 14, after the relay device 50 ends the image arrangement process, the transmission / reception unit 31 transmits content data including the image data arranged on one screen to the terminal 10aa (step). S103). Thereby, the transmission / reception unit 11 of the terminal 10aa receives the content data. Then, as shown in FIG. 18, the display control unit 17 of the terminal 10aa displays one screen after the priority image is preferentially arranged (step S104). In this case, if the content data includes sound data, the sound output unit 15b of the terminal 10aa also outputs sound. FIG. 18 shows one screen including the priority images arranged based on the priority image request. FIG. 18 is a diagram showing an example of a call screen after priority placement. FIG. 18 shows a state where the top two images are newly arranged in response to a priority arrangement request. In addition, since the process of step S102 is performed for each destination terminal, a different screen is displayed at each site in response to a request for priority placement.

<< Main effects of this embodiment >>
As described above, according to the present embodiment, the relay device 30 transmits specific image data including a priority image that is an image to be preferentially arranged on one screen to the terminal 10aa. Then, in the terminal 10aa, as shown in FIG. 18, it is possible to display an image of the other party who particularly wants to know the situation among the images of the other party. Thereby, at each base, there is an effect that it is possible to realize a state close to having a conversation at the same place.

[Supplement of Embodiment]
The relay device 30, the management system 50, the program providing system 90, and the maintenance system 100 in the above embodiment may be constructed by a single computer, or may be arbitrarily assigned by dividing each unit (function or means). It may be constructed by a plurality of computers. In addition, when the program providing system 90 is constructed by a single computer, the program transmitted by the program providing system 90 may be transmitted by dividing it into a plurality of modules. It may be transmitted. Further, when the program providing system 90 is constructed by a plurality of computers, a plurality of modules may be divided and transmitted from each computer.

  Also, a recording medium such as a CD-ROM in which the terminal program, the relay device program, or the transmission management program of each of the above embodiments is stored, the HD 204 in which these programs are stored, and the program including the HD 204 The providing system 90 is used when the terminal program, the relay device program, and the transmission management program are provided to a user or the like as a program product in Japan or abroad.

  In the above embodiment, a video conference terminal has been described as an example of a transmission terminal. However, the present invention is not limited to this, and a telephone system such as an IP (Internet Protocol) telephone or an Internet telephone may be used. Further, it may be a smartphone, a mobile phone, a car navigation terminal, a wearable computer, a surveillance camera, an electronic blackboard, a projector, a game machine, or an industrial device having a communication function. The wearable computer includes a wristwatch, a head mounted display, and the like. Industrial equipment includes office equipment such as an MFP (Multifunction Peripheral / Printer / Product), medical equipment such as an endoscope, and agricultural equipment such as a cultivator.

  In the above embodiment, image data and sound data have been described as an example of content data. However, the present invention is not limited to this, and touch data may be used. In this case, the feeling that the user touched on one terminal side is transmitted to the other terminal side. Further, the content data may be smell data. In this case, the odor (odor) on one terminal side is transmitted to the other terminal side. The content data may be at least one of image data, sound data, tactile data, and olfactory data.

  Further, in the above embodiment, the case where a video conference is performed by the transmission system 1 has been described. However, the present invention is not limited to this, and a general conversation such as a meeting, a family or a friend, or information in one direction It may be used for presentation.

DESCRIPTION OF SYMBOLS 1 Transmission system 10 Transmission terminal 11 Transmission / reception part 12 Reception part 17 Display control part 19 Memory | storage / reading process part 30 Relay apparatus 31 Transmission / reception part (an example of a transmission means, an example of a 1st reception means, an example of a 2nd reception means)
36 Judgment Unit (an example of judgment means)
37 Arrangement part (an example of arrangement means)
50 Transmission Management System 51 Transmission / Reception Unit 52 Authentication Unit 56 Selection Unit 57 Creation Unit 59 Storage / Read Processing Unit 90 Program Providing System 100 Maintenance System 1000 Storage Unit 3000 Storage Unit 3001 Arrangement Management DB
5000 storage unit 5001 authentication management DB
5002 Session management DB

JP 2014-82573 A

Claims (8)

A relay device that creates specific image data by arranging each image related to each image data transmitted from each of a plurality of transmission terminals on one screen, and transmits the specific image data to a specific transmission terminal Because
Source identification information for identifying the transmission source of the image data related to the priority image, which is the image preferentially arranged on the one screen among the images related to the image data, is received from the specific transmission terminal. First receiving means;
Second receiving means for receiving each of the image data transmitted from each of the plurality of transmission terminals;
When the number of transmission sources of the respective image data received by the second receiving unit exceeds the upper limit of the number of images that can be arranged on the one screen, it is received by the first receiving unit. Arrangement means for preferentially arranging the priority image related to the image data transmitted from the transmission source indicated by the transmission source identification information in a predetermined display area of the one screen;
Transmitting means for transmitting specific image data created by being arranged by the arranging means to the specific transmission terminal;
A relay apparatus comprising:   After the priority image is arranged on the one screen by the arrangement unit, if there is an empty display area on the one screen, the arrangement unit displays an image related to the image data received by the second reception unit. The relay apparatus according to claim 1, wherein a predetermined image that is not the priority image is randomly arranged in a vacant display area.   If the number of transmission sources of each image data received by the second receiving means does not exceed the upper limit of the number of images that can be arranged on the one screen, the arranging means 3. The relay apparatus according to claim 1, wherein an image related to the image data received by the two receiving units is randomly arranged in a predetermined display area of the one screen. Determining means for determining whether the number of transmission sources of each image data received by the second receiving means exceeds the upper limit of the number of images that can be arranged on the one screen;
When the determining unit determines that the number of the transmission sources exceeds the upper limit, the arranging unit transmits from the transmission source indicated by the transmission source identification information received by the first reception unit. 4. The relay device according to claim 1, wherein the priority image related to the processed image data is preferentially arranged in a predetermined display area of the one screen. 5. A relay device according to any one of claims 1 to 4,
The specific transmission terminal;
A transmission system comprising:   6. The transmission system according to claim 5, wherein the specific transmission terminal receives the specific image data transmitted by the transmission unit and displays the one screen including the priority image. A relay device that creates specific image data by arranging each image related to each image data transmitted from each of a plurality of transmission terminals on one screen, and transmits the specific image data to a specific transmission terminal There is an image processing method executed by
The relay device is
First receiving source identification information for identifying a source of image data related to a priority image arranged on the one screen preferentially among the images related to the image data from the specific transmission terminal Receiving step;
A second receiving step of receiving each of the image data transmitted from each of the plurality of transmission terminals;
When the number of transmission sources of each image data received in the second reception step exceeds the upper limit of the number of images that can be arranged on the one screen, the reception is performed in the first reception step. An arrangement step for preferentially arranging the priority image related to the image data transmitted from the transmission source indicated by the transmission source identification information in a predetermined display area of the one screen;
A transmission step of transmitting specific image data created by being arranged by the arrangement step to the specific transmission terminal;
The image processing method characterized by performing.   A program causing a computer to execute the method according to claim 7.

Images