JP2015177474A - Terminal, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a terminal perform suitable setting processing by itself by automatically discriminating a class of external equipment connected to an input part of the terminal.SOLUTION: A terminal 10 for transmitting image data to an opposite-side terminal over a communication network includes: an HDMI input part 20 to which image data are inputted from external equipment; a determination part 21 for determining a class of the external equipment connected to the HDMI input part 20 on the basis of data inputted from the HDMI part 20; and a CODEC/media control part 22 for performing image processing on the image data on the basis of the class determined by the determination part 21.

Description

  The present invention relates to a terminal, a method, and a program.

  In recent years, for example, for video conferencing, a camera or PC (Personal Computer) screen is captured on a terminal from an interface called HDMI (High-Definition Multimedia Interface), and the video or image is captured. It can be shared with remote terminals.

  In addition, the screens of cameras and PCs have been improved in resolution and frame rate (frame frequency), and an apparatus supporting HD (1920 × 1080) or more and 60 fps (frame per second) or more has been increasing. For this reason, for example, when a camera or PC image shared in a video conference system is transferred over a network, a compression mechanism for suppressing the use band has become complicated. Furthermore, since the purpose of use of the image is different between the camera and the PC, the required image quality may be different.

  For example, in sharing a camera image in a conference scene, it is preferable to give priority to the frame rate over resolution, while in sharing a PC image, it is preferable to give priority to resolution. Furthermore, a natural image is often input from the camera and a non-natural image is input from the PC. In this case, it is generally known that it is more efficient to use different algorithms for the natural image and the non-natural image. Also, some video conference terminals mirror left and right when drawing their own camera video on the own terminal. In such a case, it is better to perform a mirroring process on the image from the camera. Considering such circumstances, it is important to determine whether the input is a camera input or a PC input.

  However, in the conventional HDMI interface, the connection source (sink, eg, video conference terminal) cannot distinguish the device type (eg, camera, PC, tablet, etc.) of the connection destination (source, eg, camera, PC). For this reason, for example, when used in a video conference, the terminal cannot grasp whether the camera is connected to the HDMI interface or the PC, and the image data input from the external device is transferred to the external device. There is a problem that appropriate image processing cannot be controlled accordingly.

  The present invention has been made in view of the above, and an object of the present invention is to automatically determine the type of an external device connected to an input unit of a terminal and cause the terminal itself to perform appropriate image processing. To do.

  In order to solve the above-described problems and achieve the object, the present invention provides a terminal that transmits image data to a partner terminal via a communication network, and the input unit receives the image data from an external device. A determination unit that determines the type of the external device connected to the input unit based on data input from the input unit, and the image data based on the type determined by the determination unit And a control unit that controls image processing.

  According to the present invention, it is possible to automatically determine the type of an external device connected to the input unit of the terminal and allow the terminal itself to perform appropriate image processing.

FIG. 1 is a schematic diagram of a transmission system according to the first embodiment. FIG. 2 is a conceptual diagram showing a state of transmission / reception of image data, audio data, and various management information in the transmission system. FIG. 3 is a conceptual diagram illustrating the image quality of image data. FIG. 4 is an external view of the terminal according to the first embodiment. FIG. 5 is a hardware configuration diagram of the terminal according to the first embodiment. FIG. 6 is a hardware configuration diagram of the management system according to the first embodiment. FIG. 7 is a functional block diagram of each terminal, device, and system constituting the transmission system of the first embodiment. FIG. 8 is a functional configuration diagram of the final narrowing unit. FIG. 9 is a functional configuration diagram of the primary narrowing-down unit. FIG. 10 is a conceptual diagram showing the changed quality management table. FIG. 11 is a conceptual diagram showing a relay device management table. FIG. 12 is a conceptual diagram showing a terminal authentication management table. FIG. 13 is a conceptual diagram showing a terminal management table. FIG. 14 is a conceptual diagram showing a destination list management table. FIG. 15 is a conceptual diagram showing a session management table. FIG. 16 is a conceptual diagram showing an address priority management table. FIG. 17 is a conceptual diagram showing a transmission rate priority management table. FIG. 18 is a conceptual diagram showing a quality management table. FIG. 19 is a sequence diagram illustrating processing for managing state information indicating the state of each relay device transmitted from each relay device to the management system. FIG. 20 is a sequence diagram illustrating processing in a preparation stage for starting a call between a plurality of terminals. FIG. 21 is a sequence diagram illustrating processing for narrowing down relay devices. FIG. 22 is a processing flowchart showing processing for narrowing down relay devices. FIG. 23 is a diagram illustrating a point calculation state when the relay device narrowing-down process is performed. FIG. 24 is a sequence diagram illustrating processing in which a terminal selects a relay device. FIG. 25 is a process flow diagram illustrating a process of selecting a relay device at a terminal. FIG. 26 is a sequence diagram illustrating processing for transmitting and receiving image data and audio data between terminals. FIG. 27 is a sequence diagram illustrating processing in which a terminal selects a relay device. FIG. 28 is a processing flowchart for determining the device type to which the terminal is connected. FIG. 29 is a functional block diagram of each terminal, device, and system configuring the transmission system of the second embodiment. FIG. 30 is a sequence diagram of CEC communication performed between the terminal and the PC and CEC communication performed between the terminal and the camera. FIG. 31 is a processing flowchart for determining a device type connected to a terminal. FIG. 32 is a processing flowchart of software operating on the PC.

[First Embodiment]
Hereinafter, a first embodiment of a terminal, a method, and a program according to the present invention will be described with reference to FIGS.

<Overall configuration of transmission system to which terminal according to first embodiment is applied>
FIG. 1 is a schematic diagram of a transmission system 1 according to the first embodiment. FIG. 2 is a conceptual diagram showing a state of transmission / reception of image data, audio data, and various management information in the transmission system. FIG. 3 is a conceptual diagram illustrating the image quality of image data.

  The transmission system includes a data providing system for transmitting content data in one direction from one transmission terminal to the other transmission terminal via the transmission management system, information and emotions between a plurality of transmission terminals via the transmission management system, etc. A communication system that communicates with each other is included. This communication system is a system for mutually transmitting information, emotions, etc. between a plurality of communication terminals (corresponding to “transmission terminal”) via a communication management system (corresponding to “transmission management system”). An example is a conference system or a videophone system.

  In the present embodiment, assuming a video conference system as an example of a communication system, a video conference management system as an example of a communication management system, and a video conference terminal as an example of a communication terminal, a transmission system, a management system, and The terminal will be described. The terminal and management system of this embodiment are not only applied to a video conference system, but also applied to a communication system or a transmission system. Examples of transmission terminals include conference terminals, game machines, tablet terminals, mobile phones, smartphones, general-purpose PC terminals, car navigation terminals installed in automobiles, projection devices such as projectors, electronic blackboards, wearable terminals, and the like. It is done.

  1 includes a plurality of terminals (10aa, 10ab,..., 10ba, 10bb,..., 10ca, 10cb,..., 10da, 10db,...) And terminals (10aa, 10ab,. 10ba, 10bb, ..., 10ca, 10cb, ..., 10da, 10db, ...) (120aa, 120ab, ..., 120ba, 120bb, ..., 120ca, 120cb, ..., 120da, 120db, ...), multiple relays The apparatus (30a, 30b, 30c, 30d), the management system 50, the program providing system 90, and the maintenance system 100 are constructed.

  Hereinafter, an arbitrary terminal among a plurality of terminals (10aa, 10ab,..., 10ba, 10bb,..., 10ca, 10cb,..., 10da, 10db,...) Is referred to as “terminal 10”, and a plurality of displays ( 120aa, 120ab,..., 120ba, 120bb,..., 120ca, 120cb,..., 120da, 120db, ...) are indicated as “display 120”, and a plurality of relay devices (30a, 30b, 30c, 30d) are displayed. ) May be described as “relay device 30”. Further, a terminal as a request source that requests the start of a video conference may be referred to as a “request source terminal”, and a terminal as a request destination (relay destination) may be referred to as a “destination terminal”.

  The relay device 30 relays content data among a plurality of terminals 10. The management system 50 centrally manages the login authentication from the terminal 10, the management of the call status of the terminal 10, the management of the destination list, and the communication status of the relay device 30. Note that the image of the transmitted data may be a still image or a moving image, or a combination of a still image and a moving image. That is, the image described below is used as a concept including a video.

  A plurality of routers (70a, 70b, 70c, 70d, 70ab, 70cd) select an optimum route for image data and audio data. In the following, any router among the routers (70a, 70b, 70c, 70d, 70ab, 70cd) may be referred to as “router 70”.

  The program providing system 90 includes an HD (Hard Disk) 204 described below, and stores a terminal program for causing the terminal 10 to realize various functions (or to cause the terminal 10 to function as various means). The terminal program can be transmitted to The HD 204 of the program providing system 90 also stores a relay device program for causing the relay device 30 to realize various functions (or to cause the relay device 30 to function as various means). The relay device program can be transmitted. Further, the HD 204 of the program providing system 90 also stores a transmission management program for realizing various functions in the management system 50 (or causing the management system 50 to function as various means). A transmission management program can be transmitted.

  The maintenance system 100 is a computer for performing maintenance, management, or maintenance of at least one of the terminal 10, the relay device 30, the management system 50, and the program providing system 90. For example, when the maintenance system 100 is installed in the country and the terminal 10, the relay device 30, the management system 50, or the program providing system 90 is installed outside the country, the maintenance system 100 is remotely connected via the communication network 2. Maintenance such as maintenance, management, and maintenance of at least one of the terminal 10, the relay device 30, the management system 50, and the program providing system 90 is performed. In addition, the maintenance system 100 does not go through the communication network 2, but includes a model number, a manufacturing number, a sales destination, a maintenance check, at least one of the terminal 10, the relay device 30, the management system 50, and the program providing system 90. Or, maintenance such as failure history management is performed.

  By the way, the terminals (10aa, 10ab, 10ac,...), The relay device 30a, and the router 70a are communicably connected via the LAN 2a. The terminals (10ba, 10bb, 10bc,...), The relay device 30b, and the router 70b are communicably connected via a LAN 2b. The LAN 2a and the LAN 2b are communicably connected by a dedicated line 2ab including a router 70ab, and are constructed in a predetermined area A. For example, the region A is Japan, the LAN 2a is constructed in a Tokyo office, and the LAN 2b is constructed in an Osaka office.

  On the other hand, the terminals (10ca, 10cb, 10cc,...), The relay device 30c, and the router 70c are communicably connected via a LAN 2c. The terminals (10da, 10db, 10dc,...), The relay device 30d, and the router 70d are communicably connected via a LAN 2d. The LAN 2c and the LAN 2d are communicably connected via a dedicated line 2cd including the router 70cd, and are constructed in a predetermined area B. For example, region B is the United States of America, LAN 2c is built in a New York office, and LAN 2d is Washington D.C. C. Is built in the office. Area A and area B are connected to each other via routers (70ab, 70cd) via the Internet 2i. The terminals (10ca, 10cb, 10cc,...) May be directly connected to the Internet 2i without using the routers (70ab, 70cd) or the like.

  In addition, the management system 50 and the program providing system 90 are communicably connected to the terminal 10 and the relay device 30 via the Internet 2i. The management system 50 and the program providing system 90 may be installed in the region A or the region B, or may be installed in a region other than these.

  In the present embodiment, the communication network 2 of the present embodiment is constructed by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d. The communication network 2 may have a place where wireless communication such as WiFi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark) is performed in addition to wired communication.

  Further, in FIG. 1, four sets of numbers shown below each terminal 10, each relay device 30, management system 50, each router 70, and program providing system 90 are simplified IP addresses in general IPv4. Is shown. For example, the IP address of the terminal 10aa is “1.2.1.3”. In addition, IPv6 may be used instead of IPv4, but IPv4 will be used for the sake of simplicity.

  In addition, each terminal 10 is not only a call between a plurality of business establishments or a call between different rooms in the same business establishment, but a call within the same room or a call between the outdoors and indoors or outdoors and outdoors. May be used. When each terminal 10 is used outdoors, wireless communication such as a cellular phone communication network is performed.

  As illustrated in FIG. 2, in the transmission system 1, a management information session sei for transmitting and receiving various types of management information is established between the request source terminal and the destination terminal via the management system 50. The Further, between the request source terminal and the destination terminal, four pieces of data of high resolution image data, medium resolution image data, low resolution image data, and audio data are transmitted and received via the relay device 30. Four sessions are established. Here, these four sessions are collectively shown as an image / audio data session sed. In the example shown in FIG. 2, the image data sessions are divided into three, but this number is merely an example, and the number of sessions may be less than three or more than three. In the example shown in FIG. 2, the relay device 30 and the management system 50 are configured as separate servers, but both can be configured as the same server. Further, the request source terminal and the destination terminal may directly transmit and receive image / sound data without going through the relay device 30.

  Here, the resolution of the image data handled in the first embodiment will be described. As shown in FIG. 3 (a), the horizontal image is composed of 160 pixels and the vertical image is 120 pixels, and a low-resolution image serving as a base image and a horizontal image as illustrated in FIG. 3 (b). There are a medium resolution image consisting of 320 pixels and 240 pixels vertically, and a high resolution image consisting of 640 pixels horizontally and 480 pixels vertically as shown in FIG. Among these, when passing through a narrow band route, low-quality image data consisting only of low-resolution image data serving as a base image is relayed. When the band is relatively wide, low-resolution image data serving as a base image and medium-quality image data composed of medium-resolution image data are relayed. When the bandwidth is very wide, high-resolution image data composed of low-resolution image data serving as base image quality, intermediate-resolution image data, and high-resolution image data is relayed.

<Configuration of Terminal According to First Embodiment>
Next, the hardware configuration of the first embodiment will be described. FIG. 4 is an external view of the terminal according to the first embodiment. The appearance of the terminal shown in FIG. 4 is an example, and the present embodiment is not limited by the appearance of the terminal. As shown in FIG. 4, the terminal 10 includes a housing 1000, an arm 1010, and a camera housing 1020. An operation panel 1002 is formed on the upper right surface 1001 of the housing 1000. The operation panel 1002 includes a plurality of operation buttons 1003a to 1003e as operation input units, a power switch 1004, an alarm lamp 1005, a sound output surface 1006 for outputting sound from a built-in speaker, and the like.

  In addition, a housing portion 1008 as a recess for housing the arm 1010 and the camera housing 1020 is formed on the left upper surface 1007 of the housing 1000. A display 1050 is connected to the casing 1000 of the terminal 10 with a cable.

  The arm 1010 is attached to the housing 1000 via a torque hinge 1011, and the arm 1010 is configured to be able to rotate in the vertical direction with respect to the housing 1000 within a tilt angle θ1 of 135 degrees. FIG. 4 shows the terminal 10 in a state where the tilt angle θ1 is 90 degrees.

  The camera housing 1020 incorporates a camera, which will be described later, and can capture images of users, documents, rooms, and the like. The camera housing 1020 is formed with a torque hinge 1021. The camera housing 1020 is attached to the arm 1010 via a torque hinge 1021. The camera housing 1020 has a panning angle θ2 of ± 180 degrees and a tilt angle θ3 of ± 45 degrees, with the torque hinge 1021 being at 0 degrees with respect to the arm 1010 as shown in FIG. It is configured to be rotatable in the vertical and horizontal directions.

  The relay device 30, the management system 50, and the program providing system 90 have the same external appearance as a general server computer, and thus the description of the external appearance is omitted.

  FIG. 5 is a hardware configuration diagram of the terminal 10 according to the first embodiment. As shown in FIG. 5, the terminal 10 of the present embodiment has programs used for driving the CPU 101 such as a CPU (Central Processing Unit) 101 and IPL (Initial Program Loader) that control the operation of the entire terminal 10. ROM (Read Only Memory) 102, RAM (Random Access Memory) 103 used as a work area for the CPU 101, flash memory 104 for storing various data such as terminal programs, image data, and audio data, and control of the CPU 101 The SSD (Solid State Drive) 105 that controls the reading or writing of various data to the flash memory 104 according to the above, the media drive 107 that controls the reading or writing (storage) of data to the recording medium 106 such as the flash memory, and the destination of the terminal 10 Operation when selecting Operation button 108 is provided with a network I / F (Interface) 111 for the power switch 109 for switching ON / OFF of the power supply terminal 10, using the communication network 2 to the data transmission.

  The terminal 10 also includes a camera 112 that captures an image of a subject under the control of the CPU 101 to obtain image data, an image sensor I / F 113 that controls driving of the camera 112, a microphone 114 that inputs sound, a speaker 115 that outputs sound, An audio input / output I / F 116 that processes input / output of audio signals between the microphone 114 and the speaker 115 according to the control of the CPU 101, a display I / F 117 that transmits image data to the display 120 according to the control of the CPU 101, and various external devices. As shown in FIG. 5, an HDMI input I / F 118 for connection and a bus line 110 such as an address bus and a data bus for electrically connecting the above components are provided. In FIG. 5, it is assumed that the camera 112, the microphone 114, and the speaker 115 are built-in types, but they are not necessarily built-in types, and the display 120 is also an external type. Alternatively, a built-in configuration may be used.

  The display 120 is a display unit configured by a liquid crystal or an organic EL that displays an image of a subject, an operation icon, and the like. The display 120 is connected to the display I / F 117 by a cable 120c. The cable 120c may be an analog RGB (VGA) signal cable, a component video cable, or an HDMI (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal. It may be a cable.

  The camera 112 includes a lens and a solid-state image sensor that converts light into electric charges and digitizes a subject image (video). As the solid-state image sensor, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) is used. Etc. are used.

  External devices such as an external camera, an external microphone, and an external speaker can be connected to the HDMI input I / F 118 by a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven in preference to the built-in camera 112 under the control of the CPU 101. Similarly, when an external microphone is connected or when an external speaker is connected, each of the external microphones and the built-in speaker 115 is given priority over the internal microphone 114 and the internal speaker 115 according to the control of the CPU 101. An external speaker is driven.

  The recording medium 106 is detachable from the terminal 10. Further, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 101, not only the flash memory 104 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

  Further, the terminal program may be recorded in a computer-readable recording medium such as the recording medium 106 and distributed as a file in an installable or executable format. The terminal program may be stored in the ROM 102 instead of the flash memory 104.

  FIG. 6 is a hardware configuration diagram of the management system according to the first embodiment. The management system 50 stores various data such as a CPU 201 that controls the operation of the entire management system 50, a ROM 202 that stores programs used to drive the CPU 201 such as an IPL, a RAM 203 that is used as a work area for the CPU 201, and a transmission management program. The HD 204 to be stored, an HDD (Hard Disk Drive) 205 that controls reading or writing of various data to the HD 204 according to the control of the CPU 201, and a media drive 207 that controls reading or writing (storage) of data to a recording medium 206 such as a flash memory. , A display 208 that displays various information such as a cursor, menu, window, character, or image, a network I / F 209 for transmitting data using the communication network 2, characters, numerical values, A keyboard 211 having a plurality of keys for inputting various instructions, a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc., and a CD-ROM as an example of a removable recording medium (Compact Disc Read Only Memory) 213, a CD-ROM drive 214 for controlling the reading or writing of various data, and an address bus for electrically connecting the above components as shown in FIG. A bus line 210 such as a data bus is provided.

  The transmission management program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213. . The transmission management program may be stored in the ROM 202 instead of the HD 204.

  Further, since the relay device 30 has the same hardware configuration as the management system 50 shown in FIG. 6, the description thereof is omitted, and FIG. 6 is also referred to when the configuration of the relay device 30 is shown. . However, a relay device program for controlling the relay device 30 is recorded in the HD 204. Also in this case, the relay device program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good. The relay device program may be stored in the ROM 202 instead of the HD 204.

  Further, since the program providing system 90 and the maintenance system 100 have the same hardware configuration as the management system 50, the description thereof is omitted. However, a program providing program for controlling the program providing system 90 is recorded in the HD 204. Also in this case, the program providing program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Good. The program providing system program may be stored in the ROM 202 instead of the HD 204.

  As another example of the detachable recording medium, the recording medium is provided by being recorded on a computer-readable recording medium such as a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), or a Blu-ray Disc. May be.

<Functional Configuration of First Embodiment>
Next, the functional configuration of the first embodiment will be described. FIG. 7 is a functional block diagram of each terminal, device, and system constituting the transmission system 1 of the first embodiment. In FIG. 7, the terminal 10, the relay device 30, and the management system 50 are connected so that data communication can be performed via the communication network 2. The program providing system 90 shown in FIG. 1 is omitted in FIG. 7 because it is not directly related to the video conference communication.

<Functional Configuration of Terminal According to First Embodiment>
Next, each functional configuration of the terminal 10 will be described in detail. Hereinafter, in describing each functional configuration of the terminal 10, a relationship with main components for realizing each functional configuration of the terminal 10 among the respective components illustrated in FIG. 5 will also be described.

  The terminal 10 includes a transmission / reception unit 11, an operation input reception unit 12, a login request unit 13, an imaging unit 14, an audio input unit 15a, an audio output unit 15b, a final narrowing unit 16, a display control unit 17, a delay detection unit 18, and a storage A read processing unit 19, an HDMI input unit 20, a determination unit 21, and a CODEC / media control unit 22 are provided. Each of these units is a function realized by any one of the constituent elements shown in FIG. 5 being operated by a command from the CPU 101 according to the terminal program developed from the flash memory 104 onto the RAM 103, or It is a means to function. Further, the terminal 10 has a storage unit 2000 constructed by the RAM 103 shown in FIG. 5 and the flash memory 104 shown in FIG.

  The transmission / reception unit 11 of the terminal 10 shown in FIG. 7 is realized by a command from the CPU 101 shown in FIG. 5 and the network I / F 111 shown in FIG. Various data (or information) is transmitted / received to / from the terminal, device or system. The transmission / reception unit 11 starts receiving state information indicating the state of each terminal as a destination candidate from the management system 50 before starting a call with a desired destination terminal. This status information includes not only the operating status of each terminal 10 (online or offline status) but also whether it is possible to make a call even when online, whether it is in a call, whether it is away from the office, etc. Shows detailed status. In addition, the status information is not limited to the operating status of each terminal 10, but the cable 120 c is disconnected from the terminal 10 at the terminal 10, audio is output but no image is output, or audio is not output. Various states such as (MUTE) are shown. Below, the case where status information shows an operation state is demonstrated as an example.

  The operation input receiving unit 12 is realized by a command from the CPU 101 shown in FIG. 5 and the operation buttons 108 and the power switch 109 shown in FIG. 5 and receives various inputs by the user. For example, when the user turns on the power switch 109 shown in FIG. 5, the operation input receiving unit 12 shown in FIG. 7 receives the power ON and turns on the power.

  The login request unit 13 is realized by a command from the CPU 101 shown in FIG. 5, and triggers a login request from the transmission / reception unit 11 to the management system 50 via the communication network 2 when the power-on is accepted. And the current IP address of the request source terminal are automatically transmitted. When the user turns the power switch 109 from the ON state to the OFF state, the operation input receiving unit 12 completely turns off the power after the transmission / reception unit 11 transmits the state information indicating that the power supply is turned off to the management system 50. To do. As a result, the management system 50 side can grasp that the terminal 10 has been turned off from being turned on.

  The imaging unit 14 is realized by a command from the CPU 101 illustrated in FIG. 5 and the camera 112 and the image sensor I / F 113 illustrated in FIG. 5. Output data.

  The voice input unit 15a is realized by the instruction from the CPU 101 shown in FIG. 5 and the voice input / output I / F 116 shown in FIG. 5, and the user's voice is converted into a voice signal by the microphone 114. Thereafter, audio data relating to the audio signal is input. The audio output unit 15b is realized by a command from the CPU 101 shown in FIG. 5 and the audio input / output I / F 116 shown in FIG. 5, and outputs an audio signal related to the audio data to the speaker. To output sound.

  As shown in FIG. 8, the final narrowing unit 16 performs a final narrowing process for finally narrowing down from a plurality of relay devices 30 to one relay device 30, according to a command from the CPU 101 shown in FIG. A measurement unit 16a, a calculation unit 16b, and a final selection unit 16c are realized.

  Among these, the measurement unit 16 a measures the reception date and time when the pre-transmission information is received by the transmission / reception unit 11 for each pre-transmission information described later received by the transmission / reception unit 11. For each pre-transmission information whose reception date and time is measured by the measurement unit 16a, the calculation unit 16b determines the pre-transmission information based on the difference between the measured reception time and the transmission date and time included in the pre-transmission information. The required time (T) from transmission to reception is calculated. The final selection unit 16c finally selects one relay device by selecting the relay device 30 to which the pre-transmission information that required the shortest required time is relayed among the required times calculated by the calculation unit 16b. .

  The display control unit 17 is realized by the instruction from the CPU 101 shown in FIG. 5 and the display I / F 117 shown in FIG. 5. As will be described later, the received image data having different resolutions are combined. Control for transmitting the combined image data to the display 120 is performed. In addition, the display control unit 17 can transmit the destination list information received from the management system 50 to the display 120 and cause the display 120 to display the destination list.

  The delay detection unit 18 is realized by a command from the CPU 101 shown in FIG. 5 and detects a delay time (ms) of image data or audio data sent from another terminal 10 via the relay device 30. .

  The storage / reading processing unit 19 is executed by the instruction from the CPU 101 shown in FIG. 5 and the instruction from the SSD 105 shown in FIG. 5 or realized by the instruction from the CPU 101, and stores various data in the storage unit 1000. A process of reading various data stored in the unit 2000 is performed. The storage unit 2000 stores a terminal ID (Identification) for identifying the terminal 10, a password, and the like. Further, the storage unit 2000 overwrites and stores image data and audio data received when a call with the destination terminal is made. Among these, an image is displayed on the display 120 by the image data before being overwritten, and sound is output from the speaker 115 by the audio data before being overwritten.

  Note that the terminal ID of the first embodiment and the relay device ID described later indicate identification information such as a language, characters, symbols, or various signs used to identify the terminal 10 and the relay device 30, respectively. Further, the terminal ID and the relay device ID may be identification information in which at least two of the language, characters, symbols, and various indicia are combined.

  The HDMI input unit 20 is realized by the command from the CPU 101 shown in FIG. 5 and the HDMI input I / F 118 shown in FIG. 5, and the HDMI device is connected to the HDMI input I / F 118. A process of detecting that image data is received from the HDMI input I / F 118 and notifying the determination unit 21 of the information is performed. In response to a request from the determination unit 21, the HDMI input unit 20 extracts one frame of image data from the image data input to the HDMI input I / F 118 and provides the image data to the determination unit 21. Processing is also performed.

  The determination unit 21 is realized by a command from the CPU 101 illustrated in FIG. 5, and performs a process of requesting image data from the HDMI input unit 20 and receiving the image data. The determination unit 21 determines whether the received image data indicates a natural image or a non-natural image, and performs a process of transmitting the determination result to the CODEC / media control unit 22. Here, the natural image indicates an image (person, landscape, etc.) captured by a camera, for example, and the non-natural image indicates an image of a PC screen on which a document is displayed, for example.

  The determination of whether the image is a natural image or a non-natural image executed by the determination unit 21 includes, for example, a method of extracting an edge component of the image, a method of counting the number of used colors, and how the used colors change in the vicinity. It is possible to use a known method such as a method to use or a method for determining each divided area.

  The CODEC / media control unit 22 is realized by a command from the CPU 101 shown in FIG. 5, and based on the determination result made by the determination unit 21, an encoding process for image data input from the HDMI input I / F 118, etc. Control. That is, when the determination unit 21 determines that the image is a natural image, the device connected to the HDMI input I / F 118 is assumed to be a camera, and the CODEC / media control unit 22 is for a device that inputs a natural image ( Processing optimized for camera) is performed on the image data. On the other hand, when the determination unit 21 determines that the image is a non-natural image, the device connected to the HDMI input I / F 118 is assumed to be a PC, and is optimized for a device that inputs a non-natural image (for PC). The processed process is performed on the image data.

  Here, the device for inputting a natural image (for a camera) is not necessarily limited to a camera, and is a concept including general devices for inputting a natural image. However, in the following, it is simply abbreviated as “for a camera” as a typical device for inputting a natural image such as a person or a landscape. Similarly, the device for inputting a non-natural image (for PC) is not necessarily limited to a PC, but is simply abbreviated as “for PC” as a typical device for inputting a non-natural image. Therefore, even if it is simply described for a camera or PC, an external device that often inputs a natural image or an external device that often inputs a non-natural image will be described in this embodiment in the same manner as a camera or a PC. Can be used.

  For example, the following processes can be cited as processes optimized for cameras or PCs. In the case of a camera, the resolution of the received image data is resized and the frame rate is further reduced before the image data is encoded. Specifically, for example, it is conceivable that the resolution is resized from 1280 × 720 to 640 × 360, and the frame rate is thinned out from 60 fps / 30 fps to 20 fps. In the case of a PC, the frame rate is reduced without changing the resolution of the received image data before the image data is encoded. Specifically, for example, the frame rate may be thinned out from 60 fps / 30 fps to 0.5 fps.

  Furthermore, among the algorithms of encoding technology, by using an algorithm suitable for natural images and an algorithm suitable for non-natural images, an image can be encoded with higher compression efficiency depending on the connected external device. it can. For example, in H.265 (HEVC), which is currently being standardized, a specification has been added so that an image input from a PC can be more efficiently compressed. If it is determined, H.D. A process of selecting or switching 265 (HEVC) as the algorithm of the encoding technique may be performed.

  The CODEC / media control unit 22 preferably further controls the mirroring process. In the video conference system, when the camera image on the own terminal side is displayed on the display on the own terminal side, it may be displayed symmetrically in order to make the user feel as if he is looking in the mirror. Therefore, when it is determined that the device connected to the HDMI input I / F 118 is a camera, the CODEC / media control unit 22 may perform a mirroring process on the image data.

  Further, the CODEC / media control unit 22 performs different media control depending on the device connected to the HDMI input I / F 118 based on the determination result made by the determination unit 21. The media control mentioned here refers to the following control in a video conference system, for example.

  The network band that can be used during video conference communication varies due to data loss, communication retransmission, transmission delay, and the like. Therefore, it is necessary to adaptively control the quality of each of the voice, image, and screen sharing media according to the fluctuation of the network bandwidth. For example, when the available bandwidth is 1 Mbps (Megabits per second), audio media is 80 kbps (32 KHz sampling), image (video) media is 500 kbps (1280 X 720 @ 30 fps), and screen sharing media is 420 kbps (1280 X800 @ 5 fps). Suppose you were. When this reaches 500 kbps, the audio media is changed to 40 kbps (16 kHz sampling), the image (video) media is changed to 200 kbps (640 × 360 @ 15 fps), and the screen sharing media is changed to 260 kbps (1280 × 800 @ 1 fps).

  The CODEC / media control unit 22 performs media control in accordance with the change in the transmission environment as described above based on the determination result made by the determination unit 21. Specifically, when the determination result made by the determination unit 21 is a camera, control is performed to reduce the resolution by reducing the frame rate from the frame rate of 20 fps. When the determination result made by the determination unit 21 is PC, control is performed to reduce the frame rate at the minimum frame rate of 0.5 fps. If the resolution needs to be lowered, the screen sharing medium may be stopped.

<Functional configuration of relay device>
Next, the functional configuration of the relay device 30 according to the first embodiment will be described. As shown in FIG. 7, the relay device 30 includes a transmission / reception unit 31, a state detection unit 32, a data quality confirmation unit 33, a change quality management unit 34, a data quality change unit 35, and a storage / read processing unit 39. ing. Each of these units functions or functions realized by any one of the constituent elements shown in FIG. 6 operating according to a command from the CPU 201 according to the relay device program expanded from the HD 204 onto the RAM 203. Means. The relay device 30 includes a storage unit 3000 configured by the RAM 203 illustrated in FIG. 6 and / or the HD 204 illustrated in FIG.

(Change quality control table)
In the storage unit 3000, a change quality management DB (Data Base) 3001 configured by a change quality management table as shown in FIG. In the change quality management table, the IP address of the terminal 10 as the relay destination (destination) of the image data and the image quality of the image data relayed by the relay device 30 are associated with the relay destination and managed.

(Each functional configuration of the relay device)
Next, each functional configuration of the relay device 30 will be described in detail. In the following, in describing each functional configuration of the relay device 30, among the components illustrated in FIG. 6, the relationship with the main components for realizing each functional configuration of the relay device 30 is also described. explain.

  The transmission / reception unit 31 of the relay device 30 shown in FIG. 7 is realized by the command from the CPU 201 shown in FIG. 6 and the network I / F 209 shown in FIG. Various data (or information) is transmitted / received to / from other terminals, devices or systems.

  The state detection unit 32 is realized by a command from the CPU 201 illustrated in FIG. 6, and detects an operating state of the relay device 30 having the state detection unit 32. The operating state includes “online”, “offline”, “busy”, or “temporarily interrupted”.

  The data quality confirmation unit 33 is realized by a command from the CPU 201 shown in FIG. 6, searches the change quality management DB 3001 (see FIG. 10) using the IP address of the destination terminal as a search key, and relays correspondingly. The image quality of the relayed image data is confirmed by extracting the image quality of the image data.

  The change quality management unit 34 is realized by a command from the CPU 201 shown in FIG. 6 and changes the contents of the change quality management DB 3001 based on quality information to be described later sent from the management system 50. For example, a video conference is performed by transmitting and receiving high-quality image data between a request source terminal (terminal 10aa) whose terminal ID is “01aa” and a destination terminal (terminal 10db) whose terminal ID is “01db”. While the request is being made, the requesting terminal (terminal 10bb) and the destination terminal (terminal 10ca) that conduct another videoconference start the videoconference via the communication network 2, etc., so that the destination terminal (terminal 10db) When a delay in receiving the image data occurs, the relay device 30 reduces the image quality of the image data that has been relayed from the high image quality to the medium image quality. In such a case, the content of the change quality management DB 3001 is changed so as to lower the image quality of the image data relayed by the relay device 30 from the high image quality to the medium image quality based on the quality information indicating the medium image quality.

  The data quality changing unit 35 is realized by a command from the CPU 201 shown in FIG. 6, and the image quality of the image data sent from the transmission source terminal is changed based on the contents of the changed change quality management DB 3001. change.

  The storage / reading processing unit 39 is realized by an instruction from the CPU 201 illustrated in FIG. 6 and the HDD 205 illustrated in FIG. 6. The storage / reading processing unit 39 stores various data in the storage unit 3000 or is stored in the storage unit 3000. A process of reading various data is performed.

<Functional configuration of management system>
The management system 50 includes a transmission / reception unit 51, a terminal authentication unit 52, a state management unit 53, a terminal extraction unit 54, a terminal state acquisition unit 55, a primary narrowing unit 56, a session management unit 57, a quality determination unit 58, and a storage / read process. Unit 59 and delay time management unit 60. Each of these units functions or functions realized by any one of the constituent elements shown in FIG. 6 operating according to a command from the CPU 201 according to the management system program expanded from the HD 204 onto the RAM 203. Means. Further, the management system 50 has a storage unit 5000 constructed by the HD 204 shown in FIG.

(Relay device management table)
In the storage unit 5000, a relay device management DB 5001 configured by a relay device management table as shown in FIG. 11 is constructed. In this relay device management table, for each relay device ID of each relay device 30, the operating status of each relay device 30, the reception date and time when status information indicating the operating status is received by the management system 50, and the IP address of the relay device 30 , And the maximum data transmission rate (Mbps) in the relay device 30 is managed in association with each other. For example, in the relay device management table shown in FIG. 11, the relay device 30a with the relay device ID “111a” has an “online” operating state and the date and time when the status information was received by the management system 50 is “2009”. "November 10, 13:00" indicates that the IP address of this relay device 30a is "1.2.1.2" and that the maximum data transmission rate in this relay device 30a is 100 Mbps. Yes.

(Terminal authentication management table)
Furthermore, in the storage unit 5000, a terminal authentication management DB 5002 configured by a terminal authentication management table as shown in FIG. In this terminal authentication management table, each password is associated with each terminal ID of all terminals 10 managed by the management system 50 and managed. For example, the terminal authentication management table shown in FIG. 12 indicates that the terminal ID of the terminal 10aa is “01aa” and the password is “aaaa”.

(Terminal management table)
Further, in the storage unit 5000, a terminal management DB 5003 configured by a terminal management table as shown in FIG. 13 is constructed. In this terminal management table, for each terminal ID of each terminal 10, a destination name when each terminal 10 is a destination, an operating state of each terminal 10, a reception date and time when login request information described later is received by the management system 50, And the IP address of the terminal 10 is managed in association with each other. For example, in the terminal management table shown in FIG. 13, the terminal 10aa with the terminal ID “01aa” has the terminal name “Japan Tokyo Office AA terminal” and the operating state “online (transmittable)” It is indicated that the date and time when the login request information is received by the management system 50 is “13:40 on November 10, 2009” and the IP address of this terminal 10aa is “1.2.1.3”. ing.

(Destination list management table)
Furthermore, in the storage unit 5000, a destination list management DB 5004 configured by a destination list management table as shown in FIG. 14 is constructed. In this destination list management table, all terminal IDs of destination terminals registered as destination terminal candidates are managed in association with terminal IDs of request source terminals that request the start of a call in a video conference. For example, in the destination list management table shown in FIG. 14, the requesting terminal (terminal 10aa) whose terminal ID is “01aa” is the destination terminal (terminal 10db) that can request the start of a video conference call. The candidates are indicated to be a terminal 10ab having a terminal ID “01ab”, a terminal 10ba having a terminal ID “01ba”, a terminal 10bb having a terminal ID “01bb”, and the like. This destination terminal candidate is updated by being added or deleted in response to a request for addition or deletion from an arbitrary request source terminal to the management system 50.

(Session management table)
In the storage unit 5000, a session management DB 5005 configured by a session management table as shown in FIG. 15 is constructed. In this session management table, for each selection session ID used to execute a session for selecting the relay device 30, the relay device ID of the relay device 30 used for relaying image data and audio data, the request source terminal Management system that receives a terminal ID, a terminal ID of the destination terminal, a reception delay time (ms) when image data is received at the destination terminal, and delay information indicating the delay time from the destination terminal The reception date and time received at 50 is managed in association with each other. For example, in the session management table shown in FIG. 15, the relay device 30a (relay device ID “111a”) selected in the session executed using the selection session ID “se1” has the terminal ID “01aa”. "And the destination terminal (terminal 10db) whose terminal ID is" 01db "are relayed between the request source terminal (terminal 10aa) and the destination terminal (terminal 10db)" 2009 " It is shown that the delay time of the image data at “14:00 on November 10” is 200 (ms). When a video conference is performed between the two terminals 10, the reception date and time of the delay information may be managed based on the delay information transmitted from the request source terminal instead of the destination terminal. However, when a video conference is performed between three or more terminals 10, the reception date and time of delay information is managed based on the delay information transmitted from the terminal 10 on the image data and audio data receiving side.

(Address priority management table)
Furthermore, in the storage unit 5000, a priority management DB 5006 configured by an address priority management table as shown in FIG. 16 is constructed. In this address priority management table, in any terminal 10 and any relay device 30, the greater the “same” of the four sets of dot addresses in the general IPv4 IP addresses, the higher the address priority. The difference in dot address and the address priority are managed in association with each other so that the degree point becomes higher. This “same” means that the dot address portions are the same, and “different” means that the dot address portions are different. For example, in the address priority management table shown in FIG. 16, when three values are the same IP address from the top to the bottom of the dot address, the address priority point is “5”. When the two values are the same IP address from the top to the bottom of the dot address, the address priority point is “3”. In this case, whether the value of the lowest dot address is the same or not is not related to the priority. In the case of IP addresses having the same highest value of dot addresses and different second values from the top, the address priority point is “1”. In this case, whether or not the values of the third and lowest dot addresses from the top are the same has no relation to the priority. In the case of IP addresses having different uppermost values of dot addresses, the address priority point is “0”. In this case, whether or not the values of the second, third, and lowest dot addresses from the top are the same has no relation to the priority.

(Transmission speed priority management table)
The priority management DB 5006 built in the storage unit 5000 also includes a transmission speed priority management table as shown in FIG. In this transmission rate priority management table, the maximum data transmission rate and the transmission rate priority are set such that the higher the maximum data transmission rate (Mbps) value in the relay device 30, the higher the transmission rate priority point. Associated and managed. For example, in the transmission rate priority management table shown in FIG. 17, when the maximum data transmission rate in the relay apparatus 30 is 1000 Mbps or more, the transmission rate priority point is “5”. When the maximum data transmission rate in the relay device 30 is 100 Mbps or more and less than 1000 Mbps, the transmission rate priority point is “3”. When the maximum data transmission rate in the relay device 30 is 10 Mbps or more and less than 100 Mbps, the transmission rate priority point is “1”. When the maximum data transmission rate in the relay device 30 is less than 10 Mbps, the transmission rate priority point is “0”.

(Quality control table)
Furthermore, in the storage unit 5000, a quality management DB 5007 configured by a quality management table as shown in FIG. In this quality management table, the longer the delay time (ms) of the image data at the requesting terminal or the destination terminal, the lower the image data delay time and the image data quality so that the image quality of the image data relayed by the relay device 30 is lowered. (Image quality) and are managed in association with each other.

(Functional configuration of the management system)
Next, each functional configuration of the management system 50 will be described in detail. In the following, in describing each functional configuration of the management system 50, among the components shown in FIG. 6, the relationship with the main components for realizing each functional configuration of the management system 50 is also described. explain.

  The transmission / reception unit 51 is executed by the command from the CPU 201 shown in FIG. 6 and the network I / F 209 shown in FIG. Or information).

  The terminal authentication unit 52 is realized by a command from the CPU 201 shown in FIG. 6 and uses the terminal ID and password included in the login request information received via the transmission / reception unit 51 as a search key, and the storage unit 5000. The terminal authentication management DB 5002 is searched, and terminal authentication is performed by determining whether or not the same terminal ID and password are managed in the terminal authentication management DB 5002.

  The state management unit 53 is realized by a command from the CPU 201 shown in FIG. 6, and stores the request source terminal in the terminal management DB 5003 (see FIG. 13) in order to manage the operating state of the request source terminal that has requested login. The terminal ID, the operating state of the request source terminal, the reception date and time when the login request information is received by the management system 50, and the IP address of the request source terminal are stored and managed in association with each other. The state management unit 53 also manages the terminal based on the state information indicating that the power is turned off, which is sent from the terminal 10 when the user turns the power switch 109 of the terminal 10 from ON to OFF. The operation state indicating online of the DB 5003 (see FIG. 13) is changed to offline.

  The terminal extraction unit 54 is realized by a command from the CPU 201 shown in FIG. 6, and searches the destination list management DB 5004 (see FIG. 14) using the terminal ID of the request source terminal that requested the login as a key. The terminal ID is extracted by reading the terminal ID of the candidate destination terminal that can make a call. In addition, the terminal extraction unit 54 searches the destination list management DB 5004 (see FIG. 14) using the terminal ID of the request source terminal that has requested the login as a key, and registers the terminal ID of the request source terminal as a destination terminal candidate. The terminal IDs of other request source terminals are also extracted.

  The terminal status acquisition unit 55 is realized by a command from the CPU 201 shown in FIG. 6, and uses the terminal ID of the destination terminal candidate extracted by the terminal extraction unit 54 as a search key, as a terminal management DB 5003 (see FIG. 13). ) And the operation state is read for each terminal ID extracted by the terminal extraction unit 54. Thereby, the terminal state acquisition unit 55 can acquire the operating state of the candidate destination terminal that can make a call with the request source terminal that has requested the login. Further, the terminal state acquisition unit 55 searches the terminal management DB 5003 using the terminal ID extracted by the terminal extraction unit 54 as a search key, and also acquires the operating state of the request source terminal that has requested login.

  The primary narrowing unit 56 is realized by an instruction from the CPU 201 shown in FIG. 6, and this final narrowing process is performed to support a final narrowing process that finally narrows down from a plurality of relay apparatuses 30 to one relay apparatus 30. Perform the previous primary narrowing process. For this purpose, as shown in FIG. 9, the primary narrowing-down unit 56 receives a command from the CPU 101 shown in FIG. 5, as shown in FIG. 9, a selection session ID generation unit 56 a, a terminal IP address extraction unit 56 b, a primary selection The unit 56c and the priority determination unit 56d are realized.

  Among these, the selection session ID generation unit 56 a generates a selection session ID used for executing a session for selecting the relay device 30. The terminal IP address extraction unit 56b obtains the terminal management DB 5003 (see FIG. 13) based on the terminal ID of the request source terminal and the terminal ID of the destination terminal included in the start request information sent from the request source terminal. By searching, the IP address of each corresponding terminal 10 is extracted. The primary selection unit 56c selects the relay device ID of the relay device 30 whose operation state is “online” from among the relay devices 30 managed by the relay device management DB 5001 (see FIG. 11). The device 30 is selected.

  Further, the primary selection unit 56c searches the relay device management DB 5001 (see FIG. 11) based on the IP address of the request source terminal and the IP address of the destination terminal extracted by the terminal IP address extraction unit 56b. Thus, for each dot address of the IP address of the selected relay device 30, it is investigated whether it is the same as or different from each dot address in each IP address of the request source terminal and the destination terminal. Further, for each relay device, the primary selection unit 56c includes the higher two relay devices having the higher points of the integrated points obtained by integrating the higher points with respect to the terminal 10 at the address priority points and the transmission speed priority points. By selecting 30, the relay device 30 is further selected. In the present embodiment, the top two relay devices 30 with high points are selected. However, the present invention is not limited to this. If one relay device 30 can be narrowed down as many as possible, the top points with high points are selected. Three or more relay devices 30 may be selected.

  The priority determination unit 56d refers to the priority management DB 5006 (see FIG. 16), and determines an address priority point for each relay device 30 investigated by the primary selection unit 56c. Further, the priority determination unit 56d searches the priority management DB 5006 (see FIG. 17) based on the maximum data transmission rate of each relay device 30 managed by the relay device management DB 5001 (see FIG. 11). The point of the transmission rate priority is determined for each relay device 30 narrowed down by the first narrowing-down process by the primary selection unit 56c.

  Subsequently, the session management unit 57 is realized by an instruction from the CPU 102 shown in FIG. 6, and the selection generated by the selection session ID generation unit 56a in the session management DB 5005 (see FIG. 15) of the storage unit 5000. The session ID, the terminal ID of the request source terminal, and the terminal ID of the destination terminal are stored and managed in association with each other. In addition, the session management unit 57, for the session management DB 5005 (see FIG. 15), for each selection session ID, the relay device of the relay device 30 that is finally selected by the final selection unit 16c of the terminal 10 Store and manage IDs.

  The quality determination unit 58 searches the quality management DB 5007 (see FIG. 18) using the delay time as a search key and extracts the image quality of the corresponding image data, thereby determining the image quality of the image data to be relayed to the relay device 30. To do.

  The storage / reading processing unit 59 is executed by the instruction from the CPU 201 shown in FIG. 6 and the HDD 205 shown in FIG. 6, and stores various data in the storage unit 5000 or stored in the storage unit 5000. To read various data.

  The delay time management unit 60 is realized by an instruction from the CPU 201 shown in FIG. 6, and searches the terminal management DB 5003 (see FIG. 13) using the IP address of the destination terminal as a search key, thereby corresponding terminals. The ID is extracted, and further, in the session management table of the session management DB 5005 (see FIG. 15), the delay time indicated by the delay information is stored in the field of the delay time in the record including the extracted terminal ID. And manage.

<Method and Program according to First Embodiment>
Next, a processing method in the transmission system 1 according to the first embodiment will be described with reference to FIGS. FIG. 19 is a sequence diagram illustrating processing for managing state information indicating the state of each relay device 30 transmitted from each relay device 30 to the management system 50. FIG. 20 is a sequence diagram illustrating processing in a preparation stage for starting a call between a plurality of terminals 10. FIG. 21 is a sequence diagram illustrating a process of narrowing down the relay devices 30. FIG. 22 is a process flow diagram illustrating a process of narrowing down the relay devices 30. FIG. 23 is a diagram illustrating a point calculation state when the relay device 30 performs the narrowing-down process. FIG. 24 is a sequence diagram illustrating processing in which the terminal 10 selects the relay device 30. FIG. 25 is a process flow diagram illustrating a process of selecting the relay device 30 at the terminal. FIG. 26 is a sequence diagram illustrating processing for transmitting and receiving image data and audio data between terminals. FIG. 27 is a sequence diagram illustrating processing in which the terminal 10 selects a relay device. FIG. 28 is a processing flowchart for determining the device type to which the terminal 10 is connected.

  First, a process for managing state information indicating the state of each relay device 30 transmitted from each relay device 30 to the management system 50 will be described with reference to FIG. First, in each relay device 30, the state detection unit 32 shown in FIG. 7 periodically detects the operating state of the relay device 30 that is its own device (steps S1-1 to S4). Then, the transmission / reception unit 31 of each relay device 30 periodically transmits each state information to the management system 50 via the communication network 2 so that the operation state of each relay device 30 is managed in real time on the management system 50 side. (Steps S2-1 to 4). Each state information includes the relay device ID for each relay device 30 and the operating state detected by the state detection unit 32 of the relay device 30 related to each relay device ID. In the present embodiment, the relay devices (30a, 30b, 30d) operate normally and are “online”, while the relay device 30c is operating, but the relay operation of the relay device 30c is performed. The case where some trouble occurs in the program to be executed and it is “offline” is shown.

  Next, in the management system 50, the transmission / reception unit 51 receives each status information sent from each relay device 30, and the relay device management DB 5001 in the storage unit 5000 via the storage / read processing unit 59 (see FIG. 11). ), Status information is stored and managed for each relay device ID (steps S3-1 to S4). As a result, in the relay device management table as shown in FIG. 11, the operating state of “online” or “offline” is stored and managed for each relay device ID. At this time, the reception date and time when the status information is received by the management system 50 is also stored and managed for each relay device ID. If the status information is not sent from the relay device 30, the field portion of the operation state and the field portion of the reception date and time in each record of the relay device management table shown in FIG. The operation status and the reception date and time at the previous reception are shown.

  Next, a transmission / reception process of each management information in a preparation stage before starting a call between the terminal 10aa and the terminal 10db will be described with reference to FIG. In FIG. 20, various types of management information are transmitted and received through the management information session sei shown in FIG.

  First, when the user turns on the power switch 109 shown in FIG. 5, the operation input accepting unit 12 shown in FIG. 7 accepts the power on and turns on the power (step S21). The login request unit 13 automatically transmits login request information indicating a login request from the transmission / reception unit 11 to the management system 50 via the communication network 2 in response to the reception of the power ON (step S22). The login request information includes a terminal ID and a password for identifying the terminal 10aa that is the own terminal as a request source. These terminal ID and password are data read from the storage unit 1000 via the storage / read processing unit 19 and sent to the transmission / reception unit 11. When the login request information is transmitted from the terminal 10aa to the management system 50, the management system 50 that is the receiving side can grasp the IP address of the terminal 10ab that is the transmitting side.

  Next, the terminal authentication unit 52 of the management system 50 uses the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and the terminal authentication management DB 5002 of the storage unit 5000 (see FIG. 12). ) And terminal authentication is performed by determining whether the same terminal ID and password are managed in the terminal authentication management DB 5002 (step S23). Since the same terminal ID and password are managed by the terminal authentication unit 52, if it is determined that the login request is from the terminal 10 having a valid usage right, the state management unit 53 The DB 5003 (see FIG. 13) stores the terminal ID of the terminal 10aa, the operating state, the reception date and time when the login request information is received, and the IP address of the terminal 10aa in association with each other (step S24). Accordingly, in the terminal management table shown in FIG. 13, the terminal ID “01aa”, the operation state “online”, the reception date and time “2009.11.10.13:40”, and the IP address “1” of the terminal 10aa are displayed. .2.1.3 "is managed in association with each other.

  Then, the transmission / reception unit 51 of the management system 50 sends the authentication result information indicating the authentication result obtained by the terminal authentication unit 52 via the communication network 2 to the request source terminal (terminal 10aa) that made the login request. (Step S25). In the present embodiment, the case where the terminal authentication unit 52 determines that the terminal has a valid usage right will be described below.

  The terminal extraction unit 54 of the management system 50 searches the destination list management DB 5004 (see FIG. 14) using the terminal ID “01aa” of the request source terminal (terminal 10aa) that requested the login as a search key, and requests the request source terminal (terminal 10aa). The terminal IDs of candidate destination terminals that can communicate with the destination terminal are read out (step S26). Here, the terminal IDs (“01ab”, “01ba”, “01db”) of the destination terminals (terminals 10ab, 10ba, 10db) corresponding to the terminal ID “01aa” of the request source terminal (terminal 10aa) are extracted. Will be.

  Next, the terminal state acquisition unit 55 uses the terminal IDs (“01ab”, “01ba”, “01db”) of the destination terminal candidates extracted by the terminal extraction unit 54 as search keys, and the terminal management DB 5003 (FIG. 13). Search), and reading the operating state (“offline”, “online”) for each terminal ID extracted by the terminal extracting unit 54, thereby acquiring each operating state of the terminal (10ab, 10ba, 10db). (Step S27).

  Next, the transmission / reception unit 51 uses the terminal IDs (“01ab”, “01ba”, “01db”) as search keys used in step S27 and the corresponding destination terminals (terminals 10ab, 10ba, 10db). Are transmitted to the requesting terminal (terminal 10aa) via the communication network 2 (step S28). As a result, the request source terminal (terminal 10aa) has the respective current operating states (“offline” of the terminals (10ab, 10ba, 10db) that are candidates for destination terminals that can communicate with the request source terminal (terminal 10aa). ”,“ Online ”).

  Further, the terminal extraction unit 54 of the management system 50 searches the destination list management DB 5004 (see FIG. 14) using the terminal ID “01aa” of the request source terminal (terminal 10aa) that requested the login as a search key, and the request source The terminal IDs of other requesting terminals that have registered the terminal ID “01aa” of the terminal (terminal 10aa) as candidates for the destination terminal are extracted (step S29). In the destination list management table shown in FIG. 14, the terminal IDs of the other request source terminals to be extracted are “01ab”, “01ba”, and “01db”.

  Next, the terminal state acquisition unit 55 of the management system 50 searches the terminal state management DB 5003 (see FIG. 13) using the terminal ID “01aa” of the request source terminal (terminal 10aa) that has made the login request as a search key. Then, the operating state of the request source terminal (terminal 10aa) that requested the login is acquired (step S30).

  Then, the transmission / reception unit 51 includes the terminal management DB 5003 (FIG. 13) among the terminals (10ab, 10ba, 10db) related to the terminal IDs (“01ab”, “01ba”, “01db”) extracted in step S29. The terminal (10ba, 10db) whose operation state is “online” in the reference) includes the terminal ID “01aa” and the operation state “online” of the request source terminal (terminal 10aa) acquired in step S30. The destination state information is transmitted (steps S31-1, 2). When the transmission / reception unit 51 transmits the destination status information to the terminals (10ba, 10db), it is managed by the terminal management table shown in FIG. 13 based on the terminal IDs (“01ba”, “01db”). Refers to the IP address of the terminal being used. Accordingly, the request source terminal (terminal 10aa) that requested the login is sent to each of the other destination terminals (terminals 10db and 10ba) that can communicate with the request source terminal (terminal 10aa) that requested the login. The terminal ID “01aa” and the operation state “online” can be transmitted.

  On the other hand, in the other terminals 10 as well, when the user turns on the power switch 109 shown in FIG. 5, the operation input receiving unit 12 shown in FIG. Since the same processing as the processing of steps S22 to S31-1, 2 is performed, the description thereof is omitted.

  Next, processing for narrowing down the relay devices 30 will be described with reference to FIG. In FIG. 21, various types of management information are transmitted and received through the management information session sei shown in FIG. In the present embodiment, the request source terminal (terminal 10aa) is a terminal whose operation state is online based on the state information of the terminal received in steps S31-1, 2 among the terminals 10 as destination candidates. A call can be made with at least one of (10ba, 10db). Therefore, hereinafter, a case where the user of the request source terminal (terminal 10aa) has selected to start a call with the destination terminal (terminal 10db) will be described.

  First, when the user presses the operation button 108 shown in FIG. 5 and selects the terminal 10db, the operation input reception unit 12 shown in FIG. 7 starts a call with the destination terminal (terminal 10db). The request to be received is accepted (step S41). Then, the transmission / reception unit 11 of the request source terminal (terminal 10aa) includes the terminal ID “01aa” of the terminal 10aa and the terminal ID “01db” of the destination terminal (terminal 10db), and indicates a start request indicating that a call is to be started. Information is transmitted to the management system 50 (step S42). Accordingly, the transmission / reception unit 51 of the management system 50 can receive the start request information and grasp the IP address “1.2.1.3” of the request source terminal (terminal 10aa) that is the transmission source. .

  Then, based on the terminal ID “01aa” of the request source terminal (terminal 10aa) and the terminal ID “01db” of the destination terminal (terminal 10db) included in the start request information, the state management unit 53 determines the terminal management DB 5003 (see FIG. 13). ) In the terminal management table, both of the field portions of the operation state of the records each including the terminal ID “01aa” and the terminal ID “01db” are changed to “busy” (step S43). In this state, the request source terminal (terminal 10aa) and the destination terminal (terminal 10db) have not started a call, but are in a call state and the other terminal 10 is a request source terminal (terminal 10aa) or a destination terminal. When a call is made to (terminal 10db), a notification sound or a display indicating a so-called busy state is output.

  Next, a process for executing a session for selecting the relay device 30 will be described. First, the selection session ID generation unit 56a shown in FIG. 9 generates a selection session ID used for executing a session for selecting the relay device 30 (step S44). Then, the session management unit 57 stores the selection session ID “se1” generated in step S44 and the terminal ID “01aa” of the request source terminal (terminal 10aa) in the session management DB 5005 (see FIG. 15) of the storage unit 5000. And the terminal ID “01db” of the destination terminal (terminal 10db) are stored and managed in association with each other (step S45).

  Next, the primary screening unit 56 of the management system 50 shown in FIG. 7 includes a request source terminal (terminal 10aa) and a destination terminal based on the relay device management DB 5001, the terminal management DB 5003, and the priority management DB 5006. The primary narrowing down of the relay device 30 for relaying a call with the (terminal 10db) is performed (step S46).

  Here, the process in step S46 will be described in more detail with reference to FIGS. First, the terminal IP address extraction unit 56b shown in FIG. 9 has the terminal ID “01aa” of the request source terminal (terminal 10aa) included in the start communication information sent from the request source terminal (terminal 10aa). By searching the terminal management DB 5003 (see FIG. 13) based on the terminal ID “01db” of the destination terminal (terminal 10db), the IP addresses (“1.2” of the corresponding terminals (10aa, 10db) are searched. 1.3 ”and“ 1.3.2.4 ”) are extracted (step S46-1).

  Next, the primary selection unit 56c selects each of the relay devices (30a, 30b, 30d) that are “online” among the operating states of the relay device 30 managed by the relay device management DB 5001 (see FIG. 11). The relay device ID (111a, 111b, 111d) is selected (step S46-2). Further, the primary selection unit 56c extracts the IP address “1.2.1.3” of the request source terminal (terminal 10aa) and the IP address “1” of the destination terminal (terminal 10db) extracted in step S46-1. .3.2.4 "is used to search the relay device management DB 5001 (see FIG. 11), and thereby each IP address (" "of the relay device (30a, 30b, 30d) selected in step S46-2 is selected. "1.2.1.2.", "1.2.2.2", "1.3.2.2") for each dot address of the request source terminal (terminal 10aa) and destination terminal (terminal 10db). ) Is checked whether it is the same as or different from each dot address in each IP address (“1.2.1.3”, “1.3.2.4”) (step S46-3).

  Next, the priority determination unit 56d refers to the priority management DB 5006 (see FIG. 16), and determines the point of the address priority for each relay device (30a, 30b, 30d) investigated in the above step 46-3. Is determined (step S46-4). When the result of this determination process is shown in a table, the state shown in FIG. 23 is obtained. FIG. 23 is a diagram illustrating a calculation state of priority points when the relay device 30 performs the narrowing-down process. In FIG. 23, an address priority point, a transmission speed priority point, and an integration point are shown for each relay device ID. The address priority points further indicate points for the request source terminal (terminal 10aa) and points for the destination terminal (terminal 10db) of each relay device 30. The integration point is the sum of the higher one of the two address priority points and the transmission speed priority point.

  In the present embodiment, the IP address “1.2.1.2” of the relay device 30a is different from the IP address “1.2.1.3” of the request source terminal (terminal 10aa). .. ”, the address priority point is“ 5 ”as shown in FIG. Further, as shown in FIG. 1, the IP address “1.2.1.2” of the relay device 30a is different from the IP address “1.3.2.4” of the destination terminal (terminal 10db). As shown in FIG. 16, since “same. Different. Different. Different”, the point of the address priority is “1” as shown in FIG. Further, the IP address “1.2.2.2” of the relay device 30b is “same. Same. Different. Different” with respect to the IP address “1.2.1.3” of the request source terminal (terminal 10aa). Therefore, the address priority point is “3”. Further, the IP address “1.2.2.2” of the relay device 30b is “same. Different. Same. Different” with respect to the IP address “1.3.2.4” of the destination terminal (terminal 10db). Therefore, the point of address priority is “1”. Further, the IP address “1.3.2.2” of the relay device 30d is “same. Different. Different. Different” with respect to the IP address “1.2.1.3” of the request source terminal (terminal 10aa). Therefore, the address priority point is “1”. Further, the IP address “1.3.2.2” of the relay device 30d is “same. Same. Same. Different” with respect to the IP address “1.3.2.4” of the destination terminal (terminal 10db). Therefore, the address priority point is “5”.

  Next, returning to FIG. 22, the priority determination unit 56d determines the priority management DB 5006 (FIG. 17) based on the maximum data transmission rate of each relay device 30 managed by the relay device management DB 5001 (see FIG. 11). By searching for (reference), the point of the transmission speed priority is determined for each relay device (30a, 30b, 30d) narrowed down by the first narrowing-down process in step S46-2 (step S46-5). In the present embodiment, as shown in FIG. 11, the maximum data transmission rate of the relay device 30a is 100 (Mbps). Therefore, referring to the transmission rate priority shown in FIG. The degree is 3 points. Similarly, if the maximum data transmission rate of the relay device 30b is calculated to be 1000 (Mbps), the transmission rate priority is 5 points. Similarly, if the maximum data transmission rate of the relay device 30d is calculated to be 10 (Mbps), the transmission rate priority is 1 point.

  Next, for each relay device (30a, 30b, 30d), the primary selection unit 56c adds the higher point of the terminals (10aa, 10db) and the transmission speed priority point at the address priority point. Among the integrated points, the higher two or more relay devices 30 with higher points are selected (step S46-6). In the present embodiment, as shown in FIG. 23, since the relay device IDs (111a, 111b, 111d) have integration points “8”, “8”, and “6”, respectively, the relay device ID “ The relay device 30a related to “111a” and the relay device 30b related to the relay device ID “111b” are selected.

  When the narrowing-down process in step S46 is completed, the transmission / reception unit 51 illustrated in FIG. 7 transmits the number of narrowed relay apparatuses 30 to the destination terminal (terminal 10db) via the communication network 2. Relay device narrowing information is transmitted (step S47). The relay device narrowing information includes the number “2” of the relay devices 30 narrowed down in the step 46, the terminal ID “01aa” of the request source terminal (terminal 10aa), and the selection session ID “se1”. It is. Thereby, the terminal 10db can grasp how many relay devices 30 are in the execution of the session with the selection session ID “se1” and which terminal 10 has requested to start a call. At the same time, the IP address “1.1.1.2” of the management system 50 that is the transmission source of the relay device narrowing information can be grasped.

  Then, the terminal 10db transmits reception completion information indicating that reception of the relay device narrowing information is completed from the transmission / reception unit 11 to the management system 50 via the communication network 2 (step S48). This reception completion information includes the session ID “se1”. As a result, the management system 50 completes the transmission of the number of relay apparatuses being executed with the session ID “se1”, and the IP address “1.3.2.4” of the destination terminal (terminal 10db) that is the transmission source. Can be grasped.

  Next, a process in which the destination terminal (terminal 10db) selects the relay device 30 will be described with reference to FIGS. In FIG. 24, various types of management information are transmitted and received by the management information session sei shown in FIG.

  First, before starting a video conference call, the management system 50 provides pre-relay request information for requesting relay in advance to each of the relay devices (30a, 30b) narrowed down in step S46. Transmit (steps S61-1, 2). The pre-relay request information includes the session ID “se1”, the IP address “1.2.1.3” of the request source terminal (terminal 10aa), and the IP address “1.3.2” of the destination terminal (terminal 10db). .4 "is included. As a result, each relay device (30a, 30b) can grasp which selection session it belongs to, what the request source terminal is, and what the destination terminal is, and the pre-relay request information The IP address “1.1.1.2” of the management system 50 that is the sender of

  Next, each of the relay devices (30a, 30b) sends the transmission / reception unit 31 via the communication network 2 to the request source terminal (terminal 10aa) ascertained in steps S61-1, 2 before starting the call. Pre-transmission request information indicating that pre-transmission information including ping (Packet Internet Groper), which will be described later, is transmitted is transmitted to each relay device (30a, 30b) as its own device (steps S62-1, 2). . This pre-transmission information includes the session ID “se1”. Accordingly, the request source terminal (terminal 10aa) grasps that the pre-transmission information is transmitted to each relay device (30a, 30b) in the selection process of the relay device 30 executed with the session ID “se1”. The IP addresses (“1.2.1.2” and “1.2.2.2”) of the relay apparatuses (30a, 30b) that are the transmission sources of the advance transmission request information can be grasped.

  The management system 50 directly notifies the IP address of the destination terminal to the relay device 30a as in step S61-1, without notifying the request source terminal of the IP address of the destination terminal. As in step S61-2, the relay device 30a requests the request source terminal to transmit the pre-transmission request information to the own device (relay device 30a). This is to ensure security by not informing the IP address of 10.

  Next, the request source terminal (terminal 10aa) transmits the pre-transmission information from the transmission / reception unit 11 to each relay device (30a, 30b) via the communication network 2 (steps S63-1, 2). Prior to transmission of image data and audio data, this pre-transmission information is transmitted to the destination terminal (terminal 10db) via each relay device (30a, 30b) instead of these image data and audio data. This information is used to measure the time required from transmission of the request source terminal (terminal 10aa) to reception of the destination terminal (terminal 10db). In addition, the advance transmission information includes a ping for confirming that the request source terminal (terminal 10aa), the relay devices (30a, 30b), and the destination terminal (terminal 10db) are connected to be communicable, and the request source The transmission date and time when the pre-transmission information is transmitted from the terminal (terminal 10aa) and the session ID “se1” are included. Thereby, each relay device (30a, 30b) can grasp that the pre-transmission information has been sent in the execution of the session with the selection session ID “se1”, and is a transmission source of the pre-transmission information. The IP address “1.2.1.3” of the request source terminal (terminal 10aa) can be grasped.

  Next, each relay device (30a, 30b) receives the IP address “1.3.2.4 of the destination terminal (terminal 10db) included in the pre-relay request information received in steps S61-1, 2 above. ", The above-mentioned pre-transmission information is relayed (steps S64-1, 2). Thereby, the destination terminal (terminal 10db) can grasp that the pre-transmission information has been sent in the execution of the session with the session ID “se1”, and is a transmission source (relay source) of the pre-transmission information. The IP address (“1.2.1.2”, “1.2.2.2”) of the relay device (30a, 30b) can be grasped.

  Next, the final narrowing unit 16 of the destination terminal (terminal 10db) finally narrows down to one relay device 30 that relays image data and audio data in a video conference call based on the pre-transmission information (step S65). ).

  Here, the process in step S65 will be described in more detail with reference to FIGS. First, the measurement unit 16a of the final narrowing unit 16 illustrated in FIG. 8 receives the reception when the transmission / reception unit 11 of the terminal 10db receives, for each pre-transmission information relayed by each relay device (30a, 30b). The date and time are measured (step S65-1). Next, the calculation unit 16b determines, for each pre-transmission information whose reception time is measured, from the transmission of each pre-transmission information based on the difference between the reception date and time and the transmission date and time included in the pre-transmission information. The time required until reception is calculated (step S65-2). Next, the final selection unit 16c determines whether or not all pieces of pre-transmission information corresponding to the number “2” of relay devices 30 scheduled to be relayed have been received in the execution of the session with the session ID “se1” (Step S65). -3). And when not having received all (No), the last selection part 16c judges whether predetermined time (here 1 minute) passed since receiving the prior transmission information in the terminal 10db (step S65-4). ). Further, if the predetermined time has not elapsed (No), the process returns to step S65-1. On the other hand, when all the data is received in Step 65-3 (Yes) or when a predetermined time has elapsed (Yes) in Step S65-4, the final selection unit 16c has been calculated by the calculation unit 16b so far. One relay device 30 that relays the pre-transmission information that requires the shortest required time is selected (step S65-5). In the present embodiment, the relay device 30a is selected on the assumption that the pre-transmission information relayed by the relay device 30a has a shorter time from transmission to reception than the pre-transmission information relayed by the relay device 30b. Is shown.

  Next, the destination terminal (terminal 10db) transmits selection information indicating that the relay device 30a has been selected from the transmission / reception unit 11 to the management system 50 via the communication network 2 (step S66). This selection information includes the session ID “se1” and the relay device ID “111a” of the selected relay device 30a. Accordingly, the management system 50 can grasp that the relay device 30a has been selected in the execution of the session with the session ID “se1”, and can also determine the IP address “1” of the destination terminal (terminal 10db) that is the transmission source of the selection information. .3.2.4 ”.

  Next, the session management unit 57 of the management system 50 finally adds 1 to the field part of the relay device ID of the record including the session ID “se1” in the session management table of the session management DB 5005 (see FIG. 15). The relay device ID “111a” of the selected relay device 30a is stored and managed (step S67). Then, the transmission / reception unit 51 of the management system 50 transmits relay start request information indicating a request to start relaying to the relay device 30a via the communication network 2 (step S68). In this relay start request information, each IP address (“1.2.1.3”, “1.3.2.4”) of the request source terminal (terminal 10aa) and destination terminal (terminal 10db) to be relayed is included. It is included. Thereby, since the relay device 30a can grasp that the relay device 30a as its own device has been selected, three image data of low resolution, medium resolution, and high resolution between the terminals (10aa, 10db), At the same time, a session for calling the voice data is established (step S69). Therefore, the terminals (10aa, 10db) can start a video conference call.

  In step S47, the management system 50 transmits relay device narrowing information to the destination terminal (terminal 10db), and relays on the destination terminal (terminal 10db) side through steps S48 to S64-1,2. Although the device selection process (step S65) has been performed, the present invention is not limited to this. In step S47, the management system 50 transmits the relay device narrowing information to the request source terminal (terminal 10aa). From S64-1 and S64-1, the transmission source and reception source of each information may be switched between the request source terminal (terminal 10aa) and the destination terminal (terminal 10db). Thereby, the request source terminal (terminal 10aa) can perform the relay device selection process instead of step S65, and can also transmit the selection information instead of step S66.

  Next, a process for transmitting and receiving image data and audio data to perform a video conference call between the request source terminal and the destination terminal will be described with reference to FIGS. 7 and 26. Note that image data and audio data are transmitted and received in one-way processing for transmitting image data and audio data from the terminal 10aa to the terminal 10db and in reverse processing for transmitting image data and audio data from the terminal 10db to the terminal 10aa. Since the delay time detection and the like described later are the same process, the one-way communication will be described, and the reverse communication will be omitted.

  First, the request source terminal (terminal 10aa) receives the image data of the subject imaged by the imaging unit 14 and the audio input by the audio input unit 15a by the image / audio data session sed shown in FIG. The voice data is transmitted from the transmission / reception unit 11 to the relay device 30a via the communication network 2 (step S81). In the present embodiment, the high-quality image data including the low resolution, the medium resolution, and the high resolution shown in FIG. 3 and the audio data are transmitted. Thereby, in the relay device 30a, the transmission / reception unit 31 receives the image data and the audio data of the three resolutions. Then, the data quality confirmation unit 33 searches the change quality management DB 3001 (see FIG. 10) using the IP address “1.3.2.4” of the destination terminal (terminal 10db) as a search key, and the corresponding relay image By extracting the image quality of the data, the image quality of the image data to be relayed is confirmed (step S82). In the present embodiment, since the image quality of the confirmed image data is “high image quality” and is the same as the image quality of the image data received by the transmission / reception unit 31, the relay device 30a uses the session sed for image / audio data. The image data with the same image quality and the sound data with the same sound quality are transferred to the destination terminal (terminal 10db) (step S83). As a result, the destination terminal (terminal 10db) receives the high-quality image data and the audio data composed of the low resolution, the medium resolution, and the high resolution at the transmission / reception unit 11. Then, the display control unit 17 can combine the three image quality image data to display an image on the display 120, and the sound output unit 15b can output sound based on the sound data.

  Next, the delay detection unit 18 of the terminal 10db detects the reception delay time of the image data received by the transmission / reception unit 11 at regular time intervals (for example, every second) (step S84). In the present embodiment, the following description is continued for a case where the delay time is 200 (ms).

  The transmission / reception unit 11 of the destination terminal (terminal 10db) sends delay information indicating the delay time “200 (ms)” to the management system 50 via the communication network 2 by the management information session sei shown in FIG. Transmit (step S85). As a result, the management system 50 can grasp the delay time and the IP address “1.3.2.4” of the terminal 10db that is the transmission source of the delay information.

  Next, the delay time management unit 60 of the management system 50 searches the terminal management DB 5003 (see FIG. 13) using the IP address “1.3.2.4” of the destination terminal (terminal 10db) as a search key. Then, the corresponding terminal ID “01db” is extracted, and in the session management table of the session management DB 5005 (see FIG. 15), the delay information field portion of the record of the terminal ID “01db” is indicated by the delay information. The stored delay time “200 (ms)” is stored and managed (step S86).

  Next, the quality determination unit 58 searches the quality management DB 5007 (see FIG. 18) using the delay time “200 (ms)” as a search key, and extracts the image quality “medium image quality” of the corresponding image data. The image quality is determined as “medium image quality” (step S87).

  Next, the transmitting / receiving unit 51 uses the relay device ID “111a” associated with the terminal ID “01db” in the session management table of the session management DB (see FIG. 15) as a search key, and the relay device management DB 5001 (see FIG. 15). 11) and the IP address “1.2.1.2” of the corresponding relay device 30a is extracted (step S88). Then, the transmission / reception unit 51 transmits the quality “medium image quality” of the image data determined in step 87 to the relay device 30a via the communication network 2 by the management information session sei shown in FIG. Information is transmitted (step S89). This quality information includes the IP address “1.3.2.4” of the destination terminal (terminal 10db) used as the search key in step S86. Thereby, in the relay device 30a, the change quality management unit 34 stores the IP address “1.3.2.4” of the destination terminal 10 (here, the terminal 10db) in the change quality management DB 3001 (see FIG. 10). , And the image quality “medium image quality” of the relayed image data is stored and managed in association with each other (step S90).

  Next, the terminal 10aa continues to the relay device 30a through the image / audio data session sed in the same manner as in the above step S81, and the high-quality image data including the low resolution, medium resolution, and high resolution, and Audio data is transmitted (step S91). As a result, in the relay device 30a, as in step S82, the data quality confirmation unit 33 uses the IP address “1.3.2.4” of the destination terminal (terminal 10db) as a search key to change the quality management DB 3001 ( 10) and the image quality “medium image quality” of the corresponding image data to be relayed is extracted to check the image quality of the image data to be relayed (step S92). In this embodiment, since the image quality of the confirmed image data is “medium image quality” and lower than the image quality “high image quality” of the image data received by the transmission / reception unit 31, the data quality changing unit 35 The image quality of the image data is changed by suppressing the image quality from “high image quality” to “medium image quality” (step S93).

  Then, the image / sound data session sed changes the image data in which the image quality of the image data is changed to “medium image quality” and the sound quality of the sound to the terminal 10 db via the communication network 2. Audio data that has not been transmitted is transmitted (step S94). As a result, the destination terminal (terminal 10db) receives the image data and audio data of medium quality composed of two of low resolution and medium resolution by the transmission / reception unit 11. Then, the display control unit 17 can combine the two resolution image data to display the image on the display 120, and the audio output unit 15b can output the audio based on the audio data.

  FIG. 27 is a sequence diagram illustrating processing in which a terminal selects a relay device according to an example different from FIG. That is, in steps S61-1, S62-1, S61-2, and S62-2 in FIG. 24, prior request information is sent from the management system 50 to the requesting terminal (terminal 10aa) via each relay device (30a, 30b). 27 differs from the example shown in FIG. 27 in that prior request information is transmitted directly from the management system 50 to the request source terminal (terminal 10aa).

  As shown in FIG. 27, first, the management system 50 transmits advance transmission request information to the request source terminal (terminal 10aa) (step S161-1). In this case, the pre-transmission request information includes the session ID “se1” and the IP addresses (“1.2.1.2” and “1.2.2. 2 ”) and the IP address“ 1.3.2.4 ”of the destination terminal (terminal 10db). Thereby, the request source terminal (terminal 10aa) belongs to which selection session, the relay device 30 that is the final narrowing candidate by the primary narrowing down, and the IP address of the destination terminal (terminal 10db) It is possible to grasp what the address is and to grasp the IP address “1.1.1.2” of the management system 50 that is the transmission source of the pre-transmission request information.

  Next, the request source terminal (terminal 10aa) transmits the advance transmission information to the relay devices (30a, 30b) (steps S162-1, 2). This pre-transmission information includes the ping, the transmission date and time when the pre-transmission information was transmitted from the request source terminal (terminal 10aa), and the session ID “se1”, as in steps S63-1 and S63-2 in the first embodiment. In addition, the IP address “1.3.2.4” of the destination terminal (terminal 10db) is further included. Thereby, each relay device (30a, 30b) can grasp that the pre-transmission information has been sent in the execution of the session with the selection session ID “se1”, and is a transmission source of the pre-transmission information. The IP address “1.2.1.3” of the request source terminal (terminal 10aa) can be grasped, and further, the IP address “1.3 of the destination terminal (terminal 10db) that is the transmission destination of this pre-transmission information. .2.4 ". Therefore, each relay apparatus (30a, 30b) can transmit (or transfer) the pre-transmission information to the acquired IP address of the destination terminal (terminal 10db).

  Steps S163-1 to S168 in FIG. 27 are the same processes as S64-1 to S69 in FIG.

  Here, a processing flowchart for determining the type of device connected to the terminal 10 will be described with reference to FIGS. The processing flow shown in FIG. 28 is executed after the image data and audio data communication session is established in FIG. 24 (step S69) or after the image data and audio data communication session is established in FIG. 27 (step S168). The

  As shown in FIG. 28, after establishing a communication session (steps S69 and S168), the HDMI input unit 20 of the terminal 10 determines whether or not an HDMI device is connected to the HDMI input I / F 118 (step S28-1). ). If it is determined in step S28-1 that the HDMI device is not connected (No), the process for determining the device type is terminated. On the other hand, if it is determined in step S28-1 that the HDMI device is connected (Yes), the process proceeds to the next step.

  In the next step, the HDMI input unit 20 of the terminal 10 determines whether data is received from the HDMI device connected to the HDMI input I / F 118 (step S28-2). If it is determined in step S28-2 that data has not been received from the HDMI device (No), the process for determining the device type is terminated. On the other hand, if it is determined in step S28-2 that data is received from the HDMI device (Yes), the process proceeds to the next step. If the determination in step 28-1 or step S28-2 is No, if the terminal 10 has a built-in camera, the communication session for sharing the video of the built-in camera is continued, and the camera is built-in. If not, the communication session can be continued as an audio-only conference that does not share images and screens.

  In the next step, the HDMI input unit 20 extracts an image from the data input via the HDMI input I / F 118, and transmits the image to the determination unit 21 (step S28-3). Next, the determination unit 21 that has received the image from the HDMI input unit 20 determines the type of the device connected to the HDMI input I / F 118 based on the image (step S28-4). As described above, the determination of whether the image is a natural image or a non-natural image executed by the determination unit 21 is performed by, for example, a method of extracting edge components of an image, a method of counting the number of used colors, It is possible to use a publicly known method such as a method of using a state of changing in the above, a method of determining in each divided area.

  If it is determined in step S28-4 that the device connected to the HDMI input I / F 118 is a PC (PC), the CODEC / media control is performed on the image data input via the HDMI input I / F 118. The unit 22 performs post-processing for PC (step S28-5). Here, the post-processing for PC refers to media control processing and encoding processing optimized for PC. For example, the media control process optimized for a PC is to thin out the frame rate without changing the resolution of received image data in the previous stage of encoding the image data. Specifically, for example, the frame rate is thinned out from 60 fps / 30 fps to 0.5 fps. For example, the encoding process optimized for PC is H.264. H.265 (HEVC) is selected as the algorithm of the encoding technique, and image data is encoded using this.

  On the other hand, if it is determined in step S28-4 that the device connected to the HDMI input I / F 118 is a camera (camera), the image data input via the HDMI input I / F 118 is converted to CODEC / The media control unit 22 performs camera post-processing (step S28-6). Here, the post-processing for the camera refers to media control processing, encoding processing, mirroring processing, and the like optimized for the camera. For example, the media control process optimized for a camera is to resize the resolution and further thin out the frame rate before encoding the image data. Specifically, for example, the resolution is resized from 1280 × 720 to 640 × 360, and the frame rate is thinned out from 60 fps / 30 fps to 20 fps. For example, the encoding process optimized for a camera is encoding image data using an algorithm suitable for a natural image. Further, in the camera post-processing, the image data may be subjected to a mirroring process.

  When the process of step S28-5 or step S28-6 ends, the process flow ends, and the communication session continues as a meeting for sharing images and screens.

  As described above, in the processing method in the transmission system 1 according to the present embodiment, the device type connected to the HDMI input I / F 118 can be automatically determined, so the optimum setting according to the device type can be set by the user. It can be done without operation.

[Second Embodiment]
Next, a second embodiment of the terminal, method, and program according to the present invention will be described with reference to FIGS. However, since the first embodiment and the second embodiment have many common configurations, only the differences between the first embodiment and the second embodiment will be described. For example, the external appearance and hardware configuration of the terminal, the management system in the transmission system, and the hardware configuration and functional configuration of the relay device are the same in the first embodiment and the second embodiment, and thus the description is omitted. And Accordingly, it should be understood that the description omitted in the description of the second embodiment below is common to the first embodiment unless otherwise specified.

  FIG. 29 is a functional block diagram of each terminal, device, and system configuring the transmission system of the second embodiment. In FIG. 29, the terminal 10 a, the relay device 30, and the management system 50 are connected so that data communication can be performed via the communication network 2. Further, the PC 10 is connected to the terminal 10a as an example of the HDMI device. The PC 70 is an example of an HDMI device, and can be exchanged with an HDMI device such as a camera.

  Below, the structure of the terminal 10a and the structure of PC70 different from 1st Embodiment are demonstrated in detail. In describing the functional configurations of the terminal 10a according to the second embodiment, the hardware configuration of the terminal 10 according to the first embodiment shown in FIG.

  As illustrated in FIG. 29, the terminal 10a includes the transmission / reception unit 11, the operation input reception unit 12, the login request unit 13, the imaging unit 14, the voice input unit 15a, the voice output unit 15b, and the final, as in the first embodiment. A narrowing unit 16, a display control unit 17, a delay detection unit 18, a storage / read processing unit 19, an HDMI input unit 20, a determination unit 21, and a CODEC / media control unit 22 are provided. Furthermore, the terminal 10a according to the second embodiment includes an HDMI control unit 23. Each of these units is a function realized by any one of the constituent elements shown in FIG. 5 being operated by a command from the CPU 101 according to the terminal program developed from the flash memory 104 onto the RAM 103, or It is a means to function. Further, the terminal 10a has a storage unit 2000 constructed by the RAM 103 shown in FIG. 5 and the flash memory 104 shown in FIG.

  As shown in FIG. 29, the PC 70 includes an HDMI output unit 71, an HDMI control unit 72, and a general-purpose PC architecture 73. The HDMI output unit 71 and the HDMI control unit 72 may be configured as a function realized by operating according to a command from the CPU according to a program stored in a storage device in the general-purpose PC architecture 73. Further, as described above, the PC 70 is an example of an HDMI device, and may be another HDMI device such as a camera as long as it is an HDMI device having the HDMI output unit 71 and the HDMI control unit 72.

  The HDMI control unit 23 of the terminal 10a and the HDMI control unit 72 of the PC 70 are control units for performing CEC (Consumer Electronics Control) communication with each other. Here, the CEC communication indicates a device control signal and a control protocol defined by the HDMI standard (HDMI-CEC Version 1.3a). The HDMI input unit 23 of the terminal 10a and the HDMI output unit 71 of the PC 70 are compliant with HDMI-CEC Version 1.3a, and can process device control signals and control protocols for CEC communication. In the following description, description will be given according to the device control signal and control protocol of this CEC communication.

  FIG. 30 is a sequence diagram of CEC communication performed between the terminal and the PC and CEC communication performed between the terminal and the camera. FIG. 30A is a sequence diagram of CEC communication performed between the terminal and the PC, and FIG. 30B is a sequence diagram of CEC communication performed between the terminal and the camera. The transmission / reception shown in FIG. 30 (a) or (b) is performed after the communication session for image data and audio data in FIG. 24 (step S69) or the communication session for image data and audio data in FIG. 27 (step S168). ) After that, it will be executed.

  As illustrated in FIG. 30A, the HDMI control unit 23 of the terminal 10a instructs the HDMI input unit 23 to transmit <Vendor Command: GetDeviceType>. Receiving the instruction, the HDMI input unit 23 transmits <Vendor Command: GetDeviceType> to the HDMI output unit 71 of the PC 70 (S30a-1). On the other hand, the HDMI output unit 71 of the PC 70 that has received <Vendor Command: GetDeviceType> from the HDMI input unit 23 of the terminal 10a sends <Vendor Command: ReportDeviceType> [“PC”], which is a device type signal, to the HDMI input unit of the terminal 10a. 23 is returned (S30a-2). Upon receiving <Vendor Command: ReportDeviceType> [“PC”], the HDMI input unit 23 of the terminal 10a transmits the result to the determination unit 21. The determination unit 21 connects to the HDMI input unit 23 based on the reception result. The type of the HDMI device being used is determined. The determination unit 21 transmits the determination result to the CODEC / media control unit 22, and the subsequent processing is the same as in the first embodiment.

  The CEC communication can also be transmitted from the PC 70. The HDMI output unit 71 can notify the HDMI control unit 72 that the screen has been output. When receiving the screen output notification, the HDMI control unit 72 instructs the HDMI output unit 71 to transmit a device type signal <Vendor Command: SetDeviceType> [“PC”] (S30a-3). Upon receiving <Vendor Command: SetDeviceType> [“PC”], the HDMI input unit 23 of the terminal 10a transmits the result to the determination unit 21, and the determination unit 21 connects to the HDMI input unit 23 based on the reception result. The type of the HDMI device being used is determined.

  The same applies when a camera is connected to the HDMI input unit 23. As shown in FIG. 30B, the HDMI control unit 23 of the terminal 10a instructs the HDMI input unit 23 to transmit <Vendor Command: GetDeviceType>. Upon receiving the instruction, the HDMI input unit 23 transmits <Vendor Command: GetDeviceType> to the HDMI output unit of the camera (S30b-1). On the other hand, the HDMI output unit of the camera that has received <Vendor Command: GetDeviceType> from the HDMI input unit 23 of the terminal 10a sends <Vendor Command: ReportDeviceType> [“Camera”], which is a device type signal, to the HDMI input unit 23 of the terminal 10a. (S30b-2). Upon receiving <Vendor Command: ReportDeviceType> [“Camera”], the HDMI input unit 23 of the terminal 10a transmits the result to the determination unit 21, and the determination unit 21 connects to the HDMI input unit 23 based on the reception result. The type of the HDMI device being used is determined.

  The same applies when CEC communication is started from the camera. When receiving the screen output notification, the HDMI control unit 72 instructs the HDMI output unit 71 to transmit a device type signal <Vendor Command: SetDeviceType> [“Camera”] (S30b-3). When the HDMI input unit 23 of the terminal 10a receives <Vendor Command: SetDeviceType> [“Camera”], the result is transmitted to the determination unit 21, and the determination unit 21 is connected to the HDMI input unit 23 based on the reception result. The type of the HDMI device being used is determined.

  Here, a processing flow for determining a device type connected to the terminal 10a will be described with reference to FIGS. The processing flow shown in FIG. 31 is executed after the image data and audio data communication session is established in FIG. 24 (step S69) or after the image data and audio data communication session is established in FIG. 27 (step S168). The Further, this processing flow corresponds to step S30a-1 and step S30a-2 in FIG.

  As shown in FIG. 31, after establishing a communication session (steps S69 and S168), the HDMI input unit 20 of the terminal 10a determines whether or not an HDMI device is connected to the HDMI input I / F 118 (step S31-1). ). If it is determined in step S31-1 that the HDMI device is not connected (No), the process for determining the device type is terminated. On the other hand, if it is determined in step S31-1 that the HDMI device is connected (Yes), the process proceeds to the next step.

  In the next step, CEC data communication is performed (step S31-2). That is, the HDMI control unit 23 of the terminal 10a instructs the HDMI input unit 23 to transmit <Vendor Command: GetDeviceType>. Receiving the instruction, the HDMI input unit 23 transmits <Vendor Command: GetDeviceType> to the HDMI output unit 71 of the PC 70 (S30a-1).

  In the next step, the device type is determined (step S31-3). Based on <Vendor Command: GetDeviceType> received from the HDMI device connected to the HDMI input I / F 118, the determination unit 21 determines the type of the HDMI device connected to the HDMI input I / F 118.

  If it is determined in step S31-3 that the device connected to the HDMI input I / F 118 is a PC (PC), the CODEC / media control unit for the data input via the HDMI input I / F 118 22 performs post-processing for PC (step S31-4). Here, the post-processing for PC refers to media control processing and encoding processing optimized for PC. For example, the media control process optimized for a PC is to thin out the frame rate without changing the resolution of received image data in the previous stage of encoding the image data. Specifically, for example, the frame rate is thinned out from 60 fps / 30 fps to 0.5 fps. For example, the encoding process optimized for PC is H.264. H.265 (HEVC) is selected as the algorithm of the encoding technique, and image data is encoded using this.

  On the other hand, if it is determined in step S31-3 that the device connected to the HDMI input I / F 118 is a camera (camera), the CODEC / media for the data input via the HDMI input I / F 118 is used. The control unit 22 performs camera post-processing (step S31-5). Here, the post-processing for the camera refers to media control processing, encoding processing, mirroring processing, and the like optimized for the camera. For example, the media control process optimized for a camera is to resize the resolution and further thin out the frame rate before encoding the image data. Specifically, for example, the resolution is resized from 1280 × 720 to 640 × 360, and the frame rate is thinned out from 60 fps / 30 fps to 20 fps. For example, the encoding process optimized for a camera is encoding image data using an algorithm suitable for a natural image. Further, in the camera post-processing, the image data may be subjected to a mirroring process.

  When the process of step S31-5 or step S31-6 ends, the process flow ends, and the communication session is continued as a meeting for sharing images and screens.

  FIG. 32 is a processing flow of software operating on the PC. This processing flow corresponds to step S30a-3 in FIG. First, initialization is performed (step S32-1). This initialization means turning off a “flag indicating that data transmission has been completed” used in the subsequent stage. Next, it is determined whether or not the HDMI output unit 71 is outputting HDMI (step S32-2). If it is determined in step S32-2 that HDMI output has not been performed (No), the process returns to the initialization process (step S32-1). On the other hand, if it is determined that HDMI output is being performed (Yes), the process proceeds to the next step.

  In the next step, it is determined whether or not <Vendor Command: SetDeviceType> [“PC”] ”has already been transmitted by CEC communication (step S32-3). For the determination in step S32-3, a “flag indicating that data transmission has been completed” is used. If the “flag indicating that data transmission has been completed” is ON, it is determined that <Vendor Command: SetDeviceType> [“PC”] has already been transmitted (Yes), and the process returns to step S32-2. On the other hand, if the “flag indicating that data transmission has been completed” is OFF, it is determined that <Vendor Command: SetDeviceType> [“PC”] ”has not been transmitted yet (No), and the process proceeds to the next step.

  In the next step, the HDMI control unit 72 instructs the HDMI output unit 71 to transmit <Vendor Command: SetDeviceType> [“PC”], and the HDMI output unit 71 instructs the HDMI input unit 23 of the terminal 10a. <Vendor Command: SetDeviceType> [“PC”] is transmitted (step S32-4). At this time, the HDMI control unit 72 changes the “flag indicating that data transmission has been completed” to ON.

  Thereafter, a software end determination is made (step S32-4). That is, if the software has not ended (No), the process returns to step S32-2 to continue the process. On the other hand, when the software is finished (Yes), this processing flow is finished.

DESCRIPTION OF SYMBOLS 1 Transmission system 10, 10a, 10aa, 10ab, ..., 10ba, 10bb, ..., 10ca, 10cb, ..., 10da, 10db, ... Terminal 11 Transmission / reception part 12 Operation input reception part 13 Login request part 14 Imaging part 15a Voice input part 15b Audio output unit 16 Final narrowing unit 17 Display control unit 18 Delay detection unit 19 Storage / read processing unit 20 HDMI input unit 21 Determination unit 22 CODEC / media control unit 23 HDMI control units 30, 30a, 30b, 30c, 30d Relay Device 31 Transmission / reception unit 32 Status detection unit 33 Data quality confirmation unit 34 Change quality management unit 35 Data quality change unit 39 Storage / read processing unit 50 Management system 51 Transmission / reception unit 52 Terminal authentication unit 53 Status management unit 54 Terminal extraction unit 55 Terminal state Acquisition unit 56 Primary filtering unit 57 Session management unit 58 Quality determination unit 59 And reading processing unit 60 delay time management unit 70 PC
71 HDMI output unit 72 HDMI control unit 73 General-purpose PC architecture 70a, 70b, 70c, 70d, 70ab, 70cd Router 90 Program providing system 120, 120aa, 120ab, ..., 120ba, 120bb, ..., 120ca, 120cb, ..., 120da, 120 db, ... Display 100 Maintenance system 1000 Case 1001 Right upper surface 1002 Operation panel 1003a to 1003e Operation button 1004 Power switch 1005 Alarm lamp 1006 Sound output surface 1007 Left upper surface 1008 Housing portion 1010 Arm 1011 Torque hinge 1020 Camera housing 1021 Torque hinge 1050 Display 2000 storage unit 3000 storage unit 3001 change quality management DB
5000 storage unit 5001 relay device management DB
5002 Terminal authentication management DB
5003 Terminal management DB
5004 Destination list management DB
5005 Session management DB
5006 Priority management DB (an example of priority management means)
5007 Quality Management DB

JP 2008-48136 A

Claims (10)

A terminal that transmits image data to a partner terminal via a communication network,
An input unit through which the image data is input from an external device;
A determination unit that determines a type of the external device connected to the input unit based on data input from the input unit;
A control unit that controls image processing on the image data based on the type determined by the determination unit;
A terminal comprising: The determination unit determines a type of the external device connected to the input unit based on image data input from the input unit;
The terminal according to claim 1. The determination unit determines that an image included in the image data is a natural image or a non-natural image, and determines a type of the external device connected to the input unit based on the determination;
The terminal according to claim 2. A control unit that performs control to request that the external device returns a device type signal from the input unit to the external device;
The determination unit determines the type of the external device connected to the input unit based on the device type signal returned from the external device.
The terminal according to claim 1.   The terminal according to claim 4, wherein the input unit is an HDMI standard input interface, and the device type signal is a signal defined in the HDMI standard CEC communication.   The terminal according to claim 1, wherein the control unit executes control for changing a resolution and a frame rate of the image as control of the image processing.   The said control part performs the mirroring process of the said image as control of the said image process when an external apparatus is judged to be a camera by the said judgment part, The any one of Claims 1-6 characterized by the above-mentioned. Terminal.   The terminal according to claim 1, wherein the control unit selects or switches an algorithm for encoding the image data based on the type. A method of controlling image processing on the image data in a terminal that transmits image data to a counterpart terminal via a communication network, the terminal comprising:
A determination step of determining a type of an external device connected to an input unit to which the image data of the terminal is input based on data input from the input unit;
A control step for controlling image processing on the image data based on the type determined by the determination step;
How to run. A program for a terminal that transmits image data to a partner terminal via a communication network,
A determination step of determining a type of an external device connected to an input unit to which the image data of the terminal is input based on data input from the input unit;
A control step for controlling image processing on the image data based on the type determined by the determination step;
For causing the terminal to execute.

Images