JP7271080B2 - Communication device, communication system, communication method, and program - Google Patents

Description

The present invention relates to a technique for mitigating the effects of packet loss that occurs on a communication network in packet communication.

Due to recent technological advances in mobile terminals such as smartphones, there are an increasing number of cases where mobile terminals connected to wireless communication networks such as mobile phone networks perform real-time communication.

In wireless communication networks, packet loss often occurs due to radio wave conditions and the like. Known techniques for mitigating the effects of packet loss include packet retransmission techniques such as TCP (Transmission Control Protocol) and techniques such as FEC (Forward Error Correction) that provide signal redundancy. there is

JP 2013-030890 A

However, packet retransmission technology such as TCP goes through the process of detecting packet loss and retransmitting it, so waiting time occurs before the retransmitted data can be used. It is difficult to use.

In addition, in a technology such as FEC that provides signal redundancy, missing parts are supplemented by using information immediately before and after the missing part. In such a case, a so-called burst loss in which all the signals are lost is observed, and in such a case, the missing part cannot be compensated for by the technique of imparting redundancy to the signal.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for alleviating the effects of packet loss on a communication network in packet communication.

According to the disclosed technology, packet distribution means for assigning sequence numbers to sequentially received packets and distributing each sequence-numbered packet to one communication network or a plurality of communication networks;
transmitting means for transmitting each packet distributed to the one communication network or the plurality of communication networks to a receiving side using the distribution destination communication network;
The packet sorting means sorts specific packets among sequentially received packets to a specific communication network, and sorts packets other than the specific packets to a communication network other than the specific communication network,
When the packet is a video packet, the packet sorting means performs image recognition using the packet to determine whether the packet is a packet of a video portion corresponding to the central visual field of the user. A communication device is provided that determines whether or not the packet is the specific packet by doing so.

According to the disclosed technology, in packet communication, it is possible to mitigate the effects of packet loss on a communication network.

1 is a configuration diagram of a communication system according to an embodiment of the present invention; FIG. 2 is a functional configuration diagram of the communication device 100; FIG. 2 is a functional configuration diagram of the communication device 200; FIG. It is a figure which shows the example of a hardware configuration. FIG. 3 is a diagram for explaining an operation example 1 of a communication system; FIG. 11 is a diagram for explaining an operation example 2 of the communication system; FIG. 11 is a configuration diagram of a terminal 300 in operation example 3; FIG. 13 is a diagram for explaining an operation example 3 of the communication system;

An embodiment (this embodiment) of the present invention will be described below with reference to the drawings. The embodiments described below are merely examples, and embodiments to which the present invention is applied are not limited to the following embodiments.

(System configuration)
FIG. 1 is a diagram showing a configuration example of a communication system according to this embodiment. As shown in FIG. 1, the communication system has a transmission-side communication device 100 and a reception-side communication device 200, which are connected to a communication network A, a communication network B, and a communication network C. As shown in FIG. A terminal 300 is connected to the communication device 100 , and a terminal 400 is connected to the communication device 200 .

Communication network A, communication network B, and communication network C are communication networks that perform wireless communication, such as mobile phone networks and wireless LANs, respectively. For example, communication network A is the communication network of mobile phone company D, communication network B is the communication network of mobile phone company S, and communication network C is the communication network of mobile phone company K. However, it is not limited to this, and any one or all of communication network A, communication network B, and communication network C may be a wired communication network (eg, fixed telephone network). In this embodiment, lines of a plurality of communication networks are bundled and used as a virtual high-quality line.

Although FIG. 1 shows an example of using three communication networks, the number of communication networks to be used is not limited to three. For example, two or more than three communication networks may be used. may be used.

In the configuration of FIG. 1, the communication device 100 sequentially receives packets from the terminal 300, assigns sequence numbers to the received packets, and sends each sequence-numbered packet to the communication network A, the communication network B, and the communication network C. distributed to one or a plurality of communication networks among them and transmitted.

The communication device 200 receives packets transmitted by the communication device 100 from the communication network A, the communication network B, and the communication network C, and transmits the packets in the order of the sequence numbers based on the sequence numbers assigned to the packets. Send to terminal 400 .

It is assumed that the communication device 100 has a packet transmission function and the communication device 200 has a packet reception function. good too. Note that each communication device may be referred to as a "controller".

The terminal 300 on the transmission side is, for example, a device equipped with a camera, and in that case, the terminal 300 transmits a packet of video captured by the camera. In this case, terminal 400 on the receiving side displays video by decoding and reproducing the data of the received packet.

It is only an example that the transmitted packets are video packets. The technology according to the present invention is basically applicable regardless of the packet type.

FIG. 2 is a functional configuration diagram of the communication device 100 on the transmission side. As shown in FIG. 2 , the communication device 100 has an input unit 110 , a sequence number assigning unit 120 , a packet distribution processing unit 130 and a communication unit 140 .

The input unit 110 receives packets output from the terminal 300 . The sequence number assigning unit 120 assigns sequence numbers to packets in the order in which they are input. The packet distribution processing unit 130 determines a communication network to be used for transmitting each packet to the opposite communication device 200, and distributes the packet to the determined communication network. Note that when the packet distribution processing unit 130 distributes the same packet to a plurality of communication networks, the packet is duplicated (copied), and the original packet and the duplicated packet are distributed to the plurality of communication networks. More specifically, the packet distribution processing unit 130 distributes packets to a certain communication network, for example, by passing packets to a functional unit that performs wireless communication using the communication network in the communication unit 140. be.

The communication unit 140 includes a functional unit that performs communication using the communication network A, a functional unit that performs communication using the communication network B, and a functional unit that performs communication using the communication network C. The communication unit 140 also includes a function of measuring the quality of each of communication network A, communication network B, and communication network C (reception strength (RSRP, etc.), reception quality (RSSI), delay, etc.). The measured quality of each communication network is passed to the packet distribution processor 130 . The packet distribution processing unit 130 can use the quality of each communication network when selecting a communication network to which packets are to be distributed.

FIG. 3 is a functional configuration diagram of the communication device 200 on the receiving side. As shown in FIG. 3, the communication device 200 has a communication section 210, a packet order sorting section 220, and an output section 230. FIG.

The communication unit 210 includes a functional unit that performs communication using the communication network A, a functional unit that performs communication using the communication network B, and a functional unit that performs communication using the communication network C. Communication unit 210 also includes a function for measuring the quality of each communication network.

The packet sequence sorting unit 220 has a function of sorting the packets in sequence number order using the sequence numbers of the received packets. Also, when receiving a plurality of packets with the same sequence number from the communication unit 210, the packet order sorting unit 220 selects one packet from the plurality of packets and discards the other packets. For selection, the packet received from the communication network with the highest quality may be selected from among the plurality of packets.

The output unit 230 deletes the sequence number from each packet delivered from the packet order sorting unit 220 in order of the sequence number, and outputs the packet to the terminal 400 .

Each of the devices described above (terminal 300, communication device 100, communication device 200, and terminal 400) can be realized by causing a computer to execute a program describing the processing content described in this embodiment. That is, the functions of the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into the computer. The above program can be recorded in a computer-readable recording medium (portable memory, etc.), saved, or distributed. It is also possible to provide the above program through a network such as the Internet or e-mail.

FIG. 4 is a diagram showing a hardware configuration example of the above device. The device of FIG. 4 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, etc., which are interconnected by a bus B, respectively. Note that the communication device 100 and the communication device 200 may not include the display device 1006 and the input device 1007 .

A program for realizing processing in the device is provided by a recording medium 1001 such as a CD-ROM or a memory card, for example. When the recording medium 1001 storing the program is set in the drive device 1000 , the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000 . However, the program does not necessarily need to be installed from the recording medium 1001, and may be downloaded from another computer via the network. The auxiliary storage device 1002 stores installed programs, as well as necessary files and data.

The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when a program activation instruction is received. The CPU 1004 implements functions related to the device according to programs stored in the memory device 1003 . The interface device 1005 is used as an interface for connecting to the network. For example, interface device 1005 is used as input unit 110 and communication unit 140 in communication device 100 . Also, for example, the interface device 1005 is used as the communication unit 210 and the output unit 230 in the communication device 200 .

A display device 1006 displays a program-based GUI (Graphical User Interface) or the like. An input device 1007 is composed of a keyboard, a mouse, buttons, a touch panel, or the like, and is used to input various operational instructions.

An operation example of the communication system having the configuration described above will be described below. In the description of the operation example, the names of the functional units shown in FIGS. 2 and 3 are appropriately used. Note that which operation to perform can be switched, for example, by setting the communication devices 100 and 200 .

(Operation example 1)
First, operation example 1 will be described with reference to FIG. In FIG. 5, packets are indicated by numbers (corresponding to sequence numbers). For convenience, packets input to the communication device 100 (packets before sequence numbers are assigned) and packets output from the communication device 200 (packets after sequence numbers are deleted) are also indicated by numbers.

In operation example 1 shown in FIG. 5, each packet that is input to the communication device 100 and assigned a sequence number is transmitted to all communication networks. That is, as shown in FIG. 5, packets are sequentially input to the communication apparatus 100, and the sequence number assignment unit 120 assigns packets such as packet 1, packet 2, packet 3, packet 4, and packet 5 in the order of the input packets. is given a sequence number. The packet distribution processing unit 130 duplicates the packet 1 and distributes it to the communication network A, the communication network B, and the communication network C. FIG. The communication unit 140 transmits the packet 1 over the communication network A, transmits the packet 1 over the communication network B, and transmits the packet 1 over the communication network C. FIG. An operation similar to that for packet 1 is performed for each of packets 2 to 5 as well.

Packet 1, packet 2, packet 3, packet 4, and packet 5 are transmitted to communication device 200 through communication network A, communication network B, and communication network C, respectively. However, during the period from transmission to arrival of the packet at the communication device 200, packet loss occurs due to, for example, deterioration of radio wave conditions, and the communication unit 210 of the communication device 200 receives packet 1, packet 3, receive packets 1, 2, 3 and 4 from communication network B, and receive packets 1, 3 and 5 from communication network C.

The packet order sorting unit 220 of the communication device 200 selects one packet 1 from the three received packets 1, selects one packet 3 from the three received packets 3, packet 1, packet 2, packet 3, The packets are passed to the output unit 230 in order of packet 4 and packet 5, and the output 230 outputs the packets with the sequence numbers deleted in the correct order.

By transmitting the same packet over a plurality of communication networks (here, all communication networks connected to the communication device 100) as in operation example 1, even if packet loss occurs in some communication networks, the remaining If no packet loss occurs in the communication network, the communication device 200 on the receiving side can receive the packet. In general, there are communication networks with good quality and communication networks with poor quality in a plurality of communication networks. more likely to be received. Also, even if packet losses occur, the number of such losses can be kept small.

(Operation example 2)
Next, operation example 2 will be described with reference to FIG. In operation example 2, the communication device 100 transmits the same packet through one communication network, but distributes sequentially received packets to a plurality of communication networks to improve the communication speed.

In the example shown in FIG. 6, packets are sequentially input to the communication apparatus 100, and the sequence number assigning unit 120 assigns a sequence such as packet 1, packet 2, packet 3, packet 4, and packet 5 in the order of the input packets. A number is given. The packet distribution processing unit 130 distributes packets 1 and 2 to the communication network A, and distributes packets 3, 4, and 5 to the communication network C. FIG. These packets are transmitted by the communication unit 140 through the corresponding communication network.

FIG. 6 shows an example in which no packet loss occurred because communication networks A and C with good quality were used.

In the communication device 200 on the receiving side, the communication unit 210 outputs packets 1 and 2 received from the communication network A, and packets 3, 4, and 5 received from the communication network C from the output unit 230 in the correct order. be done.

The packet distribution processing unit 130 on the transmission side may, for example, distribute packets to communication networks excluding the communication network with the worst quality, or distribute packets to a plurality of communication networks up to the top N (N is a natural number) in quality. may be distributed.

For example, the packet distribution processing unit 130 may distribute odd-numbered packets to the communication network A and even-numbered packets to the communication network B, for example. Alternatively, the packet distribution processing unit 130 may perform distribution based on the remainder obtained by dividing the sequence number by the number of communication networks. For example, in the example shown in FIG. 6 where the number of communication networks is 3, if the remainder of dividing the sequence number by 3 is 0, the packet distribution processing unit 130 distributes the sequence number to communication network A, and the remainder of dividing the sequence number by 3. If the sequence number is 1, the sequence number is assigned to the communication network B, and if the remainder obtained by dividing the sequence number by 3 is 2, the sequence number is assigned to the communication network C.

In addition, operation example 1 may be combined. For example, in the example of FIG. 6, when communication network C has the best quality, packets 3, 4, and 5 are transmitted using communication network C, and packets 1 and packet 2 may be transmitted.

Note that the packet transfer speed in a wide area network (mobile phone network, Internet, etc.) is generally lower than the packet transfer speed in a private network (user's home network, etc.), so application of Operation Example 2 is effective.

That is, the speed of communication from the communication device 100 to the communication device 200 is significantly lower than the speed of communication from the terminal 300 to the communication device 100 and from the communication device 200 to the terminal 400, which is communication on the private network. Therefore, the communication from the communication device 100 to the communication device 200 can become a bottleneck of the entire system. can be avoided.

(Operation example 3)
Next, operation example 3 will be described. Operation example 3 assumes a video transfer application for VR (Virtual Reality), but the application of operation example 3 is not limited to VR.

The human viewing angle is about 90 degrees in the horizontal direction, but the person is concentrating on the central 40 degrees, and the peripheral vision is not even colored due to the structure of the eyeball. Therefore, in operation example 3, the user's line-of-sight direction is detected by a sensor in terminal 300 on the transmission side, packets of the video portion corresponding to the central visual field are transferred through a high-quality communication network, and packets other than the video portion corresponding to the central visual field are transferred. packets are sent over a communication network that is not of high quality.

FIG. 7 shows a configuration example of the terminal 300 in this case. As shown in FIG. 7 , the terminal 300 has a line-of-sight detection unit 310 , a camera 320 and a packet transmission unit 330 . The line-of-sight detection unit 310 detects the position of the user's line of sight (for example, a circular range of a predetermined size that the user is looking at intensively) within the screen range of the video captured by the camera 320, and detects the position of the user's line of sight. information to the packet transmission unit 330 .

The packet transmission unit 330 divides the screen to packetize the video captured by the camera 320, and acquires the position on the screen corresponding to the packet. Then, for example, the packet transmission unit 330 attaches an identifier (which may be referred to as metadata) indicating this to the packet of the video portion corresponding to the position of the user's line of sight, and transmits the packet to the communication device 100 . Identifiers are not attached to packets of video portions that do not correspond to the position of the user's line of sight.

In the communication device 100 that receives the packet + identifier or only the packet, sequence numbers are assigned to the packets as described above, and the packet distribution processing unit 130 sorts packets with identifiers. , for example, to the communication network with the highest quality (for example, communication network A), and packets without identifiers are distributed to communication networks other than the communication network with the highest quality (for example, communication networks B and C).

Since packets with identifiers are important packets, the packet distribution processing unit 130 applies operation example 1 and distributes them to both of multiple high-quality communication networks (for example, communication networks A and B). may

Packets without identifiers are assumed to have a large amount of data (the number of packets), so operation example 2 may be applied to transmit the packets using multiple communication networks.

The communication device 200 on the receiving side transmits the packets received from each communication network to the terminal 400 in the order of the sequence numbers. In the terminal 400, for example, the image of the central vision transferred through the high-quality communication network and the image of the peripheral vision transferred through the low-quality communication network are combined and displayed.

FIG. 8 shows an image of the above processing. As shown in FIG. 8, packets of the boy's image corresponding to the central vision portion are transferred through a high-quality communication network, and packets of the peripheral vision image are transferred through a low-quality communication network. On the receiving side, these are combined and displayed.

In the example of FIG. 8, for example, if a boy's video is delivered in high quality, user satisfaction will not be significantly reduced even if noise appears in a river video transmitted over a low-quality communication network. Also, since the user does not observe the peripheral vision intensively, the noise may not be noticed.

<Example without identifier (metadata)>
On the transmitting side, the terminal 300 may not have a function of adding identifiers to packets. In that case, for example, in the communication device 100 that has received video packets from the terminal 300, the packet sorting processing unit 130 performs image recognition using the received packets, so that the range of the central portion of the video screen (eg, A packet corresponding to a circular range of a predetermined size) is identified, the packet is distributed to a high-quality communication network in the same manner as the packet with the above identifier, and other packets are transferred to a high-quality communication network. It may be distributed to other communication networks. Further, the packet distribution processing unit 130 performs image recognition using the received packets, thereby distinguishing between packets corresponding to the range of the central portion of the video screen and packets not corresponding to the range of the central portion of the video screen. may be generated and sent as described above.

In addition, the packet distribution processing unit 130 performs image recognition using the received packets to identify the packets corresponding to the person in the video screen, and converts the packets into packets with the above identifiers. In the same way as in , the packets may be distributed to the high-quality communication network, and the other packets may be distributed to the communication networks other than the high-quality communication network. In addition, the packet distribution processing unit 130 performs image recognition using the received packets to identify the packets corresponding to the person in the video screen and the packets corresponding to the person in the video screen. The above transmission may be performed by generating a packet that does not

<Sound example>
In operation example 3, an example of transmitting video is shown, but similar processing can be applied to the case of transmitting sound. For example, the terminal 300 collects sound in an environment where the user speaks, and transmits packets of the collected sound to the communication device 100 . The packet distribution processing unit 130 of the communication device 100 that has received the packet performs speech recognition using the received packet, thereby separating packets of the user's uttered voice and other packets (eg, packets of environmental sound). It is also possible to separate packets of uttered voices into high-quality communication networks in the same manner as packets with identifiers, and to distribute other packets to communication networks other than the high-quality communication networks. That is, the packet distribution processing unit 130 identifies speech packets of the speaker and other sound packets, and distributes them to each communication network as described above.

<Others>
Also, in the above example, packets of video and audio that are considered to be important are preferentially transferred to a high-quality communication network. is not limited to

For example, assuming that the terminal 300 is a robot that walks by itself, the terminal 300 (robot) converts various data grasped by its own sensors (including a camera, a microphone, etc.) into packets to the receiving terminal 400 (eg, control (user's terminal).

In this case, for example, it is set to the terminal 300 (robot) that it is important to send high-quality images necessary for maneuvering, and the terminal 300 (robot) transmits packets of images detected by the camera. , is attached with an identifier indicating that it is important, and is transmitted to the communication device 100 . The terminal 300 (robot) transmits various packets indicating surrounding conditions to the communication device 100 in addition to the above packets. The communication device 100 that has received the packet transmits the packet with the identifier over the high-quality communication network, and transmits the other packets over the communication network other than the high-quality communication network. As a result, the user of the terminal 400 on the receiving side can quickly grasp the situation of the robot and perform appropriate maneuvers.

Further, for example, when the terminal 300 (robot) meets and talks with someone, priority is given to the sound quality of the conversation, and the communication device 100 transmits voice packets of the conversation over a high-quality communication network. good too. Further, for example, when the terminal 300 (robot) starts to work, tactile information may be prioritized over voice, and the communication device 100 may transmit packets of the tactile information through a high-quality communication network. .

Also, for example, the terminal 300 (robot) is set to be important to approach a person, and the terminal 300 (robot) detects that the distance from the person detected by the sensor is equal to or less than a predetermined threshold. In this case, a packet of data indicating this may be sent to the communication device 100 with an identifier indicating that it is important. The terminal 300 (robot) transmits various packets indicating surrounding conditions to the communication device 100 in addition to the above packets. The communication device 100 that has received the packet transmits the packet with the identifier over the high-quality communication network, and transmits the other packets over the communication network other than the high-quality communication network. As a result, the user of terminal 400 on the receiving side can quickly grasp the situation of a person approaching the robot.

Further, for example, when the terminal 300 (robot) reads a book and transmits the contents to the terminal 400 on the receiving side, in order to avoid errors due to packet loss, operation example 1 is applied to transmit packets. may be performed.

Also, in the communication device 100, the packet distribution processing unit 130 may use AI to determine which communication network is used to transmit the packet. For example, the AI is made to learn in advance the relationship between the status of received packets, such as the frequency, amount, and time of transmission, and the importance of packets, and the packet distribution processing unit 130 detects a specific status by the AI. Then, the packets at that time are distributed to high-quality communication networks.

(Summary of embodiment)
The technology according to the present embodiment makes it possible, for example, to mitigate the effects of packet loss in real-time communication in a wireless environment. For example, by using the radio waves of a plurality of mobile carriers, even if the radio waves of one carrier become extremely weak, it is possible to continue communication using the radio waves of the other carrier. Also, the technology according to the present embodiment can be implemented without updating the infrastructure on the communication network side.

As described above, according to the present embodiment, packet distribution means for assigning sequence numbers to sequentially received packets and distributing each sequence-numbered packet to one communication network or a plurality of communication networks. and transmitting means for transmitting each packet distributed to the one communication network or the plurality of communication networks to a receiving side using the distribution destination communication network. .

The packet sorting means, for example, sorts each packet assigned with the sequence number to all of the plurality of communication networks to which the communication device is connected.

The packet distribution means may select a communication network to which the packet is to be distributed, based on the sequence number attached to the packet.

The packet sorting means may sort specific packets among sequentially received packets to a specific communication network, and sort packets other than the specific packets to a communication network other than the specific communication network.

The packet sorting means may determine that a packet with a predetermined identifier is the specific packet. Further, when the packet is a video packet, the packet sorting means determines whether the packet is the specific packet by determining whether the packet corresponds to a specific portion of the video. It may be determined whether or not. Further, when the packet is a sound packet, the packet sorting means determines whether the packet is the specific packet by determining whether the packet corresponds to the voice of the speaker. It may be determined whether

Further, according to this embodiment, there is provided a communication system including the communication device and a receiving communication device that receives a packet transmitted from the communication device, wherein the receiving communication device receives the packet transmitted from the communication device. receiving means for receiving the packets received from the one communication network or the plurality of communication networks; and output means for outputting the packets sequentially received by the receiving means in the order of the sequence numbers attached to the packets. A communication system is provided comprising:

Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes can be made within the scope of the gist of the present invention described in the claims. It is possible.

100 communication device 110 input unit 120 sequence number assignment unit 130 packet distribution processing unit 140 communication unit 200 communication device 210 communication unit 220 packet order sorting unit 230 output unit 300 terminal 310 line-of-sight detection unit 320 camera 330 packet transmission unit 400 terminal 1000 drive device 1001 recording medium 1002 auxiliary storage device 1003 memory device 1004 CPU
1005 interface device 1006 display device 1007 input device

Claims (4)

a packet sorting means for assigning sequence numbers to sequentially received packets and sorting each sequence-numbered packet to one communication network or a plurality of communication networks;
transmitting means for transmitting each packet distributed to the one communication network or the plurality of communication networks to a receiving side using the distribution destination communication network;
The packet sorting means sorts specific packets among sequentially received packets to a specific communication network, and sorts packets other than the specific packets to a communication network other than the specific communication network,
When the packet is a video packet, the packet sorting means performs image recognition using the packet to determine whether the packet is a packet of a video portion corresponding to the central visual field of the user. The communication device determines whether or not the packet is the specific packet by A communication system comprising the communication device according to claim 1 and a receiving communication device that receives a packet transmitted from the communication device,
The receiving communication device,
receiving means for receiving packets transmitted from the communication device from the one communication network or the plurality of communication networks;
and output means for outputting the packets sequentially received by the receiving means in the order of the sequence numbers assigned to the packets. A packet communication method executed by a communication device,
a packet sorting step of assigning sequence numbers to sequentially received packets and sorting each sequence-numbered packet to one communication network or a plurality of communication networks;
a transmission step of transmitting each packet distributed to the one communication network or the plurality of communication networks to a receiving side using the distribution destination communication network;
In the packet sorting step, among the sequentially received packets, a specific packet is sorted to a specific communication network, and packets other than the specific packet are sorted to a communication network other than the specific communication network;
When the packet is a video packet, in the packet sorting step, image recognition is performed using the packet to determine whether or not the packet is a packet of a video portion corresponding to the central visual field of the user. and determining whether or not the packet is the specific packet. A program for causing a computer to function as each means in the communication device according to claim 1.

Images