WO2013183212A1 - Operation-shared command communication control system and operation-shared command communication control method - Google Patents

Operation-shared command communication control system and operation-shared command communication control method Download PDF

Info

Publication number
WO2013183212A1
WO2013183212A1 PCT/JP2013/002592 JP2013002592W WO2013183212A1 WO 2013183212 A1 WO2013183212 A1 WO 2013183212A1 JP 2013002592 W JP2013002592 W JP 2013002592W WO 2013183212 A1 WO2013183212 A1 WO 2013183212A1
Authority
WO
WIPO (PCT)
Prior art keywords
sharing command
network
operation sharing
side terminal
unit
Prior art date
Application number
PCT/JP2013/002592
Other languages
French (fr)
Japanese (ja)
Inventor
一彰 中島
Original Assignee
日本電気株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電気株式会社 filed Critical 日本電気株式会社
Priority to JP2014519807A priority Critical patent/JPWO2013183212A1/en
Publication of WO2013183212A1 publication Critical patent/WO2013183212A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/40—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection

Definitions

  • the present invention relates to an operation sharing command communication control system and an operation used for a communication system in which a portable terminal such as a smartphone or a PC (Personal Computer) is connected by a network in which usable bandwidth or delay varies such as the Internet or a wireless network.
  • the present invention relates to a shared command communication control method.
  • the information terminal In order to communicate with the other party using the information terminal such as a portable terminal or a PC via the Internet or a wireless network whose usable bandwidth and delay vary, the information terminal processes data according to the variation of the network. Therefore, it is necessary to send in accordance with the characteristics of the network. In many cases, the information terminal can cope with a gradual network change by measuring a network change and performing feedback type control based on the measured network change. However, the information terminal cannot follow the fluctuation when the fluctuation of the network is severe, and the quality is deteriorated.
  • Patent Document 1 discloses a feed-forward control method that predicts and controls network bandwidth fluctuations. This method predicts a change in the network state of a moving terminal, performs video distribution bit rate control according to the prediction result, and lowers the bit rate in advance when the network communication status is likely to deteriorate. It is a method of dealing with such as. Thereby, the user of the terminal can always take stable video communication.
  • screens can be obtained when remote users transmit operation sharing commands using a mobile terminal or a PC connected to the network.
  • operation sharing services that can share applications.
  • the operation sharing command is a command for a partner to share an operation, and is data indicating writing by an input operation using a touch panel or a mouse.
  • the bit rate of the video is changed based on the prediction result.
  • the bit rate of the video is changed based on the prediction result.
  • Non-Patent Document 1 and Non-Patent Document 2 lacks even one piece of data to be communicated, data transmission between the transmitting terminal and the receiving terminal is performed. Inconsistency will occur. That is, in the case of the operation sharing command, the loss leads to the subsequent data inconsistency, and thus quality control using the bit rate control described in Patent Document 1 cannot be performed. Therefore, in the operation sharing service, when the communication state of the network is deteriorated, the communication quality is deteriorated.
  • an object of the present invention is to provide an operation sharing command communication control system and an operation sharing command communication control method capable of reducing deterioration in communication quality when the network communication status in the operation sharing service deteriorates. To do.
  • An operation sharing command communication control system comprising a transmission side terminal and a reception side terminal connected via a network, and transferring and complementing an operation sharing command in an operation sharing service, wherein the transmission side terminal A network state prediction unit for predicting the network state, and when the network state is predicted to be larger than a predetermined value due to a change in available network bandwidth or a change in delay due to deterioration of the communication state of the network, Before becoming, a pre-corresponding data transmitting unit that transmits pre-corresponding data, which is data for coping with deterioration of the communication status of the network, and a value for which the fluctuation of the usable bandwidth is determined in advance If it is larger, the operation sharing command may be interrupted according to the available bandwidth.
  • An operation processing unit that transmits the operation sharing command to the receiving side terminal, and a transmission unit that transmits the operation sharing command to the receiving side terminal in a redundant session when variation in delay of the network is large.
  • the reception-side terminal generates a supplemented operation sharing command based on the pre-corresponding data storage device that stores the pre-corresponding data and the pre-corresponding data and the operation sharing command transmitted immediately before
  • An operation sharing command communication control system comprising:
  • An operation sharing command communication control method for transferring and complementing an operation sharing command in an operation sharing service using a transmission side terminal and a reception side terminal connected via a network, wherein the transmission side terminal is in a state of the network
  • the transmission side terminal is predicted that a change in available bandwidth or delay in the network will be larger than a predetermined value due to deterioration in the communication status of the network
  • the advance correspondence data which is data for coping with deterioration of the communication status of the network, is transmitted to the reception side terminal, and the transmission side terminal has predetermined fluctuations in the available bandwidth
  • the operation sharing command is thinned out according to the available bandwidth, and the transmission side
  • the operation sharing command is transmitted to the receiving terminal, and when the delay variation of the network is large, the operation sharing command is transmitted to the receiving terminal in a redundant session, and the receiving terminal
  • An operation sharing command that stores pre-correspondence data, and the receiving side terminal generates a complemented operation sharing command based on the pre-correspondence data
  • FIG. 1 is an explanatory diagram showing a configuration of an embodiment of an operation sharing command communication control system according to the present invention.
  • the operation sharing command communication control system of this embodiment includes a transmission side terminal 10, a reception side terminal 20, and a network 100 whose state varies.
  • a relay server may be interposed between the transmission side terminal 10 and the reception side terminal 20 in the same manner as devices in the network 100, but is omitted in FIG. 1 for the sake of simplicity.
  • the network 100 includes, for example, the Internet, a wireless network, and includes a large number of routers, a large number of lines, and the like, and connects the transmission side terminal 10 and the reception side terminal 20.
  • a state between the transmission side terminal 10 and the reception side terminal 20 is predicted.
  • the transmission side terminal 10 is operated by the user and transmits an operation sharing command to the reception side terminal 20. Further, the transmission side terminal 10 and the reception side terminal 20 may be configured to have the same functions and to send operation sharing commands in both directions.
  • the receiving terminal 20 receives and displays an operation sharing command and the like, thereby notifying the user on the receiver side of the operation result.
  • the transmission side terminal 10 includes an NW (Network) state prediction unit 11, a state determination unit 12, a prior correspondence data transmission unit 13, an operation generation unit 14, an operation processing unit 15, and a transmission unit 16.
  • NW Network
  • the transmission side terminal 10 includes an NW (Network) state prediction unit 11, a state determination unit 12, a prior correspondence data transmission unit 13, an operation generation unit 14, an operation processing unit 15, and a transmission unit 16.
  • the NW state prediction unit 11 monitors the transmission unit 16 and detects a change in the network state.
  • the network state is a change in available bandwidth and delay, and the NW state prediction unit 11 totals the transfer time of data transmitted and received.
  • the NW state prediction unit 11 acquires the transmission start time and the transmission end time when transferring the data measured by the transmission unit 16 for prediction, and the reception start measured by the reception unit 21 of the reception-side terminal 20 described later. Get time and reception end time.
  • the current available bandwidth is the value obtained by dividing the amount of transmitted data by the difference between the transmission start time and the reception end time.
  • the current delay is a value indicating the difference between the transmission start time and the reception start time.
  • the NW state prediction unit 11 aggregates these values for a certain period of time, and predicts that the fluctuation of the available bandwidth will continue to increase in the future when the variation of the available bandwidth tends to increase. Further, when the delay variation tends to increase, the NW state prediction unit 11 predicts that the delay variation will increase further in the future.
  • the prediction result of fluctuations in the available bandwidth and delay may have a predetermined width. For example, the change in the available bandwidth is expressed as “XX Mbps to xx Mbps”, and the delay change is expressed as “between ⁇ ms and ⁇ Mms”. This is because it is difficult to accurately predict network fluctuations.
  • the state determination unit 12 determines what processing should be performed in the future based on the network state predicted by the NW state prediction unit 11. When it is predicted that the fluctuation of the available bandwidth or the fluctuation of the delay becomes larger than a certain value and the state determination unit 12 may not be able to satisfy the assumed transmission time, the state determination unit 12 sends the advance correspondence data to the advance correspondence data transmission unit 13. Request to send
  • the state determination part 12 determines operation
  • the operation processing unit 15 is configured to thin out the operation sharing command that can be thinned out according to the predicted upper limit of the available bandwidth. Notice. By this processing, it is possible to prevent in advance that the available bandwidth is insufficient and the transmission of the operation sharing command is delayed.
  • the state determination unit 12 instructs the transmission unit 16 to send in a redundant session when the fluctuation in the available bandwidth is less than the specified value but the delay variation is predicted to exceed the specified value. put out.
  • the upper limit of the number of redundant sessions is the number obtained by dividing the predicted available bandwidth by the operation shared command transmission bandwidth (transmission data amount / desired transmission time) (rounded down). This ensures that data is transmitted as redundantly as possible within the available bandwidth, and even if any session experiences a delay, other sessions may be delayed with a low delay, The arrival time of the operation sharing command can be shortened.
  • the pre-correspondence data transmission unit 13 transmits the pre-correspondence data to the reception side terminal 20 via the transmission unit 16.
  • the prior correspondence data transmission unit 13 stabilizes the network at a stage where it is predicted that the network state may be deteriorated to a predetermined degree regardless of whether or not an operation sharing command described later is transmitted / received. Send proactive data during the time of day.
  • FIG. 2 is an explanatory diagram showing the data structure of the prior correspondence data.
  • the prior correspondence data D10 includes a transmission-side terminal status D20 and a countermeasure D30 for degradation.
  • the transmission side terminal status D20 records the internal and peripheral status of the transmission side terminal 10 at the predicted time point.
  • the transmission side terminal situation D20 includes the speed of the touch operation, the length D21, the inclination D22 of the transmission side terminal, the latest peripheral photograph D23, and the like.
  • the countermeasure D30 at the time of degradation includes a thinning method D31, a transmission redundancy D32, an alternative display means D33 being degraded, and an alternative sound effect D34 being degraded as countermeasures at the time of network degradation.
  • the speed and length D21 of the touch operation are information on the input device level such as a touch panel and a mouse before being converted into the operation sharing command.
  • the receiving terminal 20 can predict the user's writing to some extent and use it for storing the operation sharing command.
  • the slope D22 of the transmission side terminal 10 is data obtained by accumulating how the terminal is held and whether it is changing.
  • the inclination D22 of the transmission side terminal 10 for example, when there is no change in inclination, it is placed on the desk and used stably, but when the change in inclination is severe, it is estimated that it is held with one hand Is done. The estimated result is used for how much the blur width is set when the receiving terminal 20 complements the thinned operation sharing command.
  • the receiving side terminal 20 when the receiving side terminal 20 receives a handwritten stroke as an operation sharing command, when the fluctuation of the tilt is severe, the receiving side terminal 20 displays the handwritten with a certain width and a thickened stroke as an estimated value.
  • the reception side terminal 20 simply draws a handwritten stroke by performing spline interpolation when there is little variation in inclination.
  • the transmission side terminal 10 transmits and displays the latest peripheral photograph D23 to the reception side terminal 20 to display the transmission side terminal. Ten peripheral situations are shown to the user of the receiving terminal 20.
  • the user of the receiving side terminal 20 can infer the current state by looking at the state immediately before the other party while the communication is interrupted. Therefore, when sending the pre-correspondence data D10, the transmission side terminal 10 uses the camera to capture the surrounding situation and transmit it as image data.
  • the operation generation unit 14 generates an operation sharing command based on a user input operation.
  • the user input operation includes, for example, writing a handwritten stroke on the material, scrolling, enlarging / reducing, inputting characters, pressing a button, and the like.
  • the operations that can be thinned out are continuous operations such as writing, scrolling, and enlarging / reducing handwritten strokes, and intermediate operations can be thinned out, but the final result cannot be thinned out. .
  • the operation processing unit 15 performs the operation sharing command thinning according to the thinning method specified by the state determination unit 12. As described above, there are cases where the continuous operation can be thinned out, and in the continuous operations such as writing of handwritten strokes, scrolling, and enlargement / reduction, intermediate operations can be thinned out. Therefore, the operation processing unit 15 calculates a thinning rate from the data size of the operation sharing command and the desired arrival time according to the upper limit of the bandwidth specified by the state determination unit 12, and thins out the data that can be thinned out. Since the operation sharing target is shifted between the transmission side and the reception side due to the thinned result, the final result is transmitted together.
  • the transmission unit 16 transmits the operation sharing command received from the operation processing unit 15 and the advance correspondence data received from the advance correspondence data transmission unit to the reception-side terminal 20. At that time, the transmission unit 16 may perform redundant transmission in a plurality of sessions based on an instruction from the state determination unit 12.
  • the session here is a session established by a TCP / IP connection or a virtual session that can be individually transmitted for each UDP standby port.
  • the upper limit of the number of redundant sessions is the number (rounded down) obtained by dividing the predicted available bandwidth by the transmission bandwidth of the operation sharing command (transmission data amount / desired transmission time).
  • the transmission unit 16 can transmit data as redundantly as possible within the available bandwidth, and even if a delay occurs in any session, there is a possibility that the other session may be delayed with a low delay. Therefore, the arrival time of the operation sharing command can be shortened.
  • the receiving-side terminal 20 includes a receiving unit 21, a pre-corresponding data storage unit 22, an operation sharing command complementing unit 23, an operation reflecting unit 24, a pre-corresponding data storage device 26, and an operation sharing command storage device 25. .
  • the receiving unit 21 receives the advance correspondence data D10 and the operation sharing command from the transmission side terminal 10. Then, the receiving unit 21 passes the pre-corresponding data D10 to the pre-corresponding data storage unit 22, and passes the operation sharing command to the operation sharing command complementing unit 23.
  • the receiving unit 21 sends only the first operation sharing command that has arrived to the operation sharing command complementing unit 23, and later on. The operation sharing command that has been reached is discarded. However, if multiple operation sharing commands are sent in succession, even if it arrives first so that the order does not change, if there are operation sharing commands sent before that, The receiving unit 21 waits until the previously transmitted operation sharing command is received.
  • the advance correspondence data storage unit 22 transfers the received advance correspondence data D10 to the advance correspondence data storage device 26 in which the advance correspondence data is stored until the advance correspondence data storage device 26 receives and processes the operation sharing command. stand by.
  • the operation sharing command complementing unit 23 includes the prior correspondence data D10 stored in the prior correspondence data storage device 26, the latest operation sharing command sent from the reception unit 21, and the past operation stored in the operation sharing command storage device 25. Complement and generate the operation sharing command based on the sharing command.
  • the operation sharing command complementing unit 23 automatically generates an operation sharing command according to the estimated movement of the human operation by using the transmission-side terminal status D20 of the prior correspondence data D10 and the countermeasure D30 at the time of deterioration. For example, when the operation is a handwritten stroke, the first writing starts according to the operation sharing command, but there may be no operation sharing command at the timing when the stroke should be drawn at the next point. In such a case, the operation sharing command complementing unit 23 is based on the direction of the handwriting in the operation sharing command stored in the operation sharing command storage device 25, the speed of the touch operation in the prior correspondence data D10, and the length D21.
  • a point with an appropriate distance is generated as complementary data, and the thickness of the line is determined based on the slope D22 of the transmitting terminal 10. Then, the operation sharing command complementing unit 23 transmits the complemented operation sharing command to the operation reflecting unit 24, and draws a line between the previous point and the next point with a specified thickness.
  • the operation sharing command complementing unit 23 corrects the operation sharing command supplemented when all stroke data is sent after the stroke is input. Further, when the communication status of the network is severely deteriorated and the continuous operation sharing command does not reach for a long time, the operation sharing command complementing unit 23 substitutes according to the substituting display means D33 being deteriorated and the substituting sound effect D34 being deteriorated. An operation sharing command as a means is generated. When the latest peripheral photo D23 is specified in the alternative display means D33 that is deteriorated, the peripheral photo taken by the camera of the transmission side terminal 10 is displayed in the background or the periphery of the area displaying the data. Operation share command for generating.
  • the operation reflecting unit 24 executes operation sharing based on the operation sharing command sent from the operation sharing command complementing unit 23. Then, the operation sharing command is stored in the operation sharing command storage device 25 to prepare for the next complement.
  • FIG. 3 is a flowchart showing the operation of the transmission side terminal.
  • the NW state prediction unit 11 performs NW state prediction. Specifically, the NW state prediction unit 11 monitors the transmission unit 16 and detects a change in the network state.
  • the network state is a change state of the available bandwidth and the delay, and the NW state prediction unit 11 totals the transfer time of data being transmitted / received. Then, the NW state prediction unit 11 acquires the transmission start time and the transmission end time when transferring the data measured by the transmission unit 16 for prediction, and the reception start time and the reception end time measured by the reception unit 21. To get.
  • the current available bandwidth is the value obtained by dividing the amount of transmitted data by the difference between the transmission start time and the reception end time.
  • the current delay is a value obtained by calculating the difference between the transmission start time and the reception start time.
  • the NW state prediction unit 11 aggregates these values for a certain period of time, and predicts that the fluctuation of the available bandwidth will continue to increase in the future when the variation of the available bandwidth tends to increase. Further, when the delay variation tends to increase, the NW state prediction unit 11 predicts that the delay variation will increase further in the future.
  • step S11 the state determination unit 12 performs NW delay and band determination.
  • the state determination unit 12 determines what processing should be performed in the future based on the network state predicted by the NW state prediction unit 11.
  • the state determination unit 12 predicts that the fluctuation of the available bandwidth or the fluctuation of the delay is larger than a predetermined value, and if there is a possibility that the assumed transmission time cannot be satisfied, the state determination unit 12 sends the advance correspondence data to the advance correspondence data transmission unit 13.
  • Request to send The advance correspondence data transmission unit 13 sends the advance correspondence data to the network at a stage where it is predicted that the state of the network may be deteriorated to a predetermined degree regardless of whether the operation sharing command is transmitted or received. Send in a time when is stable.
  • the pre-correspondence data transmission unit 13 transmits the pre-correspondence data D10.
  • the pre-correspondence data D10 includes a transmission-side terminal situation D20 and a countermeasure D30 at the time of deterioration (of the network).
  • the transmission side terminal situation D20 includes the situation inside and around the transmission side terminal 10 at the predicted time point.
  • the transmission side terminal situation D20 includes the speed of the touch operation, the length D21, the inclination D22 of the transmission side terminal, the latest peripheral photograph, and the like.
  • the countermeasure D30 at the time of degradation includes, as a countermeasure at the time of network degradation, a thinning method D31, a transmission redundancy D32, an alternative display means D33 being deteriorated, and an alternative sound effect D34 being deteriorated.
  • the speed and length D21 of the touch operation are information on the input device level such as a touch panel and a mouse before being converted into the operation sharing command.
  • the receiving terminal 20 can predict the user's writing to some extent and use it for storing the operation sharing command.
  • the slope D22 of the transmission side terminal is data obtained by accumulating how the terminal is held and whether it is fluctuating.
  • the inclination D22 of the transmission side terminal 10 for example, when there is no change in inclination, it is placed on the desk and used stably, but when the change in inclination is severe, it is estimated that it is held with one hand Is done. The estimated result is used to determine how much the blur width is set when the reception side terminal 20 complements the operation sharing command thinned out on the reception side.
  • the receiving side terminal 20 displays the handwritten text with a certain width and a thicker stroke as an estimated value when the variation in tilt is severe.
  • the receiving terminal 20 simply performs spline interpolation and draws a handwritten stroke.
  • the transmission side terminal 10 transmits and displays the latest peripheral photograph D23 to the reception side terminal 20 to display the transmission side terminal.
  • Ten peripheral situations are shown to the user of the receiving terminal 20. The user of the receiving side terminal 20 can infer the current state by looking at the state immediately before the other party while the communication is interrupted. Therefore, when sending the pre-correspondence data D10, the transmission side terminal 10 uses the camera to capture the surrounding situation and transmit it as image data.
  • the NW state prediction unit 11 repeats the NW state prediction until a user operation occurs. If it is predicted that network fluctuations are likely to become severe, the advance correspondence data transmission unit 13 transmits advance correspondence data many times only when there is little fluctuation in the network and the bandwidth can be widely used.
  • step S20 the user's operation is started on the transmission side terminal 10.
  • the operation generation unit 14 generates an operation sharing command in accordance with a user input operation.
  • the user's input operation includes, for example, writing a handwritten stroke on the material, scrolling, enlarging / reducing, inputting characters, pressing a button, and the like.
  • these input operations those that can be thinned out are continuous operations such as writing of handwritten strokes, scrolling, enlargement / reduction, etc., and intermediate operations can be thinned out, but the final result cannot be thinned out. .
  • step S21 the state determination unit 12 determines the NW band based on the prediction and actual measurement of the NW state prediction unit 11. If it is predicted or measured that the fluctuation of the available bandwidth exceeds the specified value, the control shifts to step S22. Otherwise, control proceeds to step S23.
  • step S22 the state determination unit 12 notifies the operation processing unit 15 to thin out the operation sharing command that can be thinned out according to the predicted upper limit of the available bandwidth.
  • step S23 the state determination unit 12 determines the NW delay.
  • the state determination unit 12 issues an instruction to send to the transmission unit 16 in a redundant session when the fluctuation in the available bandwidth is less than the specified value but the delay variation is predicted to exceed the specified value. Control is transferred to step S24.
  • step S24 the transmission unit 16 is set to transmit in a redundant session.
  • the upper limit of the number of redundant sessions is the number obtained by dividing the predicted available bandwidth by the operation shared command transmission bandwidth (transmission data amount / desired transmission time) (rounded down). This allows data to be transmitted as redundantly as possible within the available bandwidth, and even if there is a delay in one of the sessions, other sessions may be delayed with a low delay, so operation sharing Command arrival time can be shortened.
  • the operation processing unit 15 performs the operation sharing command thinning according to the thinning method designated by the state determination unit 12.
  • the operation processing unit 15 calculates the thinning rate from the data size of the operation sharing command and the desired arrival time according to the upper limit of the bandwidth specified by the state determination unit 12, and thins out the data that can be thinned out. Since the operation sharing target is shifted between the transmission side and the reception side due to the thinned-out result, the final result is transmitted together.
  • the transmission unit 16 transmits the operation sharing command received from the operation processing unit 15 and the advance correspondence data to the reception side terminal 20.
  • the transmission unit 16 may perform redundant transmission in a plurality of sessions based on an instruction from the state determination unit 12.
  • the session here is a session established by a TCP / IP connection or a virtual session that can be individually transmitted for each UDP standby port.
  • the upper limit of the number of redundant sessions is the number (rounded down) obtained by dividing the predicted available bandwidth by the transmission bandwidth of the operation sharing command (transmission data amount / desired transmission time). This allows data to be transmitted as redundantly as possible within the available bandwidth, and even if there is a delay in one of the sessions, other sessions may be delayed with a low delay, so operation sharing Command arrival time can be shortened.
  • FIG. 4 is a flowchart showing the operation of the receiving terminal.
  • step S30 the receiving unit 21 receives the advance correspondence data D10 transmitted from the transmission side terminal 10.
  • step S31 the receiving unit 21 passes the pre-corresponding data D10 to the pre-corresponding data storage unit 22 to store the data.
  • step S40 the receiving unit 21 receives the operation sharing command transmitted from the transmitting terminal 10.
  • the receiving unit 21 sends only the first operation sharing command that has arrived to the operation sharing command complementing unit 23, and later.
  • the operation sharing command that arrives late is discarded.
  • the receiving unit 21 does not change the order so that the operation sharing command transmitted before that is not received even if it arrives first. In some cases, it waits until a previously sent operation sharing command is received.
  • step S41 the operation sharing command is reflected.
  • the operation reflecting unit 24 executes operation sharing based on the operation sharing command sent from the operation sharing command complementing unit 23.
  • step S42 the operation sharing command complementing unit 23 stores the operation sharing command in the operation sharing command storage device 25, and prepares for the next complement.
  • step S43 the operation sharing command complementing unit 23 performs complementing using the prior correspondence data D10 and past commands. Specifically, the operation share command complementing unit 23 stores the advance correspondence data D10 stored in the advance correspondence data storage device 26, the latest operation share command sent from the reception unit 21, and the operation share command storage device 25. The operation sharing command is complemented and generated based on the past operation sharing command.
  • the process when the operation sharing command complementing unit 23 receives a plurality of operation sharing commands for continuous operation at a predetermined interval will be described. First, when receiving the operation sharing command, the operation sharing command complementing unit 23 transmits the operation sharing command to the operation reflecting unit 24 as it is. Next, the operation sharing command complementing unit 23 performs complementing when it is clear that the reception is not performed at the determined interval after being thinned, and when the operation sharing command does not reach the interval determined by the increase in delay. .
  • the operation sharing command complementing unit 23 automatically generates an operation sharing command according to the estimated movement of the human operation by using the transmission-side terminal status D20 of the prior correspondence data D10 and the countermeasure D30 at the time of deterioration. For example, when the operation is a handwritten stroke, the first writing starts according to the operation sharing command, but there may be no operation sharing command at the timing when the stroke should be drawn at the next point. In such a case, the operation sharing command complementing unit 23 is appropriate based on the direction of the handwriting in the operation sharing command stored in the operation sharing command storage device 25, the speed of the touch operation of the advance correspondence data, and the length D21. A distance point is generated as complementary data, the thickness of the line is determined based on the slope D22 of the transmission side terminal, transmitted to the operation reflecting unit 24, and the distance between the previous point and the next point is designated Draw with a line of thickness.
  • the operation sharing command complementing unit 23 corrects the operation sharing command supplemented when all stroke data is sent after the stroke is input.
  • the operation sharing command complementing unit 23 when the communication status of the network is severely deteriorated and the continuous operation sharing command does not reach for a long time, follows the alternative display means D33 being deteriorated and the alternative sound effect D34 being deteriorated. Then, an operation sharing command is generated as an alternative means.
  • the operation sharing command complementing unit 23 displays the data of the peripheral photograph taken by the camera of the transmitting terminal 10 Generates an operation sharing command to be displayed in or around the background.
  • step S44 the operation reflecting unit 24 reflects the complemented operation sharing command.
  • the operation reflecting unit 24 executes operation sharing based on the operation sharing command sent from the operation sharing command complementing unit 23. Then, if it is clear that the next operation sharing command has been thinned out and will not be transmitted at the determined interval, and if the operation sharing command does not arrive within the interval determined by the increase in delay, the steps are repeated. The operation of S43 is repeated.
  • the operation sharing command communication control system predicts how much network bandwidth fluctuation and delay fluctuation will occur based on the current flow rate change of the network, and the transmission side terminal 10 causes the network fluctuation to become severe. If it can be predicted, send proactive data before the condition gets worse. Further, when the operation sharing command is generated, the transmission side terminal 10 performs control such as performing redundant transmission in a plurality of sessions when the delay variation becomes severe, and thinning and sending when the bandwidth variation becomes severe. Do. Further, the receiving side terminal 20 complements the operation sharing command when the network state deteriorates according to the pre-correspondence data, or uses an alternative means included in the pre-correspondence data.
  • the operation sharing command communication control system of the present embodiment it is possible to minimize the quality degradation of communication using the operation sharing command for shared data due to network fluctuations. Therefore, it is possible to minimize the influence of the deterioration of the communication status of the network, such as the delay of the operation result, on the receiving terminal user, so that the reliability of the operation sharing service and the operability can be improved. be able to.
  • FIG. 5 is a block diagram showing the configuration of the main part of the operation sharing command communication control system according to the present invention.
  • the operation sharing command communication control system according to the present invention includes, as main components, a transmission side terminal 10 and a reception side terminal 20 connected via a network 100, and an operation for transferring and complementing an operation sharing command in an operation sharing service.
  • the transmission-side terminal 10 includes a network state prediction unit 11 that predicts the state of the network 100, and fluctuations in the usable bandwidth of the network 100 due to deterioration of the communication state of the network 100.
  • the advance correspondence data which is data for dealing with the deterioration of the communication state of the network 100, is obtained before entering the network state.
  • Advance correspondence data to be transmitted to the receiving terminal 20 The communication unit 13, the operation processing unit 15 that thins out the operation sharing command according to the available bandwidth, and the operation sharing command when the fluctuation of the available bandwidth is larger than a predetermined value.
  • the operation sharing command is transmitted to the receiving side terminal 20 in a redundant session, and the receiving side terminal 20 And an operation sharing command complementing unit 23 that generates a complemented operation sharing command based on the prior correspondence data and the operation sharing command transmitted immediately before.
  • a transmission-side terminal for example, the transmission-side terminal 10 and a reception-side terminal (for example, the reception-side terminal 20) connected via a network (for example, the network 100), and an operation sharing command in the operation sharing service
  • An operation sharing command communication control system that performs transfer and complementing, wherein a transmission-side terminal predicts a network state by a network state prediction unit (for example, NW state prediction unit 11) and a network communication state due to deterioration of a network communication state.
  • NW state prediction unit 11 for example, NW state prediction unit 11
  • the advance is data for dealing with the deterioration of the communication state of the network before entering the network state.
  • Proactive data transmitter that transmits corresponding data to the receiving terminal (for example, The data transmission unit 13), an operation processing unit (for example, the operation processing unit 15) that thins out the operation sharing command according to the available bandwidth, and the operation sharing when the fluctuation of the available bandwidth is larger than a predetermined value.
  • a transmission unit (for example, transmission unit 16) that transmits a command to the reception side terminal and transmits an operation sharing command to the reception side terminal in a redundant session when the fluctuation of the delay of the network is large.
  • Operation sharing for generating a complemented operation sharing command based on a pre-corresponding data storage device (for example, the pre-corresponding data storage device 26) that stores the corresponding data, and the pre-corresponding data and the operation sharing command transmitted immediately before
  • An operation sharing command communication control system including a command complementing unit (for example, an operation sharing command complementing unit 23).
  • the pre-corresponding data includes a peripheral photograph of the transmitting terminal, and the receiving terminal displays the peripheral photograph transmitted immediately before the communication status of the network is deteriorated. It may be configured. According to such an operation sharing command communication control system, the user of the receiving side terminal can infer the current state by looking at the state immediately before the other party while the communication is interrupted.
  • the operation sharing command communication control system transmits pre-corresponding data including the status of the transmitting terminal when the transmitting terminal sends a handwritten stroke as an operation sharing command.
  • it may be configured to generate an operation sharing command complemented based on the status of the transmitting terminal transmitted immediately before. According to such an operation sharing command communication control system, even when the fluctuation of the available bandwidth exceeds a predetermined value, the available bandwidth is insufficient and the transmission of the operation sharing command is delayed. This can be prevented in advance.
  • the operation shared data may be configured to be redundantly transmitted. According to such an operation sharing command communication control system, data is transmitted as redundantly as possible within the available bandwidth, and even if a delay occurs in one of the sessions, Since there is a possibility of delaying with a low delay, the arrival time of the operation sharing command can be shortened.
  • the prediction result of the change in the available bandwidth of the network and the change in the delay by the state prediction unit may be configured to be a value having a predetermined width. According to such an operation sharing command communication control system, it is possible to reduce communication quality degradation without requiring accurate prediction of fluctuations in available bandwidth and fluctuations in delay.
  • the present invention can be applied to an operation sharing service using a mobile terminal such as a smartphone or a PC connected to a network.

Abstract

A transmission side terminal (10) comprises: a network state predicting unit (11) that predicts the state of a network; a pre-compensation data transmitting unit (13) that transmits, to a reception side terminal, pre-compensation data, which is data to be used for compensating for a degradation of the communication condition of the network; an operation processing unit (15) that thins out operation-shared commands according to an available band; and a transmitting unit (16) that transmits operation-shared commands to the reception side terminal in a redundant session. The reception side terminal comprises: a pre-compensation data storage device (26) that stores the pre-compensation data; and an operation-shared command complementing unit (23) that generates complemented operation-shared commands on the basis of the pre-compensation data and the operation-shared commands transmitted latest.

Description

操作共有コマンド通信制御システムおよび操作共有コマンド通信制御方法Operation sharing command communication control system and operation sharing command communication control method
　本発明は、スマートフォンなどの携帯端末またはＰＣ（Ｐｅｒｓｏｎａｌ　Ｃｏｍｐｕｔｅｒ）などがインターネットや無線ネットワークなどの利用可能帯域や遅延が変動するネットワークで結ばれているコミュニケーションシステムに用いられる操作共有コマンド通信制御システムおよび操作共有コマンド通信制御方法に関する。 The present invention relates to an operation sharing command communication control system and an operation used for a communication system in which a portable terminal such as a smartphone or a PC (Personal Computer) is connected by a network in which usable bandwidth or delay varies such as the Internet or a wireless network. The present invention relates to a shared command communication control method.
　携帯端末またはＰＣ等の情報端末を用いて、利用可能帯域や遅延が変動するインターネットや無線ネットワーク等を介して相手とコミュニケーションを行うには、情報端末は、ネットワークの変動に応じてデータを加工し、ネットワークの特性に即した送り方をする必要がある。情報端末は、多くの場合、ネットワークの変動を計測し、計測したネットワークの変動を元にフィードバック型の制御することで、緩やかなネットワークの変動に対応することができる。しかし、情報端末は、ネットワークの変動が激しい場合には変動に追随することができず、品質の劣化が発生してしまう。 In order to communicate with the other party using the information terminal such as a portable terminal or a PC via the Internet or a wireless network whose usable bandwidth and delay vary, the information terminal processes data according to the variation of the network. Therefore, it is necessary to send in accordance with the characteristics of the network. In many cases, the information terminal can cope with a gradual network change by measuring a network change and performing feedback type control based on the measured network change. However, the information terminal cannot follow the fluctuation when the fluctuation of the network is severe, and the quality is deteriorated.
　特に、３Ｇ（３ｒｄ．　Ｇｅｎｅｒａｔｉｏｎ）やＬＴＥ（Ｌｏｎｇ　Ｔｅｒｍ　Ｅｖｏｌｕｔｉｏｎ）などに代表されるモバイルデータ通信においては、多数の情報端末が、基地局の電波帯域を共有し、かつ、電波強度の変動の影響を受ける。そのため、多数の情報端末が集中する都市部などでは、ネットワークの利用可能帯域の変動および遅延の変動が激しいので、情報端末は、フィードバック型の技術だけでこれらの変動に対処することは難しい。 In particular, in mobile data communications represented by 3G (3rd. Generation) and LTE (Long Term Evolution), many information terminals share the radio band of the base station and are affected by fluctuations in radio field strength. receive. For this reason, in an urban area where a large number of information terminals are concentrated, fluctuations in the available bandwidth of the network and fluctuations in delay are severe, and it is difficult for the information terminals to cope with these fluctuations using only feedback-type technology.
　そこで、特許文献１には、ネットワークの帯域の変動を予測して、制御するフィードフォワード型で制御する方法が開示されている。この方法は、移動する端末のネットワークの状態の変化を予測して、予測結果に応じて映像配信のビットレート制御を実施し、ネットワークの通信状況が劣化しそうになったときには、予めビットレートを下げるなどの対応をする方法である。これにより、端末の使用者は、常に安定した映像コミュニケーションをとることが可能である。 Therefore, Patent Document 1 discloses a feed-forward control method that predicts and controls network bandwidth fluctuations. This method predicts a change in the network state of a moving terminal, performs video distribution bit rate control according to the prediction result, and lowers the bit rate in advance when the network communication status is likely to deteriorate. It is a method of dealing with such as. Thereby, the user of the terminal can always take stable video communication.
　また、非特許文献１および非特許文献２に開示されているように、遠隔にいる使用者同士が、ネットワークに接続された携帯端末またはＰＣ等を用いて、操作共有コマンドを送信することで画面やアプリケーションを共有することができる操作共有サービスが利用されている。操作共有コマンドとは、操作を共有する相手に対する指令であり、タッチパネルやマウスなどの入力操作による書き込み等を示すデータである。 In addition, as disclosed in Non-Patent Document 1 and Non-Patent Document 2, screens can be obtained when remote users transmit operation sharing commands using a mobile terminal or a PC connected to the network. And operation sharing services that can share applications. The operation sharing command is a command for a partner to share an operation, and is data indicating writing by an input operation using a touch panel or a mouse.
特開２００６－１７３９７３号公報JP 2006-173983 A
　特許文献１等に記載された技術では、予測結果に基づいて映像のビットレートを変更する。例えば、映像音声の通信については、映像の解像度や音声品質を変えるビットレート制御をすることで品質を落とすことにより、逆にネットワークの遅延を抑制することが可能である。映像音声の場合は、一瞬で流れ去るものなのでビットレート制御で一時的に欠落しても後には影響しないからである。 In the technique described in Patent Document 1, etc., the bit rate of the video is changed based on the prediction result. For example, in the case of video / audio communication, it is possible to suppress network delay by reducing the quality by controlling the bit rate to change the video resolution and audio quality. This is because in the case of video and audio, since it flows away in an instant, even if it is temporarily lost due to bit rate control, it will not be affected later.
　しかし、非特許文献１および非特許文献２に示したような操作共有サービスに用いられる操作共有コマンドは、通信するデータの一つでも欠けると、送信側の端末と受信側の端末とでデータの不整合が発生してしまう。つまり、操作共有コマンドの場合は、欠落がその後のデータの不整合につながるので、特許文献１に記載されているビットレート制御を用いた品質のコントロールができない。よって、操作共有サービスにおいて、ネットワークの通信状況が劣化した場合、コミュニケーションの品質が劣化してしまう。 However, if the operation sharing command used in the operation sharing service as shown in Non-Patent Document 1 and Non-Patent Document 2 lacks even one piece of data to be communicated, data transmission between the transmitting terminal and the receiving terminal is performed. Inconsistency will occur. That is, in the case of the operation sharing command, the loss leads to the subsequent data inconsistency, and thus quality control using the bit rate control described in Patent Document 1 cannot be performed. Therefore, in the operation sharing service, when the communication state of the network is deteriorated, the communication quality is deteriorated.
　そこで、本発明は、操作共有サービスにおいてネットワークの通信状況が劣化した場合に、コミュニケーションの品質劣化を低減することができる操作共有コマンド通信制御システムおよび操作共有コマンド通信制御方法を提供することを目的とする。 Accordingly, an object of the present invention is to provide an operation sharing command communication control system and an operation sharing command communication control method capable of reducing deterioration in communication quality when the network communication status in the operation sharing service deteriorates. To do.
　ネットワークを介して接続された送信側端末および受信側端末を備え、操作共有サービスにおける操作共有コマンドの転送および補完を行う操作共有コマンド通信制御システムであって、前記送信側端末は、前記ネットワークの状態を予測するネットワーク状態予測部と、前記ネットワークの通信状況の劣化により、前記ネットワークの利用可能帯域の変動または遅延の変動が予め定められた値より大きいネットワーク状態になると予測された場合、当該ネットワーク状態になる前に、前記ネットワークの通信状況の劣化に対応するためのデータである事前対応データを前記受信側端末に送信する事前対応データ送信部と、前記利用可能帯域の変動が予め定められた値より大きい場合、当該利用可能帯域に応じて前記操作共有コマンドを間引きする操作加工部と、前記操作共有コマンドを前記受信側端末に送信し、前記ネットワークの遅延の変動が大きい場合、前記操作共有コマンドを前記受信側端末に冗長セッションで送信する送信部とを含み、前記受信側端末は、前記事前対応データを保存する事前対応データ記憶装置と、前記事前対応データと直前に送信された前記操作共有コマンドとに基づいて、補完された操作共有コマンドを生成する操作共有コマンド補完部とを含むことを特徴とする操作共有コマンド通信制御システム。 An operation sharing command communication control system comprising a transmission side terminal and a reception side terminal connected via a network, and transferring and complementing an operation sharing command in an operation sharing service, wherein the transmission side terminal A network state prediction unit for predicting the network state, and when the network state is predicted to be larger than a predetermined value due to a change in available network bandwidth or a change in delay due to deterioration of the communication state of the network, Before becoming, a pre-corresponding data transmitting unit that transmits pre-corresponding data, which is data for coping with deterioration of the communication status of the network, and a value for which the fluctuation of the usable bandwidth is determined in advance If it is larger, the operation sharing command may be interrupted according to the available bandwidth. And an operation processing unit that transmits the operation sharing command to the receiving side terminal, and a transmission unit that transmits the operation sharing command to the receiving side terminal in a redundant session when variation in delay of the network is large. The reception-side terminal generates a supplemented operation sharing command based on the pre-corresponding data storage device that stores the pre-corresponding data and the pre-corresponding data and the operation sharing command transmitted immediately before An operation sharing command communication control system, comprising:
　ネットワークを介して接続された送信側端末および受信側端末を用い、操作共有サービスにおける操作共有コマンドの転送および補完を行う操作共有コマンド通信制御方法であって、前記送信側端末は、前記ネットワークの状態を予測し、前記送信側端末は、前記ネットワークの通信状況の劣化により、前記ネットワークの利用可能帯域の変動または遅延の変動が予め定められた値より大きいネットワーク状態になると予測された場合、当該ネットワーク状態になる前に、前記ネットワークの通信状況の劣化に対応するためのデータである事前対応データを前記受信側端末に送信し、前記送信側端末は、前記利用可能帯域の変動が予め定められた値より大きい場合、当該利用可能帯域に応じて前記操作共有コマンドを間引きし、前記送信側端末は、前記操作共有コマンドを前記受信側端末に送信し、前記ネットワークの遅延の変動が大きい場合、前記操作共有コマンドを前記受信側端末に冗長セッションで送信し、前記受信側端末は、前記事前対応データを保存し、前記受信側端末は、前記事前対応データと直前に送信された前記操作共有コマンドとに基づいて、補完された操作共有コマンドを生成することを特徴とする操作共有コマンド通信制御方法。 An operation sharing command communication control method for transferring and complementing an operation sharing command in an operation sharing service using a transmission side terminal and a reception side terminal connected via a network, wherein the transmission side terminal is in a state of the network When the transmission side terminal is predicted that a change in available bandwidth or delay in the network will be larger than a predetermined value due to deterioration in the communication status of the network, Before entering the state, the advance correspondence data, which is data for coping with deterioration of the communication status of the network, is transmitted to the reception side terminal, and the transmission side terminal has predetermined fluctuations in the available bandwidth If larger than the value, the operation sharing command is thinned out according to the available bandwidth, and the transmission side At the end, the operation sharing command is transmitted to the receiving terminal, and when the delay variation of the network is large, the operation sharing command is transmitted to the receiving terminal in a redundant session, and the receiving terminal An operation sharing command that stores pre-correspondence data, and the receiving side terminal generates a complemented operation sharing command based on the pre-correspondence data and the operation sharing command transmitted immediately before Communication control method.
　本発明によれば、操作共有サービスにおいてネットワークの通信状況が劣化した場合に、コミュニケーションの品質劣化を低減することができる。 According to the present invention, when the communication status of the network deteriorates in the operation sharing service, communication quality degradation can be reduced.
本発明による操作共有コマンド通信制御システムの実施形態の構成を示す説明図である。It is explanatory drawing which shows the structure of embodiment of the operation share command communication control system by this invention. 事前対応データのデータ構造を示す説明図である。It is explanatory drawing which shows the data structure of prior correspondence data. 送信側端末の動作を示すフローチャートである。It is a flowchart which shows operation | movement of the transmission side terminal. 受信側端末の動作を示すフローチャートである。It is a flowchart which shows operation | movement of the receiving side terminal. 本発明による操作共有コマンド通信制御システムの主要部の構成を示すブロック図である。It is a block diagram which shows the structure of the principal part of the operation share command communication control system by this invention.
　次に、本発明による操作共有コマンド通信制御システムの実施形態を、図面を参照して詳細に説明する。 Next, an embodiment of the operation sharing command communication control system according to the present invention will be described in detail with reference to the drawings.
　図１は、本発明による操作共有コマンド通信制御システムの実施形態の構成を示す説明図である。図１に示すように、本実施形態の操作共有コマンド通信制御システムは、送信側端末１０、受信側端末２０、および状態が変動するネットワーク１００を備える。なお、送信側端末１０と受信側端末２０の間に、中継サーバが、ネットワーク１００内の装置と同等に介在していてもよいが、図１では説明を簡略化するために省かれている。 FIG. 1 is an explanatory diagram showing a configuration of an embodiment of an operation sharing command communication control system according to the present invention. As shown in FIG. 1, the operation sharing command communication control system of this embodiment includes a transmission side terminal 10, a reception side terminal 20, and a network 100 whose state varies. Note that a relay server may be interposed between the transmission side terminal 10 and the reception side terminal 20 in the same manner as devices in the network 100, but is omitted in FIG. 1 for the sake of simplicity.
　ネットワーク１００は、例えば、インターネット、無線ネットワーク等により構成され、多数のルータ、多数の回線等を含み、送信側端末１０と受信側端末２０とを接続する。後述するネットワーク状態の予測は、送信側端末１０と受信側端末２０との間の状態が予測される。 The network 100 includes, for example, the Internet, a wireless network, and includes a large number of routers, a large number of lines, and the like, and connects the transmission side terminal 10 and the reception side terminal 20. In the network state prediction described later, a state between the transmission side terminal 10 and the reception side terminal 20 is predicted.
　送信側端末１０は、ユーザにより操作され、受信側端末２０に操作共有コマンドを送信する。また、送信側端末１０および受信側端末２０が、それぞれの同等の機能を具備し、双方向で操作共有コマンドを送りあう構成としてもよい。 The transmission side terminal 10 is operated by the user and transmits an operation sharing command to the reception side terminal 20. Further, the transmission side terminal 10 and the reception side terminal 20 may be configured to have the same functions and to send operation sharing commands in both directions.
　受信側端末２０は、操作共有コマンドなどを受け取り、表示することで、受信者側のユーザに操作結果を通知する。 The receiving terminal 20 receives and displays an operation sharing command and the like, thereby notifying the user on the receiver side of the operation result.
　次に送信側端末１０の構成を説明する。送信側端末１０は、ＮＷ（Ｎｅｔｗｏｒｋ）状態予測部１１と、状態判定部１２と、事前対応データ送信部１３と、操作発生部１４と、操作加工部１５と、送信部１６とを含む。 Next, the configuration of the transmitting terminal 10 will be described. The transmission side terminal 10 includes an NW (Network) state prediction unit 11, a state determination unit 12, a prior correspondence data transmission unit 13, an operation generation unit 14, an operation processing unit 15, and a transmission unit 16.
　ＮＷ状態予測部１１は、送信部１６を監視し、ネットワーク状態の変化を察知する。ネットワーク状態とは、利用可能帯域と遅延の変動であり、ＮＷ状態予測部１１は、送受信されるデータの転送時間を集計する。ＮＷ状態予測部１１は、予測のために、送信部１６が計測したデータを転送するときの送信開始時間、送信終了時間を取得し、後述する受信側端末２０の受信部２１が計測した受信開始時間、受信終了時間を取得する。 The NW state prediction unit 11 monitors the transmission unit 16 and detects a change in the network state. The network state is a change in available bandwidth and delay, and the NW state prediction unit 11 totals the transfer time of data transmitted and received. The NW state prediction unit 11 acquires the transmission start time and the transmission end time when transferring the data measured by the transmission unit 16 for prediction, and the reception start measured by the reception unit 21 of the reception-side terminal 20 described later. Get time and reception end time.
　現在の利用可能帯域は、送信したデータ量を送信開始時間と受信終了時間の差で割った値とする。現在の遅延は、送信開始時間と受信開始時間の差を示す値とする。ＮＷ状態予測部１１は、これらの値を、一定時間集計し、利用可能帯域のばらつきが大きくなる傾向にある場合には、今後も利用可能帯域の変動がより大きくなると予測する。また、ＮＷ状態予測部１１は、遅延の変動が大きくなる傾向にある場合は、今後も遅延の変動がより大きくなると予測する。また、利用可能帯域および遅延の変動の予測結果は、所定の幅があってもよい。例えば、利用可能帯域の変動は「○○Ｍｂｐｓから××Ｍｂｐｓ」と表され、遅延の変動は、「△△ｍｓから□□Ｍｍｓの間」と表される。ネットワークの変動を正確に予測することは困難だからである。 * The current available bandwidth is the value obtained by dividing the amount of transmitted data by the difference between the transmission start time and the reception end time. The current delay is a value indicating the difference between the transmission start time and the reception start time. The NW state prediction unit 11 aggregates these values for a certain period of time, and predicts that the fluctuation of the available bandwidth will continue to increase in the future when the variation of the available bandwidth tends to increase. Further, when the delay variation tends to increase, the NW state prediction unit 11 predicts that the delay variation will increase further in the future. Moreover, the prediction result of fluctuations in the available bandwidth and delay may have a predetermined width. For example, the change in the available bandwidth is expressed as “XX Mbps to xx Mbps”, and the delay change is expressed as “between ΔΔ ms and □□ Mms”. This is because it is difficult to accurately predict network fluctuations.
　状態判定部１２は、ＮＷ状態予測部１１に予測されたネットワーク状態を元に、今後、どのような処理をするべきかを判定する。状態判定部１２は、利用可能帯域の変動または遅延の変動が一定値よりも大きくなると予測され、想定した伝達時間を満たせない可能性がある場合には、事前対応データ送信部１３に事前対応データを送信するよう依頼する。 The state determination unit 12 determines what processing should be performed in the future based on the network state predicted by the NW state prediction unit 11. When it is predicted that the fluctuation of the available bandwidth or the fluctuation of the delay becomes larger than a certain value and the state determination unit 12 may not be able to satisfy the assumed transmission time, the state determination unit 12 sends the advance correspondence data to the advance correspondence data transmission unit 13. Request to send
　そして、状態判定部１２は、操作共有コマンドが送信されるときの動作を決定する。利用可能帯域の変動が予め定めた規定値よりも上まわると予測された場合には、間引き可能な操作共有コマンドを予測される利用可能帯域の上限に応じて間引くように、操作加工部１５に通知する。この処理によって、利用可能帯域が不足して操作共有のコマンドの伝達が遅延することをあらかじめ防止することできる。 And the state determination part 12 determines operation | movement when an operation share command is transmitted. When it is predicted that the fluctuation of the available bandwidth exceeds the predetermined value, the operation processing unit 15 is configured to thin out the operation sharing command that can be thinned out according to the predicted upper limit of the available bandwidth. Notice. By this processing, it is possible to prevent in advance that the available bandwidth is insufficient and the transmission of the operation sharing command is delayed.
　また、状態判定部１２は、利用可能帯域の変動が規定値を下回るが、遅延変動が規定値を上回ると予測された場合には、送信部１６に対して冗長セッションで送付するように指示を出す。冗長セッションの数は予測された利用可能帯域を操作共有コマンドの送信帯域（送信データ量÷送信希望時間）で割った数（端数切り捨て）を上限とする。これによって、利用可能な帯域内で、可能な限り冗長にデータが送信されるようになり、いずれかのセッションで遅延が発生したとしても、そのほかのセッションでは低遅延で遅れる可能性があるので、操作共有コマンドの到達時間が短縮できる。 In addition, the state determination unit 12 instructs the transmission unit 16 to send in a redundant session when the fluctuation in the available bandwidth is less than the specified value but the delay variation is predicted to exceed the specified value. put out. The upper limit of the number of redundant sessions is the number obtained by dividing the predicted available bandwidth by the operation shared command transmission bandwidth (transmission data amount / desired transmission time) (rounded down). This ensures that data is transmitted as redundantly as possible within the available bandwidth, and even if any session experiences a delay, other sessions may be delayed with a low delay, The arrival time of the operation sharing command can be shortened.
　事前対応データ送信部１３は、事前対応データを、送信部１６を介して受信側端末２０に送信する。事前対応データ送信部１３は、後述する操作共有コマンドが送受信されるかどうかにかかわらず、ネットワークの状態が予め定めた所定の程度劣化する可能性があると予測された段階で、ネットワークが安定している時間帯に事前対応データを送信する。 The pre-correspondence data transmission unit 13 transmits the pre-correspondence data to the reception side terminal 20 via the transmission unit 16. The prior correspondence data transmission unit 13 stabilizes the network at a stage where it is predicted that the network state may be deteriorated to a predetermined degree regardless of whether or not an operation sharing command described later is transmitted / received. Send proactive data during the time of day.
　図２は、事前対応データのデータ構造を示す説明図である。事前対応データＤ１０は、送信側端末状況Ｄ２０と劣化時の対処法Ｄ３０とを備える。送信側端末状況Ｄ２０には、予測された時点での送信側端末１０の内部および周辺の状況が記録されている。具体的には、送信側端末状況Ｄ２０は、タッチ操作の速度、長さＤ２１、送信側端末の傾きＤ２２、直近の周辺写真Ｄ２３などを含む。劣化時の対処法Ｄ３０は、ネットワーク劣化時の対処法として、間引き方Ｄ３１と、送信の冗長度Ｄ３２と、劣化中の代替表示手段Ｄ３３と、劣化中の代替効果音Ｄ３４とを含む。 FIG. 2 is an explanatory diagram showing the data structure of the prior correspondence data. The prior correspondence data D10 includes a transmission-side terminal status D20 and a countermeasure D30 for degradation. The transmission side terminal status D20 records the internal and peripheral status of the transmission side terminal 10 at the predicted time point. Specifically, the transmission side terminal situation D20 includes the speed of the touch operation, the length D21, the inclination D22 of the transmission side terminal, the latest peripheral photograph D23, and the like. The countermeasure D30 at the time of degradation includes a thinning method D31, a transmission redundancy D32, an alternative display means D33 being degraded, and an alternative sound effect D34 being degraded as countermeasures at the time of network degradation.
　タッチ操作の速度、長さＤ２１は操作共有コマンドに変換する前の、タッチパネルやマウスなどの入力デバイスレベルの情報である。タッチ操作の速度、長さＤ２１が受信側端末２０に送付されることで、受信側端末２０が、ある程度、ユーザの書き癖を予測して、操作共有コマンドの保管に利用できるようになる。送信側端末１０の傾きＤ２２は、端末の持ち方や変動しているかを累積したデータである。送信側端末１０の傾きＤ２２を参照することで、例えば、傾きの変動がない場合は机上に置いて安定して使っているが、傾きの変動が激しい場合は、片手で持っていることが推定される。推定された結果は、受信側端末２０が、間引きされた操作共有コマンドを補完するときに、ブレ幅をどのくらいに設定するのかに利用される。 The speed and length D21 of the touch operation are information on the input device level such as a touch panel and a mouse before being converted into the operation sharing command. By sending the speed and length D21 of the touch operation to the receiving terminal 20, the receiving terminal 20 can predict the user's writing to some extent and use it for storing the operation sharing command. The slope D22 of the transmission side terminal 10 is data obtained by accumulating how the terminal is held and whether it is changing. By referring to the inclination D22 of the transmission side terminal 10, for example, when there is no change in inclination, it is placed on the desk and used stably, but when the change in inclination is severe, it is estimated that it is held with one hand Is done. The estimated result is used for how much the blur width is set when the receiving terminal 20 complements the thinned operation sharing command.
　例えば、受信側端末２０は、手書きのストロークを操作共有コマンドとして受信した場合、傾きの変動が激しいときには、幅を持たせて推定値としてある程度ストロークを太くして手書きを表示する。また、例えば、受信側端末２０は、傾きの変動が少ない場合には単純にスプライン補完して手書きストロークを描画する。また、ネットワークの通信状況が劣化し、例えばどうしても通信できない瞬断状況に陥ったときには、送信側端末１０は、直近の周辺写真Ｄ２３を、受信側端末２０に送信して表示させて、送信側端末１０の周辺状況を受信側端末２０の使用者に見せる。受信側端末２０の使用者は、通信が瞬断した間の相手の直前の様子を見て、現在の様子を推察することができる。そのため、送信側端末１０は、事前対応データＤ１０を送付するときには、カメラを使って周辺の状況を撮影して画像データとして送信する。 For example, when the receiving side terminal 20 receives a handwritten stroke as an operation sharing command, when the fluctuation of the tilt is severe, the receiving side terminal 20 displays the handwritten with a certain width and a thickened stroke as an estimated value. In addition, for example, the reception side terminal 20 simply draws a handwritten stroke by performing spline interpolation when there is little variation in inclination. In addition, when the network communication state deteriorates, for example, when an instantaneous disconnection state where communication cannot be performed falls, the transmission side terminal 10 transmits and displays the latest peripheral photograph D23 to the reception side terminal 20 to display the transmission side terminal. Ten peripheral situations are shown to the user of the receiving terminal 20. The user of the receiving side terminal 20 can infer the current state by looking at the state immediately before the other party while the communication is interrupted. Therefore, when sending the pre-correspondence data D10, the transmission side terminal 10 uses the camera to capture the surrounding situation and transmit it as image data.
　操作発生部１４は、ユーザの入力操作に基づいて操作共有コマンドを生成する。ユーザの入力操作とは、例えば、資料に対する手書きストロークの書き込み、スクロール、拡大縮小、文字入力、ボタンの押下などである。これら入力操作の例のうち、間引き可能な操作は、手書きストロークの書き込み、スクロール、拡大縮小などの連続操作であり、途中の操作は間引き可能であるが、最終結果は間引くことができない特性がある。 The operation generation unit 14 generates an operation sharing command based on a user input operation. The user input operation includes, for example, writing a handwritten stroke on the material, scrolling, enlarging / reducing, inputting characters, pressing a button, and the like. Among these input operations, the operations that can be thinned out are continuous operations such as writing, scrolling, and enlarging / reducing handwritten strokes, and intermediate operations can be thinned out, but the final result cannot be thinned out. .
　操作加工部１５は、状態判定部１２で指定された間引き方式にしたがって操作共有コマンドの間引きを実施する。上述したように、連続操作は間引きが可能な場合があり、手書きストロークの書き込み、スクロール、拡大縮小などの連続操作において、途中の操作は間引き可能である。よって、操作加工部１５は、状態判定部１２から指定された帯域の上限にしたがって、操作共有コマンドのデータサイズおよび到達希望時間から間引き率を算出し、間引けるデータは間引きを行う。なお、間引いた結果によって操作共有対象が送信側と受信側でずれるので、最終結果をまとめて送信する。 The operation processing unit 15 performs the operation sharing command thinning according to the thinning method specified by the state determination unit 12. As described above, there are cases where the continuous operation can be thinned out, and in the continuous operations such as writing of handwritten strokes, scrolling, and enlargement / reduction, intermediate operations can be thinned out. Therefore, the operation processing unit 15 calculates a thinning rate from the data size of the operation sharing command and the desired arrival time according to the upper limit of the bandwidth specified by the state determination unit 12, and thins out the data that can be thinned out. Since the operation sharing target is shifted between the transmission side and the reception side due to the thinned result, the final result is transmitted together.
　送信部１６は、操作加工部１５から受け取った操作共有コマンド、および、事前対応データ送信部から受け取った事前対応データを受信側端末２０に送信する。そのとき、送信部１６は、状態判定部１２からの指示に基づいて、複数のセッションで冗長送信する場合がある。ここでいうセッションは、ＴＣＰ／ＩＰのコネクションで確立したセッション、または、ＵＤＰの待ち受けポートごとに個別に送信できる仮想的なセッションである。冗長セッションの数は、予測された利用可能帯域を操作共有コマンドの送信帯域（送信データ量÷送信希望時間）で割った数（端数切り捨て）を上限とする。これによって、送信部１６は、利用可能な帯域内で、可能な限り冗長にデータを送信できるようなり、いずれかのセッションで遅延が発生したとしても、そのほかのセッションでは低遅延で遅れる可能性があるので、操作共有コマンドの到達時間が短縮できる。 The transmission unit 16 transmits the operation sharing command received from the operation processing unit 15 and the advance correspondence data received from the advance correspondence data transmission unit to the reception-side terminal 20. At that time, the transmission unit 16 may perform redundant transmission in a plurality of sessions based on an instruction from the state determination unit 12. The session here is a session established by a TCP / IP connection or a virtual session that can be individually transmitted for each UDP standby port. The upper limit of the number of redundant sessions is the number (rounded down) obtained by dividing the predicted available bandwidth by the transmission bandwidth of the operation sharing command (transmission data amount / desired transmission time). As a result, the transmission unit 16 can transmit data as redundantly as possible within the available bandwidth, and even if a delay occurs in any session, there is a possibility that the other session may be delayed with a low delay. Therefore, the arrival time of the operation sharing command can be shortened.
　次に、受信側端末２０の構成を説明する。受信側端末２０は、受信部２１と、事前対応データ保存部２２と、操作共有コマンド補完部２３と、操作反映部２４と、事前対応データ記憶装置２６と、操作共有コマンド記憶装置２５とを備える。 Next, the configuration of the receiving terminal 20 will be described. The receiving-side terminal 20 includes a receiving unit 21, a pre-corresponding data storage unit 22, an operation sharing command complementing unit 23, an operation reflecting unit 24, a pre-corresponding data storage device 26, and an operation sharing command storage device 25. .
　受信部２１は、送信側端末１０から事前対応データＤ１０および操作共有コマンドを受信する。そして、受信部２１は、事前対応データＤ１０を事前対応データ保存部２２に渡し、操作共有コマンドを操作共有コマンド補完部２３に渡す。受信部２１は、複数のセッションを経由して転送された操作共有コマンドを受信した場合、同一の操作共有コマンドについては最初に到達したものだけを操作共有コマンド補完部２３に送り、後から遅れて到達した操作共有コマンドは廃棄する。ただし、複数の操作共有コマンドが連続して送られてくる場合は、順序が入れ替わらないように、最初に到達した場合でも、それよりも前に送信された操作共有コマンドがある場合には、前に送信された操作共有コマンドが受信されるまで、受信部２１は、待機する。 The receiving unit 21 receives the advance correspondence data D10 and the operation sharing command from the transmission side terminal 10. Then, the receiving unit 21 passes the pre-corresponding data D10 to the pre-corresponding data storage unit 22, and passes the operation sharing command to the operation sharing command complementing unit 23. When receiving the operation sharing command transferred via a plurality of sessions, the receiving unit 21 sends only the first operation sharing command that has arrived to the operation sharing command complementing unit 23, and later on. The operation sharing command that has been reached is discarded. However, if multiple operation sharing commands are sent in succession, even if it arrives first so that the order does not change, if there are operation sharing commands sent before that, The receiving unit 21 waits until the previously transmitted operation sharing command is received.
　事前対応データ保存部２２は、受け取った事前対応データＤ１０を事前対応データが保管された事前対応データ記憶装置２６に転送し、事前対応データ記憶装置２６が、操作共有コマンドを受信して処理するまで待機する。 The advance correspondence data storage unit 22 transfers the received advance correspondence data D10 to the advance correspondence data storage device 26 in which the advance correspondence data is stored until the advance correspondence data storage device 26 receives and processes the operation sharing command. stand by.
　操作共有コマンド補完部２３は、事前対応データ記憶装置２６に記憶された事前対応データＤ１０、受信部２１から送られた最新の操作共有コマンド、および操作共有コマンド記憶装置２５に格納された過去の操作共有コマンドに基づいて操作共有コマンドを補完し、生成する。 The operation sharing command complementing unit 23 includes the prior correspondence data D10 stored in the prior correspondence data storage device 26, the latest operation sharing command sent from the reception unit 21, and the past operation stored in the operation sharing command storage device 25. Complement and generate the operation sharing command based on the sharing command.
　操作共有コマンド補完部２３が、連続操作の操作共有コマンドを複数、決められた間隔で受信した場合の処理を説明する。まず、最初に、操作共有コマンド補完部２３が、操作共有コマンドを受信したときには、そのまま操作共有コマンドを操作反映部２４に送信する。次に、操作共有コマンドが間引かれていて決められた間隔では受信しないことが明らかな場合、および、遅延の増加で決められ間隔内に操作共有コマンドが到達しない場合、補完を行う。 Processing performed when the operation sharing command complementing unit 23 receives a plurality of operation sharing commands for continuous operation at predetermined intervals will be described. First, when the operation sharing command complementing unit 23 receives the operation sharing command, the operation sharing command is transmitted to the operation reflecting unit 24 as it is. Next, when it is clear that the operation sharing command is thinned out and is not received at the determined interval, and when the operation sharing command does not reach the interval determined by the increase in the delay, complement is performed.
　操作共有コマンド補完部２３の補完の方法を説明する。操作共有コマンド補完部２３は、事前対応データＤ１０の送信側端末状況Ｄ２０および劣化時の対処法Ｄ３０を利用して、推定される人間の操作の動きにしたがって操作共有コマンドを自動生成する。例えば、操作が手書きのストロークである場合、最初の書きだしは操作共有コマンド通りに始まるが、次の点でストロークを描画するべきタイミングで操作共有コマンドがないことがある。そのような場合、操作共有コマンド補完部２３は、操作共有コマンド記憶装置２５に蓄積された操作共有コマンド内にある筆跡の方向、事前対応データＤ１０内のタッチ操作の速度、長さＤ２１に基づいて適切な距離の点を補完データとして生成し、送信側端末１０の傾きＤ２２に基づいて線の太さを決定する。そして、操作共有コマンド補完部２３は、補完された操作共有コマンドを操作反映部２４に送信して、前の点と次の点との間を指定された太さの線で描画する。 A method of complementing the operation sharing command complementing unit 23 will be described. The operation sharing command complementing unit 23 automatically generates an operation sharing command according to the estimated movement of the human operation by using the transmission-side terminal status D20 of the prior correspondence data D10 and the countermeasure D30 at the time of deterioration. For example, when the operation is a handwritten stroke, the first writing starts according to the operation sharing command, but there may be no operation sharing command at the timing when the stroke should be drawn at the next point. In such a case, the operation sharing command complementing unit 23 is based on the direction of the handwriting in the operation sharing command stored in the operation sharing command storage device 25, the speed of the touch operation in the prior correspondence data D10, and the length D21. A point with an appropriate distance is generated as complementary data, and the thickness of the line is determined based on the slope D22 of the transmitting terminal 10. Then, the operation sharing command complementing unit 23 transmits the complemented operation sharing command to the operation reflecting unit 24, and draws a line between the previous point and the next point with a specified thickness.
　操作共有コマンド補完部２３は、全ストロークデータがストローク入力後に送られたときに補完された操作共有コマンドを修正する。また、操作共有コマンド補完部２３は、ネットワークの通信状況の劣化が激しく、連続した操作共有コマンドが長期間到達しない場合、劣化中の代替表示手段Ｄ３３および劣化中の代替効果音Ｄ３４にしたがって、代替手段としての操作共有コマンドを生成する。劣化中の代替表示手段Ｄ３３において直近の周辺写真Ｄ２３が指定されている場合には、送信側端末１０のカメラで撮影した周辺写真を、データを表示している領域のバックグラウンドまたは周辺に表示するための操作共有コマンドを生成する。 The operation sharing command complementing unit 23 corrects the operation sharing command supplemented when all stroke data is sent after the stroke is input. Further, when the communication status of the network is severely deteriorated and the continuous operation sharing command does not reach for a long time, the operation sharing command complementing unit 23 substitutes according to the substituting display means D33 being deteriorated and the substituting sound effect D34 being deteriorated. An operation sharing command as a means is generated. When the latest peripheral photo D23 is specified in the alternative display means D33 that is deteriorated, the peripheral photo taken by the camera of the transmission side terminal 10 is displayed in the background or the periphery of the area displaying the data. Operation share command for generating.
　操作反映部２４は、操作共有コマンド補完部２３から送られてきた操作共有コマンドに基づいて、操作共有を実行する。そして、操作共有コマンドを操作共有コマンド記憶装置２５に格納し、次の補完に備える。 The operation reflecting unit 24 executes operation sharing based on the operation sharing command sent from the operation sharing command complementing unit 23. Then, the operation sharing command is stored in the operation sharing command storage device 25 to prepare for the next complement.
　次に、本実施形態の操作共有コマンド通信制御システムの動作を詳細に説明する。図３は、送信側端末の動作を示すフローチャートである。 Next, the operation of the operation sharing command communication control system of this embodiment will be described in detail. FIG. 3 is a flowchart showing the operation of the transmission side terminal.
　まず、ステップＳ１０において、ＮＷ状態予測部１１は、ＮＷ状態予測を実施する。具体的には、ＮＷ状態予測部１１は、送信部１６を監視し、ネットワーク状態の変化を察知する。ネットワーク状態とは利用可能帯域と遅延の変動状況であり、ＮＷ状態予測部１１は、送受信しているデータの転送時間を集計している。そして、ＮＷ状態予測部１１は、予測のために、送信部１６が計測したデータを転送するときの送信開始時間、送信終了時間を取得し、受信部２１が計測した受信開始時間、受信終了時間を取得する。 First, in step S10, the NW state prediction unit 11 performs NW state prediction. Specifically, the NW state prediction unit 11 monitors the transmission unit 16 and detects a change in the network state. The network state is a change state of the available bandwidth and the delay, and the NW state prediction unit 11 totals the transfer time of data being transmitted / received. Then, the NW state prediction unit 11 acquires the transmission start time and the transmission end time when transferring the data measured by the transmission unit 16 for prediction, and the reception start time and the reception end time measured by the reception unit 21. To get.
　現在の利用可能帯域は、送信したデータ量を送信開始時間と受信終了時間の差で割った値とする。現在の遅延は、送信開始時間と受信開始時間の差を算出した値とする。ＮＷ状態予測部１１は、これらの値を、一定時間集計し、利用可能帯域のばらつきが大きくなる傾向にある場合には、今後も利用可能帯域の変動がより大きくなると予測する。また、ＮＷ状態予測部１１は、遅延の変動が大きくなる傾向にある場合は、今後も遅延の変動がより大きくなると予測する。 * The current available bandwidth is the value obtained by dividing the amount of transmitted data by the difference between the transmission start time and the reception end time. The current delay is a value obtained by calculating the difference between the transmission start time and the reception start time. The NW state prediction unit 11 aggregates these values for a certain period of time, and predicts that the fluctuation of the available bandwidth will continue to increase in the future when the variation of the available bandwidth tends to increase. Further, when the delay variation tends to increase, the NW state prediction unit 11 predicts that the delay variation will increase further in the future.
　次に、ステップＳ１１において、状態判定部１２は、ＮＷ遅延および帯域判定を実施する。状態判定部１２は、ＮＷ状態予測部１１で予測されたネットワーク状態を元に、今後、どのような処理をするべきかを判定する。状態判定部１２は、利用可能帯域の変動または遅延の変動が所定値よりも大きくなると予測され、想定した伝達時間を満たせない可能性がある場合には、事前対応データ送信部１３に事前対応データを送信するよう依頼する。事前対応データ送信部１３は、操作共有コマンドが送受信されるかどかにかかわらず、ネットワークの状態が予め定めた所定の程度劣化する可能性があると予測された段階で、事前対応データを、ネットワークが安定している時間帯に送付する。 Next, in step S11, the state determination unit 12 performs NW delay and band determination. The state determination unit 12 determines what processing should be performed in the future based on the network state predicted by the NW state prediction unit 11. The state determination unit 12 predicts that the fluctuation of the available bandwidth or the fluctuation of the delay is larger than a predetermined value, and if there is a possibility that the assumed transmission time cannot be satisfied, the state determination unit 12 sends the advance correspondence data to the advance correspondence data transmission unit 13. Request to send The advance correspondence data transmission unit 13 sends the advance correspondence data to the network at a stage where it is predicted that the state of the network may be deteriorated to a predetermined degree regardless of whether the operation sharing command is transmitted or received. Send in a time when is stable.
　そして、ステップＳ１２において、事前対応データ送信部１３は、事前対応データＤ１０を送信する。事前対応データＤ１０は、送信側端末状況Ｄ２０と（ネットワークの）劣化時の対処法Ｄ３０とを備える。送信側端末状況Ｄ２０には、予測された時点での送信側端末１０の内部および周辺の状況を含む。送信側端末状況Ｄ２０は、タッチ操作の速度、長さＤ２１、送信側端末の傾きＤ２２、直近の周辺写真などを含む。劣化時の対処法Ｄ３０は、ネットワーク劣化時の対処法として、間引き方Ｄ３１と、送信の冗長度Ｄ３２と、劣化中の代替表示手段Ｄ３３と、劣化中の代替効果音Ｄ３４とを含む。 And in step S12, the pre-correspondence data transmission unit 13 transmits the pre-correspondence data D10. The pre-correspondence data D10 includes a transmission-side terminal situation D20 and a countermeasure D30 at the time of deterioration (of the network). The transmission side terminal situation D20 includes the situation inside and around the transmission side terminal 10 at the predicted time point. The transmission side terminal situation D20 includes the speed of the touch operation, the length D21, the inclination D22 of the transmission side terminal, the latest peripheral photograph, and the like. The countermeasure D30 at the time of degradation includes, as a countermeasure at the time of network degradation, a thinning method D31, a transmission redundancy D32, an alternative display means D33 being deteriorated, and an alternative sound effect D34 being deteriorated.
　タッチ操作の速度、長さＤ２１は、操作共有コマンドに変換する前の、タッチパネルやマウスなどの入力デバイスレベルの情報である。タッチ操作の速度、長さＤ２１が受信側端末２０に送付されることで、受信側端末２０が、ある程度、ユーザの書き癖を予測して、操作共有コマンドの保管に利用できるようになる。送信側端末の傾きＤ２２は端末の持ち方や変動しているかを累積したデータである。送信側端末１０の傾きＤ２２を参照することで、例えば、傾きの変動がない場合は机上に置いて安定して使っているが、傾きの変動が激しい場合は、片手で持っていることが推定される。推定された結果は、受信側端末２０が、受信側で間引きされた操作共有コマンドを補完するときに、ブレ幅をどのくらいに設定するのかに利用される。 The speed and length D21 of the touch operation are information on the input device level such as a touch panel and a mouse before being converted into the operation sharing command. By sending the speed and length D21 of the touch operation to the receiving terminal 20, the receiving terminal 20 can predict the user's writing to some extent and use it for storing the operation sharing command. The slope D22 of the transmission side terminal is data obtained by accumulating how the terminal is held and whether it is fluctuating. By referring to the inclination D22 of the transmission side terminal 10, for example, when there is no change in inclination, it is placed on the desk and used stably, but when the change in inclination is severe, it is estimated that it is held with one hand Is done. The estimated result is used to determine how much the blur width is set when the reception side terminal 20 complements the operation sharing command thinned out on the reception side.
　例えば、受信側端末２０は、手書きのストロークを操作共有コマンドとして受信した場合、傾きの変動が激しいときには、幅を持たせて推定値としてある程度ストロールを太くして手書きを表示する。また、例えば、受信側端末２０は、傾きの変動が少ない場合には単純にスプライン補完して手書きストロールを描画する。また、ネットワークの通信状況が劣化し、例えばどうしても通信できない瞬断状況に陥ったときには、送信側端末１０は、直近の周辺写真Ｄ２３を、受信側端末２０に送信して表示させて、送信側端末１０の周辺状況を受信側端末２０の使用者に見せる。受信側端末２０の使用者は、通信が瞬断した間の相手の直前の様子を見て、現在の様子を推察することができる。そのため、送信側端末１０は、事前対応データＤ１０を送付するときには、カメラを使って周辺の状況を撮影して画像データとして送信する。 For example, when the handwritten stroke is received as an operation sharing command, the receiving side terminal 20 displays the handwritten text with a certain width and a thicker stroke as an estimated value when the variation in tilt is severe. In addition, for example, when there is little variation in inclination, the receiving terminal 20 simply performs spline interpolation and draws a handwritten stroke. In addition, when the network communication state deteriorates, for example, when an instantaneous disconnection state where communication cannot be performed falls, the transmission side terminal 10 transmits and displays the latest peripheral photograph D23 to the reception side terminal 20 to display the transmission side terminal. Ten peripheral situations are shown to the user of the receiving terminal 20. The user of the receiving side terminal 20 can infer the current state by looking at the state immediately before the other party while the communication is interrupted. Therefore, when sending the pre-correspondence data D10, the transmission side terminal 10 uses the camera to capture the surrounding situation and transmit it as image data.
　その後、ＮＷ状態予測部１１は、ユーザの操作が発生するまではＮＷ状態予測を繰り返す。ネットワークの変動が激しくなりそうであることが予測されたら、事前対応データ送信部１３は、ネットワークの変動が少なく、帯域が広く使えるときにだけ、事前対応データが何度も送信する。 Thereafter, the NW state prediction unit 11 repeats the NW state prediction until a user operation occurs. If it is predicted that network fluctuations are likely to become severe, the advance correspondence data transmission unit 13 transmits advance correspondence data many times only when there is little fluctuation in the network and the bandwidth can be widely used.
　次に、ステップＳ２０において、送信側端末１０に対しユーザの操作が開始される。操作発生部１４は、ユーザの入力操作に応じて操作共有コマンドを生成する。ユーザの入力操作は、例えば、資料に対する手書きストロークの書き込み、スクロール、拡大縮小、文字入力、ボタンの押下、などである。これら入力操作の例のうち、間引き可能なものは、手書きストロークの書き込み、スクロール、拡大縮小などの連続操作であり、途中の操作は間引き可能であるが、最終結果は間引くことができない特性がある。 Next, in step S20, the user's operation is started on the transmission side terminal 10. The operation generation unit 14 generates an operation sharing command in accordance with a user input operation. The user's input operation includes, for example, writing a handwritten stroke on the material, scrolling, enlarging / reducing, inputting characters, pressing a button, and the like. Among these input operations, those that can be thinned out are continuous operations such as writing of handwritten strokes, scrolling, enlargement / reduction, etc., and intermediate operations can be thinned out, but the final result cannot be thinned out. .
　そして、ステップＳ２１において、状態判定部１２は、ＮＷ状態予測部１１の予測と実測に基づいてＮＷ帯域の判定を行う。利用可能帯域の変動が規定値よりも上まわると予測または実測された場合には、ステップＳ２２に制御が移行する。そうでない場合は、ステップＳ２３に制御が移行する。 In step S21, the state determination unit 12 determines the NW band based on the prediction and actual measurement of the NW state prediction unit 11. If it is predicted or measured that the fluctuation of the available bandwidth exceeds the specified value, the control shifts to step S22. Otherwise, control proceeds to step S23.
　ステップＳ２２において、状態判定部１２は、間引き可能な操作共有コマンドを予測される利用可能帯域の上限に応じて間引くように、操作加工部１５に通知する。ステップＳ２２の処理によって、利用可能帯域が不足して操作共有のコマンドの伝達が遅延することをあらかじめ防止することができる。 In step S22, the state determination unit 12 notifies the operation processing unit 15 to thin out the operation sharing command that can be thinned out according to the predicted upper limit of the available bandwidth. By the processing in step S22, it is possible to prevent in advance that the available bandwidth is insufficient and the transmission of the operation sharing command is delayed.
　ステップＳ２３において、状態判定部１２は、ＮＷ遅延を判定する。状態判定部１２は、利用可能帯域の変動が規定値を下回るものの、遅延変動が規定値を上回ると予測された場合には、送信部１６に対して冗長セッションで送付するように指示を出し、ステップＳ２４に制御が移行する。 In step S23, the state determination unit 12 determines the NW delay. The state determination unit 12 issues an instruction to send to the transmission unit 16 in a redundant session when the fluctuation in the available bandwidth is less than the specified value but the delay variation is predicted to exceed the specified value. Control is transferred to step S24.
　ステップＳ２４では、送信部１６は、冗長セッションで送信するように設定する。冗長セッションの数は予測された利用可能帯域を操作共有コマンドの送信帯域（送信データ量÷送信希望時間）で割った数（端数切り捨て）を上限とする。これによって、利用可能な帯域内で、可能な限り冗長にデータを送信できるようなり、いずれかのセッションで遅延が発生したとしても、そのほかのセッションでは低遅延で遅れる可能性があるので、操作共有コマンドの到達時間が短縮できる。 In step S24, the transmission unit 16 is set to transmit in a redundant session. The upper limit of the number of redundant sessions is the number obtained by dividing the predicted available bandwidth by the operation shared command transmission bandwidth (transmission data amount / desired transmission time) (rounded down). This allows data to be transmitted as redundantly as possible within the available bandwidth, and even if there is a delay in one of the sessions, other sessions may be delayed with a low delay, so operation sharing Command arrival time can be shortened.
　そして、ステップＳ２５における送信の前に、操作加工部１５は、状態判定部１２が指定した間引き方式にしたがって操作共有コマンドの間引きを実施する。上述したように、連続操作は間引きが可能な場合があり、手書きストロークの書き込み、スクロール、拡大縮小などの連続操作であり、途中の操作は間引き可能である。よって、操作加工部１５は、状態判定部１２から指定された帯域の上限にしたがって、操作共有コマンドのデータサイズと到達希望時間から間引き率を算出し、間引けるデータは間引きを行う。なお、間引いた結果によって操作共有対象が送信側と受信側でずれるので、最終結果はまとめて送信する。 Then, before the transmission in step S25, the operation processing unit 15 performs the operation sharing command thinning according to the thinning method designated by the state determination unit 12. As described above, there are cases where the continuous operation can be thinned out, continuous operations such as writing of handwritten strokes, scrolling, enlargement / reduction, etc., and intermediate operations can be thinned out. Therefore, the operation processing unit 15 calculates the thinning rate from the data size of the operation sharing command and the desired arrival time according to the upper limit of the bandwidth specified by the state determination unit 12, and thins out the data that can be thinned out. Since the operation sharing target is shifted between the transmission side and the reception side due to the thinned-out result, the final result is transmitted together.
　ステップＳ２５において、送信部１６は、操作加工部１５から受け取った操作共有コマンド、および、事前対応データを受信側端末２０に送信する。そのとき、送信部１６は、状態判定部１２からの指示に基づいて、複数のセッションで冗長送信する場合がある。ここでいうセッションとは、ＴＣＰ／ＩＰのコネクションで確立したセッション、または、ＵＤＰの待ち受けポートごとに個別に送信できる仮想的なセッションである。冗長セッションの数は、予測された利用可能帯域を操作共有コマンドの送信帯域（送信データ量÷送信希望時間）で割った数（端数切り捨て）を上限とする。これによって、利用可能な帯域内で、可能な限り冗長にデータを送信できるようなり、いずれかのセッションで遅延が発生したとしても、そのほかのセッションでは低遅延で遅れる可能性があるので、操作共有コマンドの到達時間が短縮できる。 In step S25, the transmission unit 16 transmits the operation sharing command received from the operation processing unit 15 and the advance correspondence data to the reception side terminal 20. At that time, the transmission unit 16 may perform redundant transmission in a plurality of sessions based on an instruction from the state determination unit 12. The session here is a session established by a TCP / IP connection or a virtual session that can be individually transmitted for each UDP standby port. The upper limit of the number of redundant sessions is the number (rounded down) obtained by dividing the predicted available bandwidth by the transmission bandwidth of the operation sharing command (transmission data amount / desired transmission time). This allows data to be transmitted as redundantly as possible within the available bandwidth, and even if there is a delay in one of the sessions, other sessions may be delayed with a low delay, so operation sharing Command arrival time can be shortened.
　次に、受信側端末２０の動作を説明する。図４は、受信側端末の動作を示すフローチャートである。 Next, the operation of the receiving terminal 20 will be described. FIG. 4 is a flowchart showing the operation of the receiving terminal.
　まず、ステップＳ３０において、受信部２１は、送信側端末１０から送信された事前対応データＤ１０を受信する。そしてステップＳ３１において、受信部２１は、事前対応データＤ１０を事前対応データ保存部２２に渡して、データが格納される。 First, in step S30, the receiving unit 21 receives the advance correspondence data D10 transmitted from the transmission side terminal 10. In step S31, the receiving unit 21 passes the pre-corresponding data D10 to the pre-corresponding data storage unit 22 to store the data.
　そして、ステップＳ４０において、受信部２１は、送信側端末１０から送信された操作共有コマンドを受信する。受信部２１は、操作共有コマンドが複数のセッションを経由して転送されて受信する場合には、同一の操作共有コマンドについては最初に到達したものだけを操作共有コマンド補完部２３に送り、後から遅れて到達した操作共有コマンドは廃棄する。ただし、受信部２１は、複数の操作共有コマンドが連続して送られてくる場合は、順序が入れ替わらないように、最初に到達した場合でも、それよりも前に送信された操作共有コマンドがある場合には、前に送信された操作共有コマンドが受信されるまで、待機する。 In step S40, the receiving unit 21 receives the operation sharing command transmitted from the transmitting terminal 10. When the operation sharing command is transferred and received via a plurality of sessions, the receiving unit 21 sends only the first operation sharing command that has arrived to the operation sharing command complementing unit 23, and later. The operation sharing command that arrives late is discarded. However, when a plurality of operation sharing commands are continuously sent, the receiving unit 21 does not change the order so that the operation sharing command transmitted before that is not received even if it arrives first. In some cases, it waits until a previously sent operation sharing command is received.
　次に、ステップＳ４１において、操作共有コマンドの反映を行う。操作反映部２４では、操作共有コマンド補完部２３から送られてきた操作共有コマンドに基づいて、操作共有を実行する。 Next, in step S41, the operation sharing command is reflected. The operation reflecting unit 24 executes operation sharing based on the operation sharing command sent from the operation sharing command complementing unit 23.
　そして直ちに、ステップＳ４２において、操作共有コマンド補完部２３は、操作共有コマンドを操作共有コマンド記憶装置２５に格納し、次の補完に備える。 Immediately thereafter, in step S42, the operation sharing command complementing unit 23 stores the operation sharing command in the operation sharing command storage device 25, and prepares for the next complement.
　次に、ステップＳ４３において、操作共有コマンド補完部２３は、事前対応データＤ１０と過去のコマンドを使った補完を行う。具体的には、操作共有コマンド補完部２３は、事前対応データ記憶装置２６に記憶された事前対応データＤ１０、受信部２１から送られた最新の操作共有コマンド、および操作共有コマンド記憶装置２５に格納された過去の操作共有コマンドに基づいて、操作共有コマンドを補完、生成する。 Next, in step S43, the operation sharing command complementing unit 23 performs complementing using the prior correspondence data D10 and past commands. Specifically, the operation share command complementing unit 23 stores the advance correspondence data D10 stored in the advance correspondence data storage device 26, the latest operation share command sent from the reception unit 21, and the operation share command storage device 25. The operation sharing command is complemented and generated based on the past operation sharing command.
　操作共有コマンド補完部２３が、連続操作の操作共有コマンドを複数、決められた間隔で受信する場合の処理を説明する。まず、最初に、操作共有コマンド補完部２３は、操作共有コマンドを受信したときには、そのまま操作共有コマンドを操作反映部２４に送信する。次に、操作共有コマンド補完部２３は、間引かれて決められた間隔では受信しないことが明らかな場合、および、遅延の増加で決められ間隔内に操作共有コマンドが到達しない場合、補完を行う。 The process when the operation sharing command complementing unit 23 receives a plurality of operation sharing commands for continuous operation at a predetermined interval will be described. First, when receiving the operation sharing command, the operation sharing command complementing unit 23 transmits the operation sharing command to the operation reflecting unit 24 as it is. Next, the operation sharing command complementing unit 23 performs complementing when it is clear that the reception is not performed at the determined interval after being thinned, and when the operation sharing command does not reach the interval determined by the increase in delay. .
　操作共有コマンド補完部２３の補完の方法を説明する。操作共有コマンド補完部２３は、事前対応データＤ１０の送信側端末状況Ｄ２０および劣化時の対処法Ｄ３０を利用して、推定される人間の操作の動きにしたがって操作共有コマンドを自動生成する。例えば、操作が手書きのストロークである場合、最初の書きだしは操作共有コマンド通りに始まるが、次の点でストロークを描画するべきタイミングで操作共有コマンドがないことがある。そのような場合、操作共有コマンド補完部２３は、操作共有コマンド記憶装置２５に蓄積された操作共有コマンド内にある筆跡の方向、事前対応データのタッチ操作の速度、長さＤ２１に基づいて適切な距離の点を補完データとして生成し、送信側端末の傾きＤ２２に基づいて線の太さを決定し、操作反映部２４に送信して、前の点と次の点との間を指定された太さの線で描画する。 A method of complementing the operation sharing command complementing unit 23 will be described. The operation sharing command complementing unit 23 automatically generates an operation sharing command according to the estimated movement of the human operation by using the transmission-side terminal status D20 of the prior correspondence data D10 and the countermeasure D30 at the time of deterioration. For example, when the operation is a handwritten stroke, the first writing starts according to the operation sharing command, but there may be no operation sharing command at the timing when the stroke should be drawn at the next point. In such a case, the operation sharing command complementing unit 23 is appropriate based on the direction of the handwriting in the operation sharing command stored in the operation sharing command storage device 25, the speed of the touch operation of the advance correspondence data, and the length D21. A distance point is generated as complementary data, the thickness of the line is determined based on the slope D22 of the transmission side terminal, transmitted to the operation reflecting unit 24, and the distance between the previous point and the next point is designated Draw with a line of thickness.
　操作共有コマンド補完部２３は、全ストロークデータがストローク入力後に送られた時に補完された操作共有コマンドを修正する。また、操作共有コマンド補完部２３は、ネットワークの通信状況の劣化が激しく、連続した操作共有コマンドが長期間到達しない場合には、劣化中の代替表示手段Ｄ３３および劣化中の代替効果音Ｄ３４にしたがって、代替手段としての操作共有コマンドを生成する。劣化中の代替表示手段Ｄ３３において直近の周辺写真Ｄ２３が指定されている場合には、操作共有コマンド補完部２３は、送信側端末１０のカメラで撮影した周辺写真を、データを表示している領域のバックグラウンドまたは周辺に表示する操作共有コマンドを生成する。 The operation sharing command complementing unit 23 corrects the operation sharing command supplemented when all stroke data is sent after the stroke is input. The operation sharing command complementing unit 23, when the communication status of the network is severely deteriorated and the continuous operation sharing command does not reach for a long time, follows the alternative display means D33 being deteriorated and the alternative sound effect D34 being deteriorated. Then, an operation sharing command is generated as an alternative means. When the latest peripheral photograph D23 is designated in the alternative display means D33 that is deteriorated, the operation sharing command complementing unit 23 displays the data of the peripheral photograph taken by the camera of the transmitting terminal 10 Generates an operation sharing command to be displayed in or around the background.
　最後に、ステップＳ４４において、操作反映部２４は、補完された操作共有コマンドの反映を行う。操作反映部２４は、操作共有コマンド補完部２３から送られてきた操作共有コマンドに基づいて、操作共有を実行する。そして、次の操作共有コマンドが間引かれており、決められた間隔では送信されないことが明らかな場合、および、遅延の増加で決められ間隔内に操作共有コマンドが到達しない場合は、再度、ステップＳ４３の動作が繰り返される。 Finally, in step S44, the operation reflecting unit 24 reflects the complemented operation sharing command. The operation reflecting unit 24 executes operation sharing based on the operation sharing command sent from the operation sharing command complementing unit 23. Then, if it is clear that the next operation sharing command has been thinned out and will not be transmitted at the determined interval, and if the operation sharing command does not arrive within the interval determined by the increase in delay, the steps are repeated. The operation of S43 is repeated.
　本実施形態の操作共有コマンド通信制御システムは、ネットワークの現在の流量の変化を元にネットワークの帯域変動および遅延変動がどのくらい発生するのかを予測し、送信側端末１０が、ネットワークの変動が激しくなることが予測できた場合に、状態が悪くなる前に事前対応データを送付する。また、送信側端末１０が、操作共有コマンドが発生するときに、遅延変動が激しくなる場合には複数のセッションで冗長送信を行い、帯域変動が激しくなる場合には間引きし送付するなどの制御を行う。また、受信側端末２０は、事前対応データにしたがって、ネットワーク状態が劣化したときに操作共有コマンドを補完し、または事前対応データに内包された代替手段を利用する。 The operation sharing command communication control system according to the present embodiment predicts how much network bandwidth fluctuation and delay fluctuation will occur based on the current flow rate change of the network, and the transmission side terminal 10 causes the network fluctuation to become severe. If it can be predicted, send proactive data before the condition gets worse. Further, when the operation sharing command is generated, the transmission side terminal 10 performs control such as performing redundant transmission in a plurality of sessions when the delay variation becomes severe, and thinning and sending when the bandwidth variation becomes severe. Do. Further, the receiving side terminal 20 complements the operation sharing command when the network state deteriorates according to the pre-correspondence data, or uses an alternative means included in the pre-correspondence data.
　したがって、本実施形態の操作共有コマンド通信制御システムによれば、ネットワーク変動による共有データに対する操作共有コマンドを用いたコミュニケーションの品質劣化を最小化することができる。よって、受信端末の利用者が、操作結果の遅延等のネットワークの通信状況の劣化の影響を感じることを最小限にとどめることができるので、操作共有サービスの信頼性向上および操作性向上を達成することができる。 Therefore, according to the operation sharing command communication control system of the present embodiment, it is possible to minimize the quality degradation of communication using the operation sharing command for shared data due to network fluctuations. Therefore, it is possible to minimize the influence of the deterioration of the communication status of the network, such as the delay of the operation result, on the receiving terminal user, so that the reliability of the operation sharing service and the operability can be improved. be able to.
　図５は、本発明による操作共有コマンド通信制御システムの主要部の構成を示すブロック図である。本発明による操作共有コマンド通信制御システムは、主要な構成として、ネットワーク１００を介して接続された送信側端末１０および受信側端末２０を備え、操作共有サービスにおける操作共有コマンドの転送および補完を行う操作共有コマンド通信制御システムであって、前記送信側端末１０は、前記ネットワーク１００の状態を予測するネットワーク状態予測部１１と、前記ネットワーク１００の通信状況の劣化により、前記ネットワーク１００の利用可能帯域の変動または遅延の変動が予め定められた値より大きいネットワーク状態になると予測された場合、当該ネットワーク状態になる前に、前記ネットワーク１００の通信状況の劣化に対応するためのデータである事前対応データを前記受信側端末２０に送信する事前対応データ送信部１３と、利用可能帯域の変動が予め定められた値より大きい場合、当該利用可能帯域に応じて前記操作共有コマンドを間引きする操作加工部１５と、前記操作共有コマンドを前記受信側端末２０に送信し、前記ネットワーク１００の遅延の変動が大きい場合、前記操作共有コマンドを前記受信側端末２０に冗長セッションで送信する送信部１６とを含み、前記受信側端末２０は、前記事前対応データを保存する事前対応データ記憶装置２６と、前記事前対応データと直前に送信された前記操作共有コマンドとに基づいて、補完された操作共有コマンドを生成する操作共有コマンド補完部２３とを含む。 FIG. 5 is a block diagram showing the configuration of the main part of the operation sharing command communication control system according to the present invention. The operation sharing command communication control system according to the present invention includes, as main components, a transmission side terminal 10 and a reception side terminal 20 connected via a network 100, and an operation for transferring and complementing an operation sharing command in an operation sharing service. In the shared command communication control system, the transmission-side terminal 10 includes a network state prediction unit 11 that predicts the state of the network 100, and fluctuations in the usable bandwidth of the network 100 due to deterioration of the communication state of the network 100. Alternatively, when it is predicted that the network condition is greater than a predetermined value, the advance correspondence data, which is data for dealing with the deterioration of the communication state of the network 100, is obtained before entering the network state. Advance correspondence data to be transmitted to the receiving terminal 20 The communication unit 13, the operation processing unit 15 that thins out the operation sharing command according to the available bandwidth, and the operation sharing command when the fluctuation of the available bandwidth is larger than a predetermined value. And when the delay variation of the network 100 is large, the operation sharing command is transmitted to the receiving side terminal 20 in a redundant session, and the receiving side terminal 20 And an operation sharing command complementing unit 23 that generates a complemented operation sharing command based on the prior correspondence data and the operation sharing command transmitted immediately before.
　また、上記の実施形態には、以下の（１）～（５）に示すような操作共有コマンド通信制御システムも開示されている。 In addition, the above-described embodiment also discloses an operation sharing command communication control system as shown in the following (1) to (5).
（１）ネットワーク（例えば、ネットワーク１００）を介して接続された送信側端末（例えば、送信側端末１０）および受信側端末（例えば、受信側端末２０）を備え、操作共有サービスにおける操作共有コマンドの転送および補完を行う操作共有コマンド通信制御システムであって、送信側端末は、ネットワークの状態を予測するネットワーク状態予測部（例えば、ＮＷ状態予測部１１）と、ネットワークの通信状況の劣化により、ネットワークの利用可能帯域の変動または遅延の変動が予め定められた値より大きいネットワーク状態になると予測された場合、当該ネットワーク状態になる前に、ネットワークの通信状況の劣化に対応するためのデータである事前対応データを受信側端末に送信する事前対応データ送信部（例えば、事前対応データ送信部１３）と、利用可能帯域の変動が予め定められた値より大きい場合、当該利用可能帯域に応じて操作共有コマンドを間引きする操作加工部（例えば、操作加工部１５）と、操作共有コマンドを受信側端末に送信し、ネットワークの遅延の変動が大きい場合、操作共有コマンドを受信側端末に冗長セッションで送信する送信部（例えば、送信部１６）とを含み、受信側端末は、事前対応データを保存する事前対応データ記憶装置（例えば、事前対応データ記憶装置２６）と、事前対応データと直前に送信された操作共有コマンドとに基づいて、補完された操作共有コマンドを生成する操作共有コマンド補完部（例えば、操作共有コマンド補完部２３）とを含むことを特徴とする操作共有コマンド通信制御システム。 (1) A transmission-side terminal (for example, the transmission-side terminal 10) and a reception-side terminal (for example, the reception-side terminal 20) connected via a network (for example, the network 100), and an operation sharing command in the operation sharing service An operation sharing command communication control system that performs transfer and complementing, wherein a transmission-side terminal predicts a network state by a network state prediction unit (for example, NW state prediction unit 11) and a network communication state due to deterioration of a network communication state. When it is predicted that the fluctuation of the available bandwidth or the fluctuation of the delay will be larger than a predetermined value, the advance is data for dealing with the deterioration of the communication state of the network before entering the network state. Proactive data transmitter that transmits corresponding data to the receiving terminal (for example, The data transmission unit 13), an operation processing unit (for example, the operation processing unit 15) that thins out the operation sharing command according to the available bandwidth, and the operation sharing when the fluctuation of the available bandwidth is larger than a predetermined value. A transmission unit (for example, transmission unit 16) that transmits a command to the reception side terminal and transmits an operation sharing command to the reception side terminal in a redundant session when the fluctuation of the delay of the network is large. Operation sharing for generating a complemented operation sharing command based on a pre-corresponding data storage device (for example, the pre-corresponding data storage device 26) that stores the corresponding data, and the pre-corresponding data and the operation sharing command transmitted immediately before An operation sharing command communication control system including a command complementing unit (for example, an operation sharing command complementing unit 23).
（２）操作共有コマンド通信制御システムは、事前対応データが、送信側端末の周辺写真を含み、受信側端末が、ネットワークの通信状況が劣化したときに直前に送信された周辺写真を表示するように構成されていてもよい。このような操作共有コマンド通信制御システムによれば、受信側端末の使用者が、通信が瞬断した間の相手の直前の様子を見て、現在の様子を推察することができる。 (2) In the operation sharing command communication control system, the pre-corresponding data includes a peripheral photograph of the transmitting terminal, and the receiving terminal displays the peripheral photograph transmitted immediately before the communication status of the network is deteriorated. It may be configured. According to such an operation sharing command communication control system, the user of the receiving side terminal can infer the current state by looking at the state immediately before the other party while the communication is interrupted.
（３）操作共有コマンド通信制御システムは、送信側端末が、手書きのストロークを操作共有コマンドとして送付するときに、送信側端末の状況を含む事前対応データを送信し、受信側端末が、ネットワークの通信状況が劣化した場合、直前に送信された送信側端末の状況に基づいて補完された操作共有コマンドを生成するように構成されていてもよい。このような操作共有コマンド通信制御システムによれば、利用可能帯域の変動が予め定めた規定値よりも上まわる場合であっても、利用可能帯域が不足して操作共有のコマンドの伝達が遅延することをあらかじめ防止することできる。 (3) The operation sharing command communication control system transmits pre-corresponding data including the status of the transmitting terminal when the transmitting terminal sends a handwritten stroke as an operation sharing command. When the communication status deteriorates, it may be configured to generate an operation sharing command complemented based on the status of the transmitting terminal transmitted immediately before. According to such an operation sharing command communication control system, even when the fluctuation of the available bandwidth exceeds a predetermined value, the available bandwidth is insufficient and the transmission of the operation sharing command is delayed. This can be prevented in advance.
（４）操作共有コマンド通信制御システムは、送信側端末が、利用可能帯域の変動が予め定められた値より少なく、遅延の変動が予め定められた値より大きくなると予測された場合、複数のセッションで操作共有データを冗長送信するように構成されていてもよい。このような操作共有コマンド通信制御システムによれば、利用可能な帯域内で、可能な限り冗長にデータが送信されるようになり、いずれかのセッションで遅延が発生したとしても、そのほかのセッションでは低遅延で遅れる可能性があるので、操作共有コマンドの到達時間が短縮できる。 (4) In the operation sharing command communication control system, when the transmitting side terminal predicts that the fluctuation of the usable bandwidth is less than a predetermined value and the fluctuation of the delay is larger than the predetermined value, a plurality of sessions The operation shared data may be configured to be redundantly transmitted. According to such an operation sharing command communication control system, data is transmitted as redundantly as possible within the available bandwidth, and even if a delay occurs in one of the sessions, Since there is a possibility of delaying with a low delay, the arrival time of the operation sharing command can be shortened.
（５）状態予測部によるネットワークの利用可能帯域の変動および遅延の変動の予測結果は、所定の幅を有する値であるように構成されていてもよい。このような操作共有コマンド通信制御システムによれば、利用可能帯域の変動および遅延の変動の正確な予測を必要とせずにコミュニケーションの品質劣化を低減することができる。 (5) The prediction result of the change in the available bandwidth of the network and the change in the delay by the state prediction unit may be configured to be a value having a predetermined width. According to such an operation sharing command communication control system, it is possible to reduce communication quality degradation without requiring accurate prediction of fluctuations in available bandwidth and fluctuations in delay.
　この出願は、２０１２年６月７日に出願された日本出願特願２０１２－１２９７４３を基礎とする優先権を主張し、その開示の全てをここに取り込む。 This application claims priority based on Japanese Patent Application No. 2012-129743 filed on June 7, 2012, the entire disclosure of which is incorporated herein.
　以上、実施形態を参照して本願発明を説明したが、本願発明は上記実施形態に限定されるものではない。本願発明の構成や詳細には、本願発明のスコープ内で当業者が理解し得る様々な変更をすることができる。 The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
産業上の利用の可能性Industrial applicability
　本発明は、ネットワークに接続されたスマートフォン等の携帯端末、またはＰＣ等を用いた操作共有サービスに適用できる。 The present invention can be applied to an operation sharing service using a mobile terminal such as a smartphone or a PC connected to a network.
１０　送信側端末
１１　状態予測部
１１　ＮＷ状態予測部
１２　状態判定部
１３　事前対応データ送信部
１４　操作発生部
１５　操作加工部
１６　送信部
２０　受信側端末
２１　受信部
２２　事前対応データ保存部
２３　操作共有コマンド補完部
２４　操作反映部
２５　操作共有コマンド記憶装置
２６　事前対応データ記憶装置
１００　ネットワーク DESCRIPTION OF SYMBOLS 10 Transmission side terminal 11 State prediction part 11 NW state prediction part 12 State determination part 13 Advance correspondence data transmission part 14 Operation generation part 15 Operation processing part 16 Transmission part 20 Reception side terminal 21 Reception part 22 Advance correspondence data storage part 23 Operation sharing Command complement unit 24 Operation reflection unit 25 Operation shared command storage device 26 Advance correspondence data storage device 100 Network

Claims (6)

  1. 　ネットワークを介して接続された送信側端末および受信側端末を備え、操作共有サービスにおける操作共有コマンドの転送および補完を行う操作共有コマンド通信制御システムであって、
    　前記送信側端末は、
    　前記ネットワークの状態を予測するネットワーク状態予測部と、
    　前記ネットワークの通信状況の劣化により、前記ネットワークの利用可能帯域の変動または遅延の変動が予め定められた値より大きいネットワーク状態になると予測された場合、当該ネットワーク状態になる前に、前記ネットワークの通信状況の劣化に対応するためのデータである事前対応データを前記受信側端末に送信する事前対応データ送信部と、
    　前記利用可能帯域の変動が予め定められた値より大きい場合、当該利用可能帯域に応じて前記操作共有コマンドを間引きする操作加工部と、
    　前記操作共有コマンドを前記受信側端末に送信し、前記ネットワークの遅延の変動が大きい場合、前記操作共有コマンドを前記受信側端末に冗長セッションで送信する送信部と
    　を含み、
    　前記受信側端末は、
    　前記事前対応データを保存する事前対応データ記憶装置と、
    　前記事前対応データと直前に送信された前記操作共有コマンドとに基づいて、補完された操作共有コマンドを生成する操作共有コマンド補完部と
    　を含む
    　ことを特徴とする操作共有コマンド通信制御システム。     An operation sharing command communication control system comprising a transmission side terminal and a reception side terminal connected via a network, and transferring and complementing an operation sharing command in an operation sharing service,
    The transmitting terminal is
    A network state prediction unit for predicting the state of the network;
    If it is predicted that due to the deterioration of the communication status of the network, the fluctuation of the available bandwidth of the network or the fluctuation of the delay will be larger than a predetermined value, the communication of the network will be performed before the network status is reached. A pre-corresponding data transmitting unit that transmits pre-corresponding data that is data for coping with deterioration of the situation to the receiving terminal;
    When the fluctuation of the available bandwidth is larger than a predetermined value, an operation processing unit that thins out the operation sharing command according to the available bandwidth,
    A transmission unit that transmits the operation sharing command to the receiving terminal, and when the fluctuation of the delay of the network is large, a transmitting unit that transmits the operation sharing command to the receiving terminal in a redundant session;
    The receiving terminal is
    A proactive data storage device for storing the proactive data;
    An operation sharing command communication control system comprising: an operation sharing command complementing unit that generates a complemented operation sharing command based on the prior correspondence data and the operation sharing command transmitted immediately before.
  2. 　事前対応データは、送信側端末の周辺写真を含み、
    　受信側端末は、ネットワークの通信状況が劣化したときに直前に送信された前記周辺写真を表示する
    　請求項１記載の操作共有コマンド通信制御システム。     Pro-ready data includes peripheral photos of the sending terminal,
    The operation sharing command communication control system according to claim 1, wherein the receiving side terminal displays the peripheral photograph transmitted immediately before the communication state of the network deteriorates.
  3. 　送信側端末は、手書きのストロークを操作共有コマンドとして送付するときに、当該送信側端末の状況を含む事前対応データを送信し、
    　受信側端末は、ネットワークの通信状況が劣化した場合、直前に送信された前記送信側端末の状況に基づいて補完された操作共有コマンドを生成する
    　請求項１または請求項２記載の操作共有コマンド通信制御システム。     When sending the handwritten stroke as an operation sharing command, the sending terminal sends advance correspondence data including the situation of the sending terminal,
    The operation sharing command communication according to claim 1 or 2, wherein, when the communication state of the network is deteriorated, the receiving side terminal generates an operation sharing command complemented based on the state of the transmitting side terminal transmitted immediately before. Control system.
  4. 　送信側端末は、利用可能帯域の変動が予め定められた値より少なく、遅延の変動が予め定められた値より大きくなると予測された場合、複数のセッションで操作共有データを冗長送信する
    　請求項１から請求項３のうちのいずれか１項に記載の操作共有コマンド通信制御システム。     The transmission side terminal redundantly transmits the operation shared data in a plurality of sessions when it is predicted that the fluctuation of the available bandwidth is less than a predetermined value and the fluctuation of the delay is larger than the predetermined value. The operation sharing command communication control system according to any one of claims 1 to 3.
  5. 　状態予測部によるネットワークの利用可能帯域の変動および遅延の変動の予測結果は、所定の幅を有する値である
    　請求項１から請求項４のうちのいずれか１項に記載の操作共有コマンド通信制御システム。     The operation sharing command communication control according to any one of claims 1 to 4, wherein a prediction result of fluctuations in available network bandwidth and delay fluctuations by the state prediction unit is a value having a predetermined width. system.
  6. 　ネットワークを介して接続された送信側端末および受信側端末を用い、操作共有サービスにおける操作共有コマンドの転送および補完を行う操作共有コマンド通信制御方法であって、
    　前記送信側端末は、前記ネットワークの状態を予測し、
    　前記送信側端末は、前記ネットワークの通信状況の劣化により、前記ネットワークの利用可能帯域の変動または遅延の変動が予め定められた値より大きいネットワーク状態になると予測された場合、当該ネットワーク状態になる前に、前記ネットワークの通信状況の劣化に対応するためのデータである事前対応データを前記受信側端末に送信し、
    　前記送信側端末は、前記利用可能帯域の変動が予め定められた値より大きい場合、当該利用可能帯域に応じて前記操作共有コマンドを間引きし、
    　前記送信側端末は、前記操作共有コマンドを前記受信側端末に送信し、前記ネットワークの遅延の変動が大きい場合、前記操作共有コマンドを前記受信側端末に冗長セッションで送信し、
    　前記受信側端末は、前記事前対応データを保存し、
    　前記受信側端末は、前記事前対応データと直前に送信された前記操作共有コマンドとに基づいて、補完された操作共有コマンドを生成する
    　ことを特徴とする操作共有コマンド通信制御方法。     An operation sharing command communication control method for transferring and complementing an operation sharing command in an operation sharing service using a transmission side terminal and a reception side terminal connected via a network,
    The transmitting terminal predicts the state of the network;
    If it is predicted that due to deterioration of the communication status of the network, a change in usable bandwidth of the network or a change in delay will be larger than a predetermined value, the transmission side terminal may be in a network state. In addition, the advance correspondence data which is data for dealing with the deterioration of the communication status of the network is transmitted to the receiving terminal,
    When the variation of the available bandwidth is greater than a predetermined value, the transmitting terminal thins out the operation sharing command according to the available bandwidth,
    The transmitting terminal transmits the operation sharing command to the receiving terminal, and when the delay variation of the network is large, transmits the operation sharing command to the receiving terminal in a redundant session,
    The receiving terminal stores the advance correspondence data,
    The receiving side terminal generates a complemented operation sharing command based on the prior correspondence data and the operation sharing command transmitted immediately before. The operation sharing command communication control method, characterized in that:
PCT/JP2013/002592 2012-06-07 2013-04-17 Operation-shared command communication control system and operation-shared command communication control method WO2013183212A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2014519807A JPWO2013183212A1 (en) 2012-06-07 2013-04-17 Operation sharing command communication control system and operation sharing command communication control method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2012129743 2012-06-07
JP2012-129743 2012-06-07

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US14/611,784 Continuation US9778857B2 (en) 2012-08-07 2015-02-02 Recording device, access device, recording system, and recording method

Publications (1)

Publication Number Publication Date
WO2013183212A1 true WO2013183212A1 (en) 2013-12-12

Family

ID=49711622

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2013/002592 WO2013183212A1 (en) 2012-06-07 2013-04-17 Operation-shared command communication control system and operation-shared command communication control method

Country Status (2)

Country Link
JP (1) JPWO2013183212A1 (en)
WO (1) WO2013183212A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017154565A1 (en) * 2016-03-09 2017-09-14 コニカミノルタ株式会社 Joint operation system, edting control program, and editing control method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000057094A (en) * 1998-08-10 2000-02-25 Fujitsu Ltd Other terminal manipulator
JP2000174693A (en) * 1998-12-02 2000-06-23 Canon Inc Radio communication system
JP2004280695A (en) * 2003-03-18 2004-10-07 Sony Corp Data-sharing system, transmission side terminal device, reception side terminal device, program, and processing method of the transmission side terminal device

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000057094A (en) * 1998-08-10 2000-02-25 Fujitsu Ltd Other terminal manipulator
JP2000174693A (en) * 1998-12-02 2000-06-23 Canon Inc Radio communication system
JP2004280695A (en) * 2003-03-18 2004-10-07 Sony Corp Data-sharing system, transmission side terminal device, reception side terminal device, program, and processing method of the transmission side terminal device

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017154565A1 (en) * 2016-03-09 2017-09-14 コニカミノルタ株式会社 Joint operation system, edting control program, and editing control method

Also Published As

Publication number Publication date
JPWO2013183212A1 (en) 2016-01-28

Similar Documents

Publication Publication Date Title
CN107743698B (en) Method and apparatus for multi-path media delivery
US20200314382A1 (en) Video frame interpolation method, storage medium and terminal
US8723913B2 (en) Rate adaptation for video calling
EP3016395B1 (en) Video encoding device and video encoding method
EP2442543A1 (en) Portable terminal apparatus, method for controlling portable terminal apparatus, communication system, communication apparatus, and method for controlling communication apparatus
JP2008042850A (en) Repeater, repeating method, and repeating program
US9094527B2 (en) Seamlessly transferring a communication
US20110134909A1 (en) Data communication with compensation for packet loss
CN113423143A (en) Multipath data transmission method and device and electronic equipment
US20140028779A1 (en) Video transmitting apparatus and video transmitting method
US9058311B1 (en) User timeframe selection for delivery of media resources
CN102984784A (en) Transmitting data over multiple networks
CN104836672A (en) Method, device, system and terminal equipment for data transmission of multi-screen interaction
JP2008005394A (en) Communication terminal
JP5720787B2 (en) Content distribution system, cache server, and content distribution method
WO2013183212A1 (en) Operation-shared command communication control system and operation-shared command communication control method
EP2731016B1 (en) Content distribution system and content distribution method
KR20160029498A (en) Method and System for Providing Video Stream, and Relaying Apparatus
CN112422747A (en) Call method and device
US20080144623A1 (en) System and method for implementing a rtp-signaled terminal hand over
JP2005228228A (en) Client server system and its gui display method
JP6200870B2 (en) Data transfer control device, method and program
CN102984785A (en) Transmitting data over multiple networks
JP2011019150A (en) Communication apparatus, and line speed switching method in the same
JP4636510B2 (en) Terminal rate control method, terminal, system and program in communication / broadcasting cooperation system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13800553

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2014519807

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13800553

Country of ref document: EP

Kind code of ref document: A1