WO2014034059A1

WO2014034059A1 - Operation information control device, operation information control method, and operation information control program

Info

Publication number: WO2014034059A1
Application number: PCT/JP2013/004981
Authority: WO
Inventors: 亜紀子水藤
Original assignee: 日本電気株式会社
Priority date: 2012-08-28
Filing date: 2013-08-23
Publication date: 2014-03-06
Also published as: JP6183368B2; JPWO2014034059A1

Abstract

[Problem] To provide an operation information control device for reducing impediment in smooth communication among a plurality of devices that can communicate with each other and that share operation data by transfer, the impediment being caused by a delay in operation data transfer from a transfer source device. [Solution] An operation information control device according to the present invention includes: an operation reception means for receiving at time intervals from a terminal that detects a preparation action for a user operation on the terminal and the operation, preparation information indicating the preparation action and operation information indicating the operation; an operation storage means for storing the type of the preparation action and the type of the operation in association with each other; an operation estimation means for estimating, from the preparation information, the operation of the type associated with the type of the preparation information, the operation being to be performed after the preparation action; an expected time calculation means for calculating, from the preparation information and the reception time of the preparation information, an expected time when the operation information is received from the terminal and sent to a display device that receives the operation information, executes the operation in the operation information, and displays the result; and an operation output means for sending the operation information to the display device when the operation information is received by the expected time and for sending the operation information of the estimated operation to the display device at the expected time when the operation information is not received by the expected time.

Description

Operation information control apparatus, operation information control method, and operation information control program

The present invention relates to an operation information control device, an operation information control method, and an operation information control program.

In a system including two or more devices connected via a network, it is conceivable that a user of each device communicates by performing an operation on a screen shared by each device. Such operations include drawing characters and figures, moving, enlarging, and reducing objects on the screen. The screen shared by each device represents an area on the screen of each device that is kept in a state where the same image is drawn on each device. When a user of one of the devices performs an operation such as drawing of a character or a figure with a change in the state of the screen on such a screen, the screen of the other device performs the same operation. It becomes a state.

A system capable of such communication can be realized by each device sharing operation data representing operations such as drawing of characters and figures by users of the devices. The operation data is data including data representing a locus of a curve constituting a character or a graphic drawn by the user, for example. Such data includes, for example, data representing a coordinate sequence of points constituting a curve. Alternatively, the operation data may be data representing a movement instruction including information representing a movement destination for an object on the screen. The operation data may be a change instruction for enlarging or reducing the object around any fixed point and changing the size of the object.

Each device may change the display, such as drawing of characters and drawings, and enlargement or reduction of a figure, based on the operation data to be shared, with respect to an area for drawing a screen shared by each device. In addition, sharing of operation data can be realized by each device transmitting operation data acquired by the user to another device. An example of such a system is an online conference system.

The operation data can be configured to include information indicating the time from the start of the operation. For example, the operation data for drawing a curve may include information on the time at which the point is drawn in addition to the coordinates of the point on the drawn curve. The operation data for moving the object may include a plurality of sets of time information and coordinates representing the position of the object at that time. The operation data for changing the size of the object may include a plurality of sets of time information and the rate of change in size at that time. Further, the operation data may not be transferred collectively after a series of operations is completed, but may be transferred sequentially each time one individual element constituting the operation data is obtained. The individual elements constituting the operation data are, for example, a set of coordinates and time or a set of change rate of magnitude and time. In this case, it is possible to obtain a process of changing the screen due to the user's operation even with a device other than the device through which the user inputs the operation.

In order to realize smooth communication in a system that shares operation data of operations input to each device as described above, for example, there is no transfer delay of operation data between devices, and a user's The drawing by operation should just be performed in real time by each apparatus. However, there is a delay in the transmission of operation data due to the network connecting the devices.

Non-Patent Document 1 describes a remote drawing teaching system that maintains interactiveness even when there is a delay in the transfer of operation data. In the remote drawing teaching system, the operation is represented by cursor position information. The operation data is the position information. The terminal of the remote drawing teaching system of Non-Patent Document 1 sequentially receives the position information of the cursor from other terminals, and draws a figure at a point on the screen based on the received position information. When the terminal of Non-Patent Document 1 draws a figure based on position information indicating the position of the cursor of another terminal, the position of the cursor at a predetermined time ahead is predicted from the position information already received and predicted. Draw a figure at the position.

Patent Document 1 describes an information processing apparatus that predicts an operation performed by a finger when the finger contacts the touch panel before the finger contacts the touch panel. The information processing apparatus predicts an operation performed by the finger from the distance between the capacitive touch panel and the finger, the coordinates of the point closest to the finger on the touch panel, and the change with time. In the case where the instruction given by the predicted operation is an instruction of a process that can be performed in the background, the information processing apparatus of Patent Literature 1 performs the process indicated by the predicted operation before the instruction is actually input. Execute.

Patent Document 2 describes a communication game system in which the movement of an object is represented by a movement trajectory. The child device of the communication game system of Patent Literature 2 transmits a movement trajectory representing the trajectory of the object acquired from the input by the pointing device to the parent device. The parent device transmits drawing information for drawing the virtual game space including the received object to the child device.

Patent Document 3 discloses a game device in which each game machine sequentially transmits data calculated by each game machine to other game machines via a loop-shaped data transmission line to which a plurality of game machines are connected. Is described.

JP 2011-170834 A JP 2007-275472 A Japanese Patent Publication No. 03-039711

Operation data for drawing is transferred to each other, and when drawing operation data is delayed due to fluctuations in communication between multiple devices that communicate with each other by drawing each other, drawing on the destination device is also delayed. . And the smooth communication between the users who use those apparatuses is obstructed by the delay of drawing. When a user of each device performs communication by performing an operation on the screen while performing a voice call, if the transfer of operation data is delayed, the timing of the voice and the drawing are shifted. Particularly in this case, smooth communication is greatly hindered by this timing shift.

In the technique described in Non-Patent Document 1, the position of the cursor after a predetermined time is predicted based on the received information on the locus of the cursor. In other words, in the technique of Non-Patent Document 1, the position of the cursor that is the state of the operation after a predetermined time when the same operation is continued from a part of the user's operation represented by the locus of the cursor. is expected. In the technique of Non-Patent Document 1, if operation data does not reach due to a delay in transfer, the operation cannot be predicted. Therefore, with the technique of Non-Patent Document 1, smooth communication hindrance due to delay in transfer of operation data cannot be reduced.

The technique of Patent Document 1 is a technique for predicting an operation by a user and performing in advance only a process that does not appear on the screen display among processes performed by the information processing apparatus by the operation. In the technique of Patent Document 1, smooth communication hindrance due to a delay in the transfer of data representing an operation from another device due to a variation in the magnitude of communication delay cannot be reduced.

The techniques of Patent Document 2 and Patent Document 3 cannot alleviate the hindrance to smooth communication due to the delay in the transfer of data representing operations from other devices.

One of the objects of the present invention is an operation for reducing smooth communication hindrance due to a delay in the transfer of operation data from other devices among a plurality of devices that can communicate with each other and share the operation data by transfer. To provide an information control device.

The operation information control apparatus according to the present invention receives the preparation operation of the user's operation on the terminal and the operation information indicating the operation from the terminal that detects the operation at intervals of the preparation information indicating the preparation operation and the operation information. An operation receiving unit, an operation storage unit that associates and stores the type of the preparatory operation and the type of operation, and the preparatory information, the type of the operation related to the type of the preparatory information after the preparatory operation An operation estimating means for receiving the operation information, executing the operation of the operation information and displaying the result, and displaying a result of receiving the operation information from the terminal and transmitting the operation information. Scheduled time calculation means for calculating from the reception time of the preparation information, and when the operation information is received before the scheduled time, the operation information is transmitted to the display device, Wherein if the operation information is not received, including the operation information of the estimated said an operation estimation operation, and a operation output means for transmitting to said display device to the scheduled time.

The operation information control method according to the present invention receives a preparation operation for a user operation on a terminal and preparation information representing the preparation operation and operation information representing the operation from the terminal that detects the operation at intervals. , Storing the type of the preparatory operation and the type of operation in the operation storage means, estimating the type of the operation related to the type of the preparatory information after the preparatory operation from the preparatory information, The scheduled time for receiving and transmitting the operation information from the terminal to the display device that receives the information and executes the operation of the operation information and displays the result is calculated from the reception time of the preparation information and the preparation information. If the operation information is received by the scheduled time, the operation information is transmitted to the display device. If the operation information is not received by the scheduled time, the estimated operation is performed. The operation information of a certain estimation operation, and transmits to the display device to the scheduled time, including.

The operation information control program according to the present invention causes a computer to perform a time interval between preparation information indicating the preparation operation and operation information indicating the operation from the terminal that detects the operation of the user operation on the terminal and the operation. Receiving the operation receiving means, the operation storing means for storing the type of the preparatory action and the type of the operation in association with each other, and the type of the preparation information after the preparatory action from the preparation information. An operation estimation means for estimating an operation; and a scheduled time for receiving and transmitting the operation information from the terminal to a display device that receives the operation information and displays the result by executing the operation of the operation information. Preparation information and a scheduled time calculation means for calculating from the reception time of the preparation information, and when the operation information is received by the scheduled time, the operation information is displayed on the display device. And Shin, if the not received scheduled time the operation information so far, the operation information of the estimated said an operation estimation operation, an operation output unit for transmitting to said display device to the scheduled time, to operate as a.

Furthermore, the present invention can also be realized by a non-transient tangible recording medium readable by a computer in which the above-described operation information control program is stored.

According to the present invention, it is possible to reduce smooth communication hindrance due to a delay in the transfer of operation data from other devices among a plurality of devices that can communicate with each other and share the operation data by transfer. effective.

FIG. 1 is a block diagram illustrating a configuration of an information processing system 100 according to the first embodiment. FIG. 2 is a diagram illustrating an example of connection of each device included in the information processing system 100 according to the first embodiment. FIG. 3 is a flowchart showing the operation of the terminal 3 of the first, second, and fourth embodiments and the information processing apparatus 5B of the third embodiment when acquiring input device information. FIG. 4 is a flowchart showing the operation of the operation information control device 1 of the first and fourth embodiments and the information processing device 5B of the third embodiment when receiving the preparation information. FIG. 5 is a diagram illustrating the transmission and reception timing of each data when the communication delay between the terminal 3 and the operation information control apparatus 1 is increased. FIG. 6 is a block diagram illustrating a configuration of an information processing system 100A according to the second embodiment. FIG. 7 is a flowchart showing the operation of the operation information control apparatus 1A according to the second embodiment when receiving the preparation information. FIG. 8 is a block diagram illustrating the configuration of the information processing apparatus 5B according to the third embodiment. FIG. 9 is a block diagram illustrating connections of devices included in the information processing system 100B according to the third embodiment. FIG. 10 is a block diagram illustrating a configuration of an information processing system 100C according to the fourth embodiment. FIG. 11 is a block diagram illustrating connections of devices included in the information processing system 100C according to the fourth embodiment. FIG. 12 is a diagram illustrating a configuration example based on the first embodiment. FIG. 13 is a block diagram illustrating the configuration of the operation information control apparatus 1 according to the fifth embodiment. FIG. 14 is a diagram illustrating a configuration of a computer 1000 for realizing each device.

Next, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of the information processing system 100 according to the first embodiment of this invention.

Referring to FIG. 1, the information processing system 100 of the present embodiment includes a terminal 3 and an information processing device 5. The information processing device 5 includes an operation information control device 1 and a display device 2. The terminal 3 may be a device that includes all the components of the terminal 3 and the components of the information processing device 5 and further functions as the information processing device 5. Similarly, the information processing device 5 may be a device that includes all the components of the terminal 3 and the components of the information processing device 5 and further functions as the terminal 3.

The operation information control device 1 includes an operation receiving unit 10, an operation storage unit 11, an operation estimation unit 12, a scheduled time calculation unit 13, and an operation output unit 14. The operation information control apparatus 1 may further include a delay detection unit 15.

The display device 2 includes a screen generation unit 20 and a display unit 21.

The terminal 3 includes an input unit 30, a preparation operation detection unit 31, an operation detection unit 32, a screen generation unit 33, and a display unit 34.

FIG. 2 is a diagram illustrating an example of connection of each device included in the information processing system 100. Each device included in the information processing system 100 is connected as in the example illustrated in FIG. 2, for example.

In the present embodiment, operation data of an operation performed by the user of the terminal 3 is transmitted from the terminal 3 to the information processing apparatus 5. Each user of the terminal 3 and the information processing apparatus 5 shares operation data of an operation performed by the user of the terminal 3.

Referring to FIG. 2, the information processing apparatus 5 and the terminal 3 are connected via a network 4 which is a communication network. The network 4 is realized by, for example, the Internet or a dedicated communication network. The connection between each device and the network 4 may be wireless or wired.

The input unit 30 of the terminal 3 acquires data representing an operation performed on the terminal 3 by the user of the terminal 3. The input unit 30 is an input device such as a touch panel, a touch screen, a touch pad, a mouse, a key, a push button, and a dial.

In the description of the present embodiment, the operations performed by the user on the terminal 3 include screen display such as generation of objects on the screen such as drawing of characters and graphics, movement, enlargement, and reduction of objects. It is an operation to change. Operations that the user performs on the terminal 3 may be operations that change the display of the screen other than these operations. The user's operation may be an operation in which the user inputs some instruction to the terminal 3. Hereinafter, these operations are also referred to as actual operations.

The input unit 30 further acquires data representing a preparation operation that is an operation performed by the user before the actual operation. The preparation operation is, for example, an operation of bringing a finger close to the touch panel so that the user can input characters and figures using the touch panel. In the following description, the user's operation on the touch panel is described as being performed with a finger, but the operation on the touch panel may be performed with a tool such as a stylus pen instead of the finger. If the input unit 30 is a capacitive touch panel, for example, when a finger is brought close to the input unit 30, the input unit 30 determines the position on the touch panel closest to the finger and the distance between the finger and the touch panel at that position. It can be measured. Therefore, in this case, even when the finger is not in contact with the touch panel that is the input unit 30, the input unit 30 can acquire data representing the preparatory operation. For example, when the user inputs a figure using the mouse, the preparation operation is an actual operation on the terminal 3 such as an operation of stopping the mouse pointer at the drawing start position on the screen before starting drawing. Good. The preparation operation may be an operation or an actual operation other than the operation and the actual operation described above. Hereinafter, the data representing the preparatory operation acquired by the input unit 30 and the data representing the operation are collectively referred to as input device data or input device information.

In the following, data representing an operation is expressed as operation data or operation information. Further, the operation data representing the actual operation actually performed by the user of the terminal 3 is denoted as actual operation data. In addition, data representing the preparatory operation is expressed as predictive data or preparatory information. The predictor data only needs to include information indicating the type of the preparation operation and the content of the preparation operation. The actual operation data only needs to include information indicating the type of operation and the content of the operation.

The preparation operation detection unit 31 detects predictive data representing preparation operation from the data acquired by the input unit 30. The preparation operation detection unit 31 transmits the detected sign data to the operation reception unit 10 of the operation information control apparatus 1. The preparation operation detection unit 31 may further measure a communication delay between the terminal 3 and the information processing device 5. In that case, the preparatory movement detection unit 31 may transmit the measured delay time to the delay detection unit 15 via, for example, the operation reception unit 10.

The operation detection unit 32 detects actual operation data representing an actual operation from the data acquired by the input unit 30. The operation detection unit 32 transmits the detected actual operation data to the screen generation unit 33 and the operation reception unit 10 of the operation information control device 1.

The screen generator 33 receives actual operation data. The screen generation unit 33 generates a screen when an operation represented by the received actual operation data is performed. Then, the screen generation unit 33 displays the generated screen on the display unit 34.

The display unit 34 displays the screen generated by the screen generation unit 33. The display unit 34 is, for example, a display device that includes a liquid crystal panel and other display devices.

The screen generation unit 33 and the display unit 34 may be the same as the screen generation unit 20 and the display unit 21, respectively. The terminal 3 may include the display device 2 instead of the screen generation unit 33 and the display unit 34.

The operation receiving unit 10 of the operation information control device 1 receives actual operation data and predictive data from the terminal 3 via the network 4.

The operation storage unit 11 stores a plurality of sets of types of preparation operations represented by the predictive data and types of operations related to the types of preparation operations. Specifically, the operation storage unit 11 only needs to store a plurality of sets of identifiers representing the types of preparation operations and identifiers representing the types of operations. The operation storage unit 11 may store the probability that the type of operation is performed following the type of preparation operation in association with the set of the type of preparation operation and the type of operation. For example, the designer of the information processing system 100 can store in the operation storage unit 11 in advance the types of preparation operations and the types of operations that are likely to be performed after the types of preparation operations. That's fine. The designer of the information processing system 100 may derive the relationship between a plurality of types of preparation operations and the types of operations and the aforementioned probabilities from the statistics of the preparation operations and operations performed after the preparation operations. Alternatively, for example, the operation estimation unit 12 aggregates the predictor data received from the terminal 3 and the actual operation data following the predictor data, and based on the result of the aggregation, information on statistics of the preparation operation and operations performed after the preparation operation May be generated. Then, based on the statistical information, the operation estimation unit 12 derives a set of types of preparatory actions, types of operations that are likely to be performed after that type of preparatory actions, and probabilities related to those sets. May be. Further, the operation estimation unit 12 may cause the operation storage unit 11 to store the derived set of the type of preparation operation and the type of operation and the probability associated with the set. A series of a plurality of types of preparation operations may be associated with one type of operation.

The operation estimation unit 12 estimates the operation performed by the user following the preparatory operation represented by the predictive data based on the received predictive data and the type of operation associated with the type of preparatory operation represented by the predictive data. The operation estimation unit 12 transmits operation information representing an estimated operation, which is an estimated operation, to the operation output unit 14. Hereinafter, the estimation operation is also referred to as a complement operation. As described above, the operation information is also expressed as operation data. The operation data representing the complementary operation is also described as complementary data. The operation estimation unit 12 may estimate a complementary operation based on two or more predictive data.

The scheduled time calculation unit 13 receives, based on the received predictive data and the reception time of the received predictive data, the actual operation data of the actual operation performed by the user following the preparation operation represented by the predictive data from the terminal 3 Is calculated. The scheduled time calculation unit 13 sets the calculated time to the scheduled time that is the time for transmitting the complementary data to the display device 2.

The delay detection unit 15 detects a delay time in communication between the terminal 3 and the operation information control device 1. The method for the delay detection unit 15 to detect the communication delay time may be any existing method. The scheduled time calculation unit 13 may calculate the scheduled time using the delay time detected by the delay detection unit 15. As described above, when the preparation operation detection unit 31 of the terminal 3 measures the delay time, the delay detection unit 15 may receive the delay time from the terminal 3 via, for example, the operation reception unit 10. Hereinafter, a case where the delay detection unit 15 detects a delay will be described.

The operation output unit 14 estimates the operation estimation unit 12 when the operation data of the actual operation performed by the user following the preparation operation represented by the received predictive data is not received from the terminal 3 by the calculated scheduled time. The operation data of the complementary operation is transmitted to the display device 2 at the calculated scheduled time. When the operation output unit 14 receives operation information of an actual operation performed by the user following the preparation operation represented by the received predictive data from the receiving terminal 3 by the calculated scheduled time, the operation output unit 14 displays the operation information. 2 to send.

The screen generation unit 20 of the display device 2 generates a screen when an operation represented by the received actual operation data or complementary data is performed, and displays the generated screen on the display unit 21.

The display unit 21 displays the screen generated by the screen generation unit 20. The display unit 21 is a display device including a liquid crystal panel and other display devices, for example.

Next, the operation of the terminal 3 of this embodiment will be described in detail with reference to the drawings.

FIG. 3 is a flowchart showing the operation of the terminal 3.

When the terminal 3 starts operating, the input unit 30 starts the user's preparation operation and acquisition of input device data representing the operation (step S10).

The preparation operation detection unit 31 detects the preparation operation by detecting data representing the preparation operation from the data acquired by the input unit 30 (step S11). When the preparatory movement is detected (step S11, Y), the preparatory movement detecting unit 31 transmits the detected preliminary movement prediction data to the operation receiving unit 10 of the operation information control apparatus 1 (step S12).

The preparation operation detection unit 31 may hold in advance conditions for input device data for determining whether or not the input device data is predictive data for each type of predictive data to be detected. Then, when the input device data matches any condition, the preparation operation detection unit 31 may use the input device data that matches the condition as predictive data.

For example, if the input unit 30 is a capacitive touch panel, it is conceivable that a finger approaches the touch panel as a preparation operation for a tap operation, a drag operation, or a flick operation that is a user operation on the terminal 3. In this case, it is further conceivable to determine which kind of operation is the preparatory operation for the approach of the finger to the touch panel based on the magnitude of the movement of the finger parallel to the touch panel. Further, as a preparatory operation for a pinch operation such as pinch-out or pinch-in, it is conceivable that a finger touches two places less than a predetermined distance on the touch panel or a finger touches two places above a predetermined distance on the touch panel. The preparatory operation may be other than these.

When the input unit 30 is a capacitive touch panel, time-series information of the position where the finger on the touch panel is in contact can be obtained. In addition, when a finger approaches the capacitive touch panel, the capacitance of the place where the finger approaches the touch panel changes. Further, the capacitance depends on the distance between the finger and the touch panel. Therefore, by using a capacitive touch panel, even if the finger is not in contact with the touch panel, information on the position of the finger can be obtained as long as the finger and the touch panel are close to each other.

In this case, the input device data is data in which, for example, a plurality of capacitances in a plurality of electrodes arranged in a matrix on the touch panel are continuous in time series. For example, the preparatory motion detection unit 31 sets the center or the center of gravity of the region where the capacitance has changed on the touch panel as the finger position. Then, the preparatory movement detection unit 31 estimates the finger movement from the calculated finger positions from a plurality of continuous capacitance data. The preparation operation detection unit 31 determines whether or not the estimated finger movement matches, for example, the above-described preparation operation example. The preparatory movement detection unit 31 may use the input sensor data representing the movement of the finger determined to match the preparatory movement as predictive data. The predictive data may be data representing a change in the position of the finger estimated from the input device data, not the input sensor data itself.

When the input unit 30 is a mouse, an operation that the user performs on the terminal 3 is an operation of drawing a character or a figure using the mouse. As a preparation operation for an operation of drawing a figure in a drawing area displayed on the screen, for example, it is conceivable to stop the mouse pointer in the drawing area for a predetermined time or more. In this case, the preparation operation detection unit 31 continuously acquires the data of the position of the mouse pointer from the input unit 30 as input device data. Then, the preparation operation detection unit 31 determines whether or not the mouse pointer has been stopped for a predetermined time or more from the time series data of the position of the mouse pointer. When it is determined that the mouse pointer is stopped for a predetermined time or more, the preparation operation detection unit 31 generates predictive data including information indicating that the mouse pointer is stopped and position information.

The preparatory movement detection unit 31 may add an identifier indicating the type of the preparatory movement to the predictor data and transmit it to the operation reception unit 10.

When the preparation operation is not detected (step S11, Y), the preparation operation detector 31 does not transmit the predictive data (step S11, N).

Next, the operation detection unit 32 detects the actual operation by determining whether or not the input device data is data representing the actual operation (step S13). If no actual operation is detected (N in step S13), the process returns to step S10.

The operation detection unit 32 may detect an actual operation by an existing arbitrary method that detects a user operation using an input device.

When an actual operation is detected (step S13, Y), the operation detection unit 32 determines whether or not the detected actual operation is an operation for instructing the end of the operation of the terminal 3 (step S14). When the detected actual operation is an operation for instructing the end of the operation of the terminal 3 (step S14, Y), the terminal 3 ends the operation.

When the detected actual operation is not an operation for instructing the end of the operation of the terminal 3 (step S14, N), the operation detection unit 32 generates operation data representing the actual operation and transmits the operation data to the screen generation unit 33. To do.

The screen generation unit 33 generates a screen when an operation represented by the received operation data is performed, and displays the generated screen on the display unit 34 (step S15).

The operation data can be, for example, data of a part representing an actual operation in the input device data. The operation data may be data representing a specific operation of the user determined from the input device data. The operation data may be data representing an instruction to the user terminal 3 by the specific operation. The data representing the specific operation of the user is, for example, data representing coordinates on the touch panel and a tap operation at the coordinates. In this case, for example, if the location tapped by the user is an area of the touch panel on which a button for erasing a graphic drawn on the screen is drawn, data representing an instruction to the terminal 3 is drawn. It is data representing an instruction to erase a figure. Or the data showing a user's specific operation | movement may be the time series data of the position where the finger is contacting the touch panel, for example. In this case, for example, if the terminal 3 is in the drawing mode when the input device data is acquired, the data representing the instruction to the terminal 3 is drawn by drawing a figure drawn on the touch panel by moving the finger while touching the touch panel. It is data representing an instruction.

Next, the operation detection unit 32 transmits the operation data to the operation receiving unit 10 of the operation information control device 1 (step S16). The operation detection unit 32 may add an identifier indicating the type of actual operation to the operation data and transmit the operation data to the operation reception unit 10. The order of step S15 and step S16 may be reversed.

The preparatory operation detection unit 31 may continuously detect the preparatory operation regardless of the operation of FIG. When the preparation operation is detected, the preparation operation detection unit 31 transmits predictive data of the preparation operation. For example, the preparatory movement detection unit 31 may transmit the preliminary data of the preparatory movement during every predetermined time.

The operation detection unit 32 may continuously detect the actual operation regardless of the operation of FIG. When the operation detection unit 32 detects an actual operation, the operation detection unit 32 transmits actual operation data that is operation data of the actual operation. For example, the operation detection unit 32 may transmit actual operation data during every predetermined time.

Next, the operation of the operation information control apparatus 1 of the present embodiment will be described in detail with reference to the drawings.

FIG. 4 is a flowchart showing the operation of the operation information control apparatus 1 when the predictive data is received. As described above, in this embodiment, the predictor data is also expressed as preparation information. For example, when the operation information control apparatus 1 receives actual operation data instead of predictive data after receiving other actual operation data, the operation information control apparatus 1 transmits the received actual operation data to the screen generation unit 20 regardless of the operation of FIG. Good.

Referring to FIG. 4, first, the operation receiving unit 10 receives predictive data (step S20).

The operation estimation unit 12 estimates the actual operation performed by the user following the preparatory operation represented by the predictive data from the predictive data received by the operation receiving unit 10 (step S21).

The operation estimation unit 12 reads from the operation storage unit 11 the type of operation associated with the type of preparation operation represented by the received predictive data. Then, the operation estimation unit 12 estimates an actual operation performed following the preliminary operation represented by the predictor data from the predictor data, the received predictor data, and the read operation type. And the operation estimation part 12 produces | generates the complementary data showing the estimated actual operation.

For example, when the predictor data includes data representing a change in the position of the finger in a state where the finger is approaching the touch panel, the operation estimation when the type of the preparatory operation of the predictor data is the approach of the finger to the touch panel The operation of the unit 12 is as follows. In this case, the operation estimation unit 12 first reads the type of actual operation associated with the approach of the finger to the touch panel, which is the type of preparation operation. If the type of the read actual operation is a finger touch on the touch panel, the operation estimation unit 12 then coordinates the position where the finger is estimated to touch the touch panel from the finger position data included in the predictive data. Is calculated. Then, the operation estimation unit 12 generates data representing an event in which the finger touches the calculated position as complementary data. Or the operation estimation part 12 may produce | generate the data showing the event which generate | occur | produces when a finger contacts the calculated position as complementary data. For example, when, for example, a button is displayed at the calculated position, and the contact of the finger at the calculated position means an instruction to press the button, the operation estimation unit 12 complements the data indicating the button press. What is necessary is just to produce | generate as data.

For example, the predictor data includes data representing two positions separated by a predetermined distance or more where the finger is in contact with the touch panel at the same time. The operation of the operation estimation unit 12 in the case of contact is as follows. In this case, the operation estimation unit 12 first reads out the type of actual operation associated with finger contact with two distant points, which is the type of preparation operation. The type of the actual operation associated with the finger contact with the two distant points is, for example, a pinch-in operation, which is an operation of bringing the finger close to the touch panel. In this case, the operation estimation unit 12 generates, as complementary data, data representing an event in which the finger that has touched each of the two positions included in the predictor data is dragged in the direction toward the other point. Or the operation estimation part 12 may produce | generate the data showing the event which generate | occur | produces when the finger | toe which contacted each of the position of 2 points | pieces included in the sign data is dragged in the direction of the other point as complementary data. For example, if both the finger contact points included in the predictive data are included in the area where the object is displayed and the above drag means an instruction to reduce the displayed object, the object is reduced. An event to be performed may be generated as complementary data.

For example, when the type of the preparation operation is a finger contact with two points and the content of the preparation operation is information indicating the position of the two points with which the finger is in contact, the operation estimation unit 12 is as follows. It may work. The operation estimation unit 12 first reads out the type of actual motion associated with finger contact with two points, which is the type of preparation motion. The type of actual motion associated with the contact of the finger with the two points is, for example, a pinch operation, which is an operation that moves the finger closer to or away from the touch panel. The operation estimation unit 12 calculates the distance between the two points in contact with the finger from information representing the position of the two points in contact with the finger, which is information on the content of the preparation operation. If the distance between two points in contact with the finger is equal to or greater than a predetermined distance, the operation estimation unit 12 estimates that the actual operation is a pinch-in operation, which is an operation of bringing the finger into contact with the touch panel. To do. In this case, for example, as described above, the operation estimation unit 12 generates, as complementary data, data representing an event in which a finger touching each of the two points included in the predictor data is dragged in a direction toward the other point. On the other hand, if the distance between the two points in contact with the finger is less than the predetermined distance, the operation estimation unit 12 performs an actual operation by a pinch-out operation, which is an operation of keeping the finger away from the touch panel. Presume that there is. In this case, for example, the operation estimation unit 12 generates, as complementary data, data representing an event in which a finger touching each of the two points included in the predictor data is dragged away from the other point.

For example, the operation of the operation estimation unit 12 in the case where the predictor data includes data indicating the position where the mouse pointer has stopped for a certain time or more and the type of the operation of preparing the predictor data is the stop of the mouse pointer is as follows. It is as follows. In this case, the operation estimation unit 12 first reads the type of actual operation associated with the stop of the mouse pointer, which is the type of preparation operation. When the type of the actual operation associated with the stop of the mouse pointer is a mouse button click, the operation estimation unit 12 displays an event in which the mouse button is clicked at the position of the mouse pointer included in the actual operation. Data to represent is generated as complementary data. When there are a plurality of mouse buttons, the operation estimation unit 12 may generate, for example, data representing an event in which a button that is clicked most frequently is clicked as complementary data.

Next, the scheduled time calculation unit 13 calculates the scheduled time at which the operation receiving unit 10 receives the operation data of the actual operation performed following the preparation operation represented by the predictor data. Then, the scheduled time calculation unit 13 sets the calculated time to the scheduled time that is the time for transmitting the complementary data (step S22).

The scheduled time calculation unit 13 estimates, for example, the time from when the user performs a preparatory operation represented by predictive data until the actual operation following the preparatory operation is performed. Then, the scheduled time calculation unit 13 calculates the scheduled time from the estimated time and the time when the operation receiving unit 10 receives the predictive data. If the communication delay between the terminal 3 and the operation information control device 1 is constant, the scheduled time calculation unit 13 can calculate the scheduled time by adding the estimated time to the reception time of the predictive data.

For example, when the predictive data is data indicating that a finger approaches the touch panel, the actual operation is a finger touch on the touch panel. The scheduled time calculation unit 13 calculates the time from the sign data acquisition to the touch of the finger on the touch panel based on the distance between the touch panel and the finger and the speed at which the finger approaches the touch panel. The predictive data only needs to include data representing the transition of the distance between the touch panel and the finger. The scheduled time calculation unit 13 may calculate the speed at which the finger approaches the touch panel from the time series information of the distance between the touch panel and the finger included in the sign data.

The time from the acquisition of the predictive data to the actual operation may be, for example, a time obtained experimentally in advance for each type of preparation operation. In this case, the scheduled time calculation part 13 should just hold | maintain the time from acquisition of predictive data to actual operation for every kind of preparation operation | movement. Then, the scheduled time calculation unit 13 may set the time obtained by adding the time associated with the type of preparation operation of the predictor data to the reception time of the predictor data in the operation receiving unit 10.

When the operation receiving unit 10 does not receive actual operation data by the calculated scheduled time (step S23, N), the operation output unit 14 displays the generated complementary data at the calculated scheduled time. It outputs to the screen generation part 20 of the apparatus 2 (step S25).

The screen generation unit 20 that has received the complementary data executes a process based on an instruction represented by the operation of the complementary data, and generates a screen after the operation. The screen generation unit 20 displays the generated screen on the display unit 21. For example, if the operation of the complementary data is drawing of a character or a graphic, the screen generation unit 20 generates a character or a graphic drawn by the operation of the complementary data on the screen displayed on the display unit 21.

Note that the user of the terminal 3 may stop the operation following the preparation operation represented by the predictive data. The operation output unit 14 is a cancel instruction that is an instruction to return the screen to a state where the operation of the complementary data is not performed when the operation reception unit 10 does not receive the actual operation data within a predetermined time after the indication data is received. May be transmitted to the screen generation unit 20. For example, the operation output unit 14 designates the complementary data to be cancelled by the identifier assigned to each complementary data. In addition to the cancellation instruction, the operation output unit 14 may transmit complementary data that is a target of the cancellation instruction. When the screen generation unit 20 receives the cancel instruction, the screen generation unit 20 returns the screen to the state before the processing by the operation of the complementary data. For example, if the operation of complementary data is drawing of characters and figures as described above, the screen generation unit 20 erases the characters and figures drawn by the operation of complementary data generated on the screen displayed on the display unit 21. To do.

In addition, when the operation receiving unit 10 receives the actual operation data after the operation output unit 14 transmits the complementary data, the operation output unit 14 displays the received actual operation data and an instruction to cancel the complementary data on the screen. It transmits to the generation unit 20. In this case, the screen generation unit 20 draws a character or a figure by an operation of actual operation data by superimposing it on a character or a figure drawn by the operation of complementary data on the screen. Then, the screen generation unit 20 erases characters and figures drawn by the operation of the complementary data from the screen. The order of transmission of the actual operation data and the complementary data cancellation instruction by the operation output unit 14 is preferably the above-described order, but may be reversed.

When the operation receiving unit 10 receives actual operation data by the calculated scheduled time (step S23, Y), the operation output unit 14 generates the screen of the display device 2 from the actual operation data received by the operation receiving unit 10. It outputs to the part 20 (step S25). In this case, the operation output unit 14 may output the actual operation data to the screen generation unit 20 by the calculated scheduled time. The operation output unit 14 may output actual operation data to the screen generation unit 20 at the calculated scheduled time.

The screen generation unit 20 that has received the actual operation data executes processing instructed by the operation of the actual operation data, and generates a screen after the operation. The screen generation unit 20 displays the generated screen on the display unit 21.

In this embodiment, there is an effect that smooth communication hindrance due to a delay in the transfer of operation data from other devices can be reduced between a plurality of devices that can communicate with each other and share the operation data by transfer. .

The reason is that the operation output unit 14 represents the estimated operation when the operation receiving unit 10 does not receive the actual operation data by the operation estimated from the predictor data and the scheduled time calculated from the reception time of the predictor data. This is because complementary data is output.

FIG. 5 is a diagram illustrating an example of timing of transmission of the predictor data and actual operation data in the terminal 3, reception of the predictor data and actual operation data, and transmission of complementary data in the operation information control device 1. In the example of FIG. 5, the delay time at the time of actual operation data transmission is expanded as compared with the delay time at the time of predictive data transmission. The difference between the predictive data reception time and the actual operation data reception time in the operation information control device 1 is larger than the difference between the predictor data transmission time and the actual operation data transmission time at the terminal 3.

When the delay of the transfer of operation data from the terminal 3 to the information processing apparatus 5 sharing the operation result screen with the terminal 3 increases due to an increase in communication delay or congestion in the network 4, the operation is performed on the terminal 3. The time lag until the operation result is displayed on the display unit 21 is also increased. Even if a communication delay exists, if the delay time is constant and the time lag described above is constant, the user of the terminal 3 and the user of the information processing device 5 confirm that the time lag of a certain length exists. As a premise, it is possible to communicate smoothly. However, particularly when the operation data transfer delay via the network 4 increases, the user of the terminal 3 and the user of the information processing device 5 smoothly communicate via the screen shared by the terminal 3 and the information processing device 5. I can't do that.

In the present embodiment, the operation estimation unit 12 estimates complementary data representing the operation of the user of the terminal 3 from the predictor data. When the operation receiving unit 10 does not receive actual operation data by the scheduled time calculated by the scheduled time calculating unit 13, the operation output unit 14 outputs complementary data to the screen generating unit 20 at the scheduled time. In the above-described example, the scheduled time calculated by the scheduled time calculation unit 13 is an estimated value of the time at which the operation receiving unit 10 receives actual operation data related to the preliminary operation of the predictor data. The operation information control apparatus 1 according to the present embodiment does not increase the time lag when the communication delay of the network 4 is increased. As a result, the effect of reducing the hindrance due to fluctuations in the transfer delay of the smooth communication between the user of the terminal 3 and the user of the information processing device 5 can be obtained.

(First modification of the first embodiment)
As described above, in the first embodiment, the scheduled time calculated by the scheduled time calculation unit 13 is an estimation of the time at which the operation receiving unit 10 receives the actual operation data related to the preliminary operation of the predictor data. Value. The scheduled time in this case is a time at which the operation receiving unit 10 receives the actual operation data when the delay time when the actual operation data is transferred is the same as the delay time when the predictive data is transferred.

The scheduled time calculation unit 13 does not calculate the scheduled time as in the first embodiment described above, but estimates the time when the user of the terminal 3 performs the actual operation, and at that time, the information processing device 5 The scheduled time may be calculated so that the result of the complementary operation of the actual operation is displayed on the display unit 21. The scheduled time of this modification is a time at which the operation receiving unit 10 receives actual operation data from the terminal 3 when it is assumed that there is no delay time at the time of transfer of actual operation data or can be ignored.

In this case, in step S22, the scheduled time calculation unit 13 performs, for example, an indication data reception time, a delay time when the indication data is detected detected by the delay detection unit 15, and an operation following the preparation operation from the indication data preparation operation. The estimated time is calculated from the estimated time until the operation. The estimated time may be an estimated value of the time from the time when the preparation operation is detected to the time when the actual operation following the preparation operation is detected. Hereinafter, this estimated time is also referred to as preparation operation time. The scheduled time calculation unit 13 adds the estimated time from the preparation operation to the actual operation following the preparation operation to the indication data reception time, and calculates a time obtained by subtracting the delay time when the detected indication data is received as the estimation time. do it.

In this case, since the scheduled time is an estimated time when the user of the terminal 3 performs the actual operation, if the estimated time is correct, the actual operation data is not received until the calculated scheduled time. If the estimated estimated time is not correct, actual operation data may be received by the estimated time.

When the actual operation data is not received by the calculated scheduled time, the operation output unit 14 transmits the complementary data to the screen generation unit 20 at the scheduled time. When the time at which the scheduled time is calculated is before the calculated scheduled time, the operation output unit 14 immediately transmits the complementary data to the screen generation unit 20. When actual operation data is received before the calculated scheduled time, the operation output unit 14 may transmit the received actual operation data to the screen generation unit 20 at the scheduled time.

The delay detector 15 only needs to measure the data transfer and the delay time of the audio signal at all times or every predetermined time. When the predictive data reception time and the delay time measurement time are different, the scheduled time calculation unit 13 may set the delay time measured at the time closest to the predictive data reception time as the delay time when the predictive data is received.

As described above, when the scheduled time calculation unit 13 calculates the scheduled time, the operation information control apparatus 1 estimates that the user of the terminal 3 performs an operation even when the communication delay of the network 4 is not negligible. The result of the estimated operation of the operation is displayed on the display unit 21 at the time to be displayed. As a result, it is possible to obtain an effect of further reducing the hindrance caused by fluctuations in transfer delay in the smooth communication between the user of the terminal 3 and the user of the information processing device 5.

The scheduled time calculation unit 13 may calculate the display change time by adding the preparation operation time to the reception time of the predictor data in addition to calculating the scheduled time of the above-described modification. Further, the operation output unit 14 may transmit the complemented data to which the calculated display change time and an identifier indicating that the operation data is supplemental data are added to the screen generation unit 20.

In this case, if an identifier indicating that the operation data is complementary data is added to the complementary data, the screen generation unit 20 determines whether drawing by the operation indicating the complementary data and drawing by the operation indicating the actual operation data are performed. Generate a screen after the operation represented by the complementary data in a possible format. For example, when the complementary data represents drawing of a figure or character such as a stroke or a dot, the screen generation unit 20 sets a color that is not used as a color of a figure or character that is set in advance by the operation represented by the actual operation data. It may be used to draw a figure or character of the operation represented by the complementary data. The screen generation unit 20 may blink only a figure or a figure drawn by an operation representing complementary data. The screen generation unit 20 may generate a screen in which a predetermined graphic such as an arrow or an icon is added only to a character or graphic drawn by the operation represented by the complementary data.

Then, when the screen generation unit 20 does not receive the actual operation related to the complementary operation by the screen change time, the screen generation unit 20 changes the drawing by the operation represented by the complementary data to the same format as the drawing by the actual operation at the screen change time. . If the reception time of the complementary data is after the display change time, the screen generation unit 20 may perform the drawing by the operation represented by the complementary data in the same form as the drawing by the actual operation.

(Second modification of the first embodiment)
The scheduled time calculation unit 13 does not calculate the scheduled time as in the first embodiment described above, but after the user of the terminal 3 performs an actual operation, the operation is performed on the display unit 21 of the information processing device 5. The scheduled time may be calculated so that the time until the result is displayed is constant.

In this case, in step S22, the scheduled time calculation unit 13 performs, for example, an indication data reception time, a delay time when the indication data is detected detected by the delay detection unit 15, and an operation following the preparation operation from the indication data preparation operation. The estimated time is calculated from the estimated time until the operation. The scheduled time calculation unit 13 adds the predetermined time delay and the estimated time from the preparatory operation to the actual operation following the preparatory operation to the predictive data reception time, and subtracts the delay time when the detected predictive data is received. The time may be calculated as the scheduled time.

If the actual operation data is not received by the calculated scheduled time, the operation output unit 14 transmits the complementary data to the screen generation unit 20 at the scheduled time. When the actual operation data is received before the calculated scheduled time, the operation output unit 14 may transmit the received actual operation data to the screen generation unit 20 at the scheduled time.

As described above, when the scheduled time calculation unit 13 calculates the scheduled time, the operation information control apparatus 1 can perform the operation after the user of the terminal 3 performs the operation even when the communication delay of the network 4 varies. The time until the result is displayed on the display unit 21 can be kept constant. As a result, it is possible to obtain an effect of further reducing the hindrance caused by fluctuations in transfer delay in the smooth communication between the user of the terminal 3 and the user of the information processing device 5.

(Third Modification of First Embodiment)
In addition to the communication using the shared screen, the user of the terminal 3 and the user of the information processing apparatus 5 may make a voice call using a soft phone that operates on each of the terminal 3 and the information processing apparatus 5, for example. . In that case, when the delay time at the time of transfer of the actual operation data is equal to the delay time of the audio signal, the scheduled time calculation unit 13 sets the time at which the operation reception unit 10 receives the actual operation data as the above-described scheduled time. It may be calculated.

In this case, in step S22, the scheduled time calculation unit 13 uses, for example, the data transfer and audio signal delay time between the terminal 3 and the operation information control device 1 detected by the delay detection unit 15, Calculate the scheduled time. The delay detection unit 15 may measure the data transfer and the delay time of the audio signal at all times or every predetermined time.

The method for detecting the delay time of data transfer may be any existing method such as a method of measuring RTT (Round Trip Time) using a measurement packet.

The detection method of the delay time of the audio signal may be any existing method. For example, the information processing apparatus 5 can measure the delay of the measurement packet transmitted from the terminal 3 and estimate the delay time of the audio signal from the measurement result. If the time from the voice input to the voice signal transmission at the terminal 3 and the time from the voice signal reception to the voice output at the information processing apparatus 5 are known, the delay of the measurement packet sent from the terminal 3 is measured. By doing so, it is possible to estimate the delay time of the audio signal. Alternatively, the information processing device 5 can also measure the time from the input of sound at the terminal 3 to the output of the sound at the information processing device 5. In this case, the terminal 3 may transmit a measurement sound and transmit it to the information processing apparatus 5 as a sound signal. The terminal 3 may further transmit the time when the measurement sound is transmitted to the information processing apparatus 5. The information processing device 5 may detect the measurement sound from the sound generated from the received sound signal. And the information processing apparatus 5 should just calculate the difference of the transmission time and detection time of the sound for a measurement.

Then, the scheduled time calculation unit 13 may calculate the above-described scheduled time when the data transfer is delayed for the same time as the delay time of the detected audio signal. The scheduled time calculation unit 13 subtracts the detected delay time of data transfer from the reception time of the predictive data in the operation receiving unit 10, and is associated with the detected delay time of the audio signal and the type of preparation operation of the predictive data. What is necessary is just to make the time which added the time which has been made into scheduled time.

In communication via an IP (Internet Protocol) network, data transfer is often performed by TCP (Transmission Control Protocol). On the other hand, in an IP phone implemented as a soft phone, transmission of a voice signal is often performed by UDP (User Datagram Protocol). In addition, IP telephones often convert received voice signal packets into voice after buffering them. Therefore, even in communication via the same IP network, the data transfer delay time and the audio signal delay time are not always the same.

The operation output unit 14 outputs the complementary data to the screen generation unit 20 at the scheduled time when the actual operation data has not arrived by the scheduled time calculated as described above. When actual operation data arrives before the scheduled time, the operation output unit 14 may output the actual operation data to the screen generation unit 20 at the scheduled time.

As a result, even when the delay time of the data transfer and the delay time of the audio signal are different, the screen change due to the operation of the operation data or the complementary data in the information processing device 5 and the timing of the sound reproduction are the operations of the user of the terminal 3 And the timing of the utterance. Therefore, for example, when the user of the terminal 3 gives an explanation by voice while drawing a figure on the screen, the timing of drawing of the figure and the reproduction of the voice in the information processing device 5 is the drawing and the drawing at the time of explanation by the user of the terminal 3 It becomes equivalent to the timing of utterance. Therefore, an effect of further reducing the obstruction of smooth communication between the user of the terminal 3 and the user of the information processing device 5 due to the difference in the delay time of the transfer of actual operation data and the delay time of the audio signal can be obtained.

(Second Embodiment)
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 6 is a block diagram showing the configuration of the information processing system 100A of this embodiment.

6 and 1 are different from the information processing system 100 in that the information processing system 100A includes the information processing device 5A instead of the information processing device 5. The information processing device 5 </ b> A is different from the information processing device 5 in that the information processing device 5 </ b> A includes the operation information control device 1 </ b> A instead of the operation information control device 1.

Further, the operation information control apparatus 1A is different from the operation information control apparatus 1 in that it includes a difference determination unit 16.

The difference determination unit 16 calculates the difference between the operation of the complementary data transmitted to the display device 2 and the operation represented by the actual operation data received from the terminal 3 that is associated with the same sign data. The difference determination unit 16 compares the calculated difference with a predetermined threshold value. The difference determination unit 16 may calculate an evaluation value of an operation difference determined for each operation type. And the difference determination part 16 should just compare the calculated evaluation value and the threshold value with respect to the evaluation value defined for every kind of operation.

When the calculated difference is larger than the difference represented by the threshold, the operation output unit 14 further outputs a cancel instruction for canceling the display based on the complementary data for which the difference from the actual operation data is calculated, and the actual operation data to the screen generation unit. 20 to send.

Upon receiving the cancel instruction, the screen generation unit 20 returns the screen displayed on the display unit 21 to the state before the processing by the operation of the complementary data that is the target of the cancel instruction. Then, the screen generation unit 20 generates a screen to be displayed on the display unit 21 by performing processing based on the operation of the received actual operation data. The screen generation unit 20 displays the generated screen on the display unit 21.

The order in which the operation output unit 14 transmits the actual operation data and the instruction for canceling the complementary data may be reverse to the above-described order. In that case, the screen generation unit 20 performs drawing by operating the actual operation data on the screen displayed on the display unit 21 in a manner superimposed on the drawing by operating the complementary data. Then, the screen generation unit 20 erases the drawing by operating the complementary data from the screen.

Next, the operation of the operation information control apparatus 1A of the present embodiment will be described in detail with reference to the drawings.

FIG. 7 is a flowchart showing the operation of the operation information control apparatus 1A of the present embodiment.

7 is the same as the operation of the step with the same reference numeral in FIG. 4 and the description thereof is omitted.

When the operation receiving unit 10 has not received the actual operation data by the scheduled time in step S23 (step S23, N), the operation receiving unit 10 transmits the actual operation data after the transmission of the complementary data (step S25). Receive (step S26).

The difference determination unit 16 derives the difference between the complementary operation represented by the complementary data and the actual operation represented by the actual operation data based on the transmitted complementary data and the received actual operation data. If the derived difference is smaller than a predetermined threshold for the difference (step S27, N), the process returns to step S20.

As described above, the operation output unit 14 transmits a cancel instruction to return to a state in which the operation of the complementary data generated from the predictor data is not performed when a predetermined time or more has elapsed since the predictor data was received. Also good. When the actual operation data is received after the cancellation instruction is transmitted, the operation information control apparatus 1A may output the actual operation data to the screen generation unit 20 without performing the operations of Step S27 and Step S28 described later.

差異 The existing calculation method may be used as the difference calculation method. For example, when the actual operation data and the complementary data represent an operation of inputting a graphic, the difference determination unit 16 determines the distance between the starting point of the graphic drawn by the actual operational data and the starting point of the graphic drawn by the complementary data as the difference. Set to evaluation value. In the case where the actual operation data and the complementary data represent, for example, an operation of inputting characters using a software keyboard, the difference determination unit 16 may determine the difference between the characters of the actual operation data and the characters of the complementary data. Then, the difference determination unit 16 sets the difference evaluation value to 0 when the character of the actual operation data is the same as the character of the complementary data, and sets the difference evaluation value to be predetermined when the character of the actual operation data is different from the character of the complementary data. It can be a value. The threshold value in this case may be a value between 0 and the predetermined value.

When the calculated difference is larger than a predetermined threshold for the difference (step S27, Y), the operation output unit 14 transmits a complementary data cancellation instruction (step S28).

The screen generation unit 20 that has received the complementary data cancellation instruction returns the screen displayed on the display unit 21 to the state before the processing by the operation represented by the complementary data is performed.

The operation output unit 14 transmits the cancellation instruction, and then transmits the received actual operation data to the screen generation unit 20 (step S24).

The screen generation unit 20 that has received the actual operation data generates a screen as a result of performing the process by the operation represented by the received actual operation data, and causes the display unit 21 to display the generated screen.

If the calculated difference is greater than a predetermined threshold for the difference (step S27, Y), the operation output unit 14 transmits the received actual operation data to the screen generation unit 20 (step S24). A complementary data cancellation instruction may be transmitted (step S28).

In addition to the same effects as those of the first embodiment, this embodiment described above has an effect that the user of the terminal 3 and the user of the information processing apparatus 5 can share an accurate operation.

The reason is that the difference determination unit 16 according to the present embodiment has a difference represented by a predetermined threshold value between the operation represented by the transmitted complementary data and the operation represented by the actual operation data received after the complementary data transmission. This is because the presence or absence of a larger difference is determined. When such a difference exists, the operation output unit 14 outputs the complementary data cancellation instruction and the received actual operation data to the screen generation unit 20. As a result, when the difference between the actual operation and the complementary operation is large, the display by the complementary operation is corrected to the display by the actual operation.

(Third embodiment)
Next, a third embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 8 is a block diagram showing the configuration of the information processing apparatus 5B of this embodiment.

When FIG. 8 is compared with FIG. 1, the information processing device 5B includes an input unit 30 and a preparation operation detection unit 31 that are components of the terminal 3 in addition to the components of the information processing device 5 of the first embodiment. And an operation detection unit 32. Each component of the information processing apparatus 5B is the same as the component given the same number in FIG.

FIG. 9 is a diagram illustrating a configuration of the information processing system 100B according to the present embodiment.

9 is compared with FIG. 2, the information processing system 100B is obtained by replacing the terminal 3 and the information processing apparatus 5 of the information processing system 100 of the first embodiment with the information processing apparatus 5B, respectively. Although the number of information processing devices 5B connected to the network 4 in FIG. 9 is two, three or more information processing devices 5B may be connected to the network 4.

Next, the operation of the information processing apparatus 5B according to the present embodiment will be described in detail with reference to the drawings.

The information processing apparatus 5B operates as both the terminal 3 and the information processing apparatus 5 of the first embodiment. For example, when a user of an information processing device 5B performs an operation on the information processing device 5B, the information processing device 5B operates as the terminal 3, and the other information processing device 5B operates as the information processing device 5. Each information processing apparatus 5B may operate as the terminal 3 and the information processing apparatus 5 at the same time. That is, the information processing device 5B transmits the predictor data and the actual operation data to the other information processing device 5B as the terminal 3, while the information processing device 5 transmits the predictor data and the actual operation data from the other information processing device 5B. You may receive it.

FIG. 3 is a flowchart showing the operation of the transmission-side information processing device 5B operating as the terminal 3 when acquiring input device information.

FIG. 4 is a flowchart showing an operation of the reception information processing apparatus 5B operating as the information processing apparatus 5 when the operation information control apparatus 1 receives the preparation information. As described above, the preparation information is also expressed as predictive data.

The operation of the information processing device 5B on the transmission side is the same as the operation of the terminal 3 in the first embodiment, and the operation of the information processing device 5B on the reception side is the same as the operation of the information processing device 5 in the first embodiment. Since it is the same, explanation is omitted.

The present embodiment described above has the same effect as the first embodiment in communication in any direction in communication performed by a set of two information processing devices 5B.

The information processing device 5B may include the operation information control device 1A of the second embodiment instead of the operation information control device 1.

In this case, the present embodiment has the same effect as the second embodiment in communication in any direction in communication performed by a set of two information processing devices 5B.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 10 is a block diagram showing the configuration of the information processing system 100C of this embodiment.

Referring to FIG. 10, the information processing system 100 </ b> C includes an operation information control device 1, a display device 2, and a terminal 3. The operation information control device 1, the display device 2, and the terminal 3 of the present embodiment are the same as the operation information control device 1, the display device 2, and the terminal 3 of the first embodiment, respectively. Therefore, description of each device is omitted.

FIG. 11 is a diagram illustrating connection of devices included in the information processing system 100C.

Each device is connected via the network 4. One display device 2 may be provided. Two or more display devices 2 may exist as shown in FIG.

Next, the operation of this embodiment will be described in detail with reference to the drawings.

FIG. 3 is a flowchart showing the operation when the input device information of the terminal 3 of this embodiment is acquired.

FIG. 4 is a flowchart showing the operation of the operation information control apparatus 1 according to this embodiment when receiving the preparation information.

The operation information control device 1 of the present embodiment transmits complementary data and actual operation data to each of the display devices 2 when a plurality of display devices 2 are connected to the network 4.

Other operations of the terminal 3 and the operation information control device 1 of the present embodiment are the same as the operations of the terminal 3 and the operation information control device 1 of the first embodiment, respectively, and thus description thereof is omitted.

The present embodiment described above has the same effect as the first embodiment.

This embodiment further has an effect of reducing the cost when there are a plurality of users on the actual operation data receiving side.

The reason is that devices held by a plurality of users on the receiving side do not include the operation information control device 1. In the present embodiment, each of the plurality of users on the receiving side only needs to hold the display device 2. Only one operation information control device 1 is required.

(Configuration example)
Next, a configuration example based on the first embodiment of the present invention will be described. This configuration example is a handwriting sharing system.

FIG. 12 is a diagram showing the configuration of the handwriting sharing system of this configuration example.

Referring to FIG. 12, the handwriting sharing system includes a tablet terminal 400 and a PC 410 (Personal Computer) connected via the Internet 420. The tablet terminal 400 is the terminal 3 of the first embodiment. The PC 410 is the information processing apparatus 5. The Internet 420 is the network 4.

The tablet terminal 400 includes a capacitive touch screen. In the tablet terminal 400, a handwriting program that allows the user to draw characters and strokes is operating. The tablet terminal 400 operates as the terminal 3 in FIG. 1 by a handwriting program executed by a CPU (Central Processing Unit) of the tablet terminal 400. The handwriting program causes the tablet terminal 400 to operate as the input unit 30, the preparation operation detection unit 31, the operation detection unit 32, the screen generation unit 33, and the display unit 34. The user A operates the tablet terminal 400 that executes the handwriting program by bringing the finger of the hand into contact with the touch screen. The touch screen of the tablet terminal 400 operates as the input unit 30 and the display unit 34.

The PC 410 includes a display device. The PC 410 operates as the operation information control apparatus 1 in FIG. 1 according to an operation information control program executed by the CPU of the PC 410. The operation information control program causes the PC 410 to operate as the operation reception unit 10, the operation storage unit 11, the operation estimation unit 12, the scheduled time calculation unit 13, and the operation output unit 14. The operation information control program may further cause the PC 410 to operate as the delay detection unit 15.

The PC 410 further operates as the display device 2 in FIG. 1 by a viewer program executed by the CPU of the PC 410. The viewer program causes the PC 410 to operate as the screen generation unit 20 and the display unit 21. The display device of the PC 410 operates as the display unit 21.

The call program is operating in each of the tablet terminal 400 and the PC 410.

In this configuration example, the user A using the tablet terminal 400 and the user B using the PC 410 perform remote communication using the handwriting sharing system while talking using the call software.

Each program operating on the tablet terminal 400 is stored in a storage device included in the tablet terminal 400, transferred to the memory of the tablet terminal 400, and executed by the CPU of the tablet terminal 400.

Each program operating on the PC 410 is stored in a storage device included in the PC 410, transferred to the memory of the PC 410, and executed by the CPU of the PC 410.

The storage device of the tablet terminal 400 and the PC 410 is, for example, a hard disk or a nonvolatile memory. The storage device, memory, and CPU of the tablet terminal 400 and the PC 410 are not shown.

When the handwriting program of the tablet terminal 400 is activated, the tablet terminal 400 acquires the capacitance value of the touch screen of the tablet terminal 400.

Capacitance type touch screen detects the position by detecting the change in capacitance between the fingertip and the conductive film. Since electrostatic coupling occurs only when the finger approaches the sensor surface, a change in the situation can be detected before the finger actually touches the touch screen. The capacitance is inversely proportional to the distance between the fingertip and the touch screen. The capacitance value and the distance between the fingertip and the touch screen in the following description are merely examples, and are not limited to these.

Suppose that the capacitor built in the touch screen of the tablet terminal 400 has a capacitance of 100 μF (micro (= 1/6) farad) when the finger touches the touch screen.

The preparation operation detection unit 31 determines that the finger is approaching the touch screen when the capacitance indicates 1 μF, and generates predictive data. When the capacitance is 1 μF, the distance between the finger and the touch screen is about 2 cm.

Preparation operation detection unit 31 transmits first predictive data (1) to PC 410. The predictor data includes the identifier for the predictor data, the operation occurrence time, the center position information of the range in which the capacitance has changed, the distance between the finger and the touch screen, and the measured network delay between the tablet terminal 400 and the PC 410. including. In the predictive data (1), the distance between the finger and the touch screen is “2 cm”, and the network delay is “0.1 s”. The preparation operation detection unit 31 measures the network delay using RTT (Round Trip Time).

The preparatory movement detection unit 31 monitors the change in the capacitance by measuring the capacitance at intervals of 0.2 seconds until the next actual operation data is detected when the sign data is detected. Assume that the capacitance measurement result 0.2 seconds after the transmission of the first predictive data is 40 μF. In this case, it is assumed that the distance between the finger and the touch screen is “1.2 cm”. Further, it is assumed that the measurement result of the network delay is “0.1 s”. The preparation operation detection unit 31 transmits predictive data (2) including these values to the PC 410. Further, the measurement result of the capacitance after 0.2 seconds is assumed to be 80 μF. In this case, it is assumed that the distance between the finger and the touch screen is “0.4 cm”. Further, it is assumed that the measurement result of the network delay is “0.15 s”. The preparation operation detection unit 31 transmits predictive data (3) including these values to the PC 410.

And when the finger actually touches the tablet terminal 400, the capacitance becomes 100 μF. The operation detection unit 32 transmits actual operation data (1) including an identifier for actual operation data, an operation occurrence time, and touched position information to the PC 410. The screen generation unit 33 performs drawing on the screen of the handwriting program.

The operation receiving unit 10 of the PC 410 receives data transmitted from the tablet terminal 400. The operation receiving unit 10 determines whether the received data is “predictive data” or “actual operation data” with reference to the identifier. The identifier of this configuration example is information that specifies whether the data is “predictive data” or “actual operation data”.

The operation receiving unit 10 acquires the predictive data (1) and then acquires the predictive data (2).

The operation estimation unit 12 calculates the predicted position where the finger touches the touch screen from the center position information of the range in which the capacitance has changed in the predictor data (1) and the predictor data (2) and the distance between the finger and the touch screen. Is calculated and complementary data is generated.

The scheduled time calculation unit 13 calculates the moving speed of the finger from the distance between the finger and the touch screen of the predictor data (1) and the predictor data (2). The scheduled time calculation unit 13 actually touches the touch screen from the distance, the calculated moving speed, and the measurement time of the distance between the finger and the touch screen of the predictor data (1) or the predictor data (2). Estimate the time until the time. In the above example, the estimated time is “0.5 seconds”. Then, the scheduled time calculation unit 13 calculates the time when the actual operation data is received as the scheduled time from the estimated time and the reception time of the predictive data (1) or the predictive data (2). To do. The scheduled time calculation unit 13 may calculate the scheduled time in consideration of the network delay, as in each modification of the first embodiment.

When the operation receiving unit 10 receives the next predictive data (3) before the scheduled time, the operation estimating unit 12 may regenerate complementary data using the predictive data (3). In this case, the scheduled time calculation unit 13 may recalculate the scheduled time using the predictor data (3). The operation estimation unit 12 and the scheduled time calculation unit 13 may discard the predictor data (3).

When the operation receiving unit 10 does not receive actual operation data until the scheduled time, the operation output unit 14 outputs the complementary data to the screen generating unit 20 at the scheduled time.

The screen generation unit 20 displays the result of the operation represented by the complementary data on the display unit 21 that is a display.

When the actual operation data (1) is received after the supplementary data is transmitted, the PC 410 may operate like the operation information control device 1A of the second embodiment. That is, when the difference between the operations of the complementary data and the actual operation data (1) is large, the operation output unit 14 may transmit the instruction for canceling the complementary data and the actual operation data (1) to the screen generation unit 20. Good. Alternatively, the operation output unit 14 may transmit the actual operation data (1) to the screen generation unit 20. In this case, the character or figure by the operation of the actual operation data (1) is displayed superimposed on the character or figure by the complementary data. Thereafter, the operation output unit 14 may transmit a complementary data cancellation instruction to the screen generation unit 20.

In the above example, the scheduled time calculation unit 13 calculates the moving speed of the finger from the distance between the finger and the touch screen, but the scheduled time calculation unit 13 calculates the finger movement from the amount of change in capacitance. It is also possible to calculate the moving speed.

When the user A and the user B share the handwritten drawing while having a conversation, if the handwritten drawing is delayed, smooth communication cannot be established. Therefore, according to each embodiment and configuration example of the present invention, smooth communication is possible even in an environment where the delay width is increased or decreased, such as the Internet.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 13 is a diagram illustrating a configuration of the operation information control apparatus 1 according to the present embodiment.

Referring to FIG. 13, the operation information control apparatus 1 according to the present embodiment includes a preparation operation for a user operation on the terminal 3, preparation information representing the preparation operation, and operation information representing the operation from the terminal 3 that detects the operation. The operation receiving unit 10 that receives the time interval and the operation storage unit 11 that stores the type of the preparatory operation and the type of operation in association with each other, and the preparation information after the preparatory operation from the preparation information The operation estimation unit 12 that estimates the type of operation related to the type of the operation, the display device 2 that receives the operation information, executes the operation of the operation information, and displays the result. A scheduled time calculation unit 13 that calculates a scheduled time to be received and transmitted from the preparation information and a reception time of the preparation information, and the operation information when the operation information is received by the scheduled time. An operation output unit 14 that transmits the operation information of the estimated operation, which is the estimated operation, to the display device 2 at the scheduled time when the operation information is transmitted to the display device 2 and is not received by the scheduled time ,including.

The operation information control device 1, the operation information control device 1A, the display device 2, the terminal 3, the information processing device 5, the information processing device 5A, and the information processing device 5B are respectively a computer, a program for controlling the computer, dedicated hardware, Alternatively, it can be realized by a combination of a computer and a program for controlling the computer and dedicated hardware.

FIG. 14 shows the configuration of the computer 1000 used to realize the operation information control device 1, the operation information control device 1A, the display device 2, the terminal 3, the information processing device 5, the information processing device 5A, and the information processing device 5B. FIG.

Referring to FIG. 14, the computer 1000 includes a processor 1001, a memory 1002, a storage device 1003, and a communication interface 1004. The communication interface 1004 is connected to the network 4. The processor 1001 can communicate with other devices connected to the network 4 via the communication interface 1004. The processor 1001 can access the recording medium 1005. The storage device 1003 is realized by a storage device such as a memory or a hard disk device included in the computer 1000. The recording medium 1005 is realized by a storage device such as a hard disk device or a portable recording medium. The storage device 1003 may operate as the recording medium 1005. The recording medium 1005 stores a program that causes the computer 1000 to operate as the operation information control device 1, the operation information control device 1A, the display device 2, the terminal 3, the information processing device 5, the information processing device 5A, or the information processing device 5B. ing. The processor 1001 reads a program stored in the recording medium 1005 and loads the read program into the memory 1002. Then, the processor 1001 executes the program loaded in the memory 1002. When the processor 1001 executes the program loaded in the memory 1002, the computer 1000 causes the operation information control device 1, the operation information control device 1A, the display device 2, the terminal 3, the information processing device 5, the information processing device 5A, or It operates as the information processing apparatus 5B.

Operation reception unit 10, operation estimation unit 12, scheduled time calculation unit 13, operation output unit 14, delay detection unit 15, difference determination unit 16, screen generation unit 20, display unit 21, input unit 30, preparation operation detection unit 31, The operation detection unit 32, the screen generation unit 33, and the display unit 34 execute, for example, a dedicated program for realizing the function of each unit read from the recording medium 1005 that stores the program into the memory 1002, and the program. It can be realized by the processor 1001. The operation storage unit 11 can be realized by a storage device 1003 that is a memory included in a computer or a hard disk device. Alternatively, the operation reception unit 10, the operation storage unit 11, the operation estimation unit 12, the scheduled time calculation unit 13, the operation output unit 14, the delay detection unit 15, the difference determination unit 16, the screen generation unit 20, the display unit 21, and the input unit 30. In addition, a part or all of the preparation operation detection unit 31, the operation detection unit 32, the screen generation unit 33, and the display unit 34 can be realized by a dedicated circuit that realizes the function of each unit.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2012-187588 filed on August 28, 2012, the entire disclosure of which is incorporated herein.

The present invention can be applied to a communication service using a network and a remote operation system.

DESCRIPTION OF SYMBOLS 1, 1A Operation information control apparatus 2 Display apparatus 3 Terminal 4

Network

5, 5A, 5B Information processing apparatus 10 Operation receiving part 11 Operation memory | storage part 12 Operation estimation part 13 Scheduled time calculation part 14 Operation output part 15 Delay detection part 16

Difference determination Unit

20, 33

Screen generation unit

21, 34 Display unit 30 Input unit 31 Preparation operation detection unit 32

Operation detection unit

100, 100A, 100B, 100C Information processing system 400 Tablet terminal 410 PC
420 Internet 1000 Computer 1001 Processor 1002 Memory 1003 Storage Device 1004 Recording Medium

Claims

An operation receiving means for receiving preparation information indicating the preparation operation and operation information indicating the operation from the terminal that detects the preparation operation of the user's operation on the terminal and the operation;
Operation storage means for associating and storing the type of the preparation operation and the type of operation;
From the preparation information, operation estimation means for estimating the type of operation related to the type of the preparation information after the preparation operation,
Receiving the operation information and executing the operation of the operation information and displaying a result, and displaying the result of receiving the operation information from the terminal and transmitting the operation information, the reception time of the preparation information and the preparation information A scheduled time calculation means to calculate from
When the operation information is received by the scheduled time, the operation information is transmitted to the display device, and when the operation information is not received by the scheduled time, the operation information of the estimated operation that is the estimated operation Operation output means for transmitting to the display device at the scheduled time;
An operation information control device.
Delay detection means for detecting a delay time of communication with the terminal and a delay time of a voice call,
The scheduled time calculating means calculates a preparation operation time from the detection of the operation to the detection of the operation calculated from the preparation information, and determines the communication delay from the sum of the preparation operation time and the delay time of the voice call. The operation information control apparatus according to claim 1, wherein a time obtained by subtracting time is calculated as the scheduled time from the reception time of the preparation information.
Delay detection means for detecting a delay time of communication with the terminal,
The scheduled time calculation means calculates a preparation operation time from detection of the operation calculated from the preparation information to detection of the operation, adds the preparation operation time to the reception time of the preparation information, and The operation information control device according to claim 1, wherein a time obtained by subtracting a delay time is calculated as the scheduled time.
A difference determination means for determining a value representing a difference between the estimation operation and the operation and determining whether the value is greater than a predetermined threshold;
When the difference between the estimation operation for which the operation information has been transmitted and the operation received from the terminal related to the estimation operation is greater than the predetermined threshold, the operation output means A cancel instruction to cancel the display and the operation information are transmitted to the display device,
The operation information control device according to claim 1, wherein the display device further deletes the display by the estimation operation when the cancellation instruction is received.
The value representing the difference is a distance between a position where the finger contacts in the operation which is a finger contact with the touch panel and a position where the finger contacts in the estimation operation of the operation. Operation information control device.
The preparation information includes information indicating the type of the preparation operation and the content of the preparation operation,
The operation estimation unit reads the operation type related to the type of the preparatory operation from the operation storage unit, and from the information indicating the content of the preparatory operation, the type is the type of the read operation. The operation information according to claim 1, wherein the content of the estimation operation is estimated, the operation information of the estimation operation including information representing the content is generated, and the operation information is transmitted to the display device. Control device.
The operation storage means stores a pinch operation representing an operation of bringing two fingers in contact with each other while keeping them in contact with each other as the operation type in association with the type of the preparation operation that is a finger contact with two points. And
The operation estimating means calculates a distance between the two points when the kind of the preparatory movement is a finger contact with two points and the content of the preparatory movement is information representing the coordinates of the two points. When the calculated distance is greater than or equal to a predetermined distance, the content of the operation is a pinch-in operation in which two fingers are brought close to each other while the two fingers are in contact, and the distance between the two points is a predetermined distance The operation information control apparatus according to claim 6, wherein the operation information control apparatus estimates that the estimation operation is a pinch-out operation in which two fingers are kept in contact with each other in the pinch operation.
An information processing system including the terminal, the display device, and the operation information control device according to any one of claims 1 to 7.
From the terminal that detects the user operation preparation operation and the operation on the terminal, the preparation information representing the preparation operation and the operation information representing the operation are received at time intervals,
Storing the type of the preparatory operation and the type of operation in the operation storage means in association with each other;
From the preparation information, estimate the type of the operation related to the type of the preparation information after the preparation operation,
Receiving the operation information and executing the operation of the operation information and displaying a result, and displaying the result of receiving the operation information from the terminal and transmitting the operation information, the reception time of the preparation information and the preparation information Calculated from
When the operation information is received by the scheduled time, the operation information is transmitted to the display device, and when the operation information is not received by the scheduled time, the operation information of the estimated operation that is the estimated operation To the display device at the scheduled time,
An operation information control method including:
Computer
An operation receiving means for receiving preparation information indicating the preparation operation and operation information indicating the operation from the terminal that detects the preparation operation of the user's operation on the terminal and the operation;
Operation storage means for associating and storing the type of the preparation operation and the type of operation;
From the preparation information, operation estimation means for estimating the type of operation related to the type of the preparation information after the preparation operation,
Receiving the operation information and executing the operation of the operation information and displaying a result, and displaying the result of receiving the operation information from the terminal and transmitting the operation information, the reception time of the preparation information and the preparation information A scheduled time calculation means to calculate from
When the operation information is received by the scheduled time, the operation information is transmitted to the display device, and when the operation information is not received by the scheduled time, the operation information of the estimated operation that is the estimated operation Operation output means for transmitting to the display device at the scheduled time;
Operation information control program to be operated as