JP2016134673A - Transmission device, transmission method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problems that an output video is disordered to let a user feel unpleasant if a radio MR system enters a worse radio communication state to cause packet loss or delay, and the user feels unpleasant very much, especially, if one output video corresponding to the left eye or right eye is continuously disordered or not renewed.SOLUTION: A transmission device comprises: communication state grasping means of grasping a communication state between the transmission device and a reception device; setting means of setting, based upon the communication state, error correction strength for data for error correction used by the reception device to correct content data that the reception device does not receive normally among transmitted content data to first and second content data respectively; error correction data generating means of generating first and second data for error correction on the basis of the error correction strength set for the first and second content data respectively; and transmitting means of transmitting the first and second content data and the first and second data for error correction to the reception device.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a transmission device, a transmission method, and a program.

  Currently, MR (Mixed Reality) systems that seamlessly integrate the real world and virtual space in real time have been used. One method for realizing the MR system is a video see-through method. As an example of the video see-through method, first, the visual field region of the HMD user is imaged with two video cameras corresponding to the left eye and the right eye attached to an HMD (Head Mounted Display). Next, CG (Computer Graphics) is superimposed on the captured image. Finally, the video on which the CG is superimposed is observed with two displays corresponding to the left eye and the right eye attached to the HMD.

  Since the processing for superimposing CG on the captured image is computationally expensive, it is connected to an external device such as a PC, temporarily transmits the captured image to the external device, superimposes the CG on the external device, and sends it back to the HMD. Or display from. In such a case, in order for the HMD user to experience the MR space while freely moving, it is desirable to communicate wirelessly with an external device that superimposes the HMD and the CG. However, generally, wireless communication has a narrower communication band than wired communication, and packet loss and delay are likely to occur during communication. When a communication error such as packet loss or delay occurs, the video sent back to the HMD cannot be normally output, and the video may be distorted or the video may not be updated for a long time.

  There is a forward error correction (FEC) technique as a technique for improving resistance to communication errors. In the error correction technique, a redundant error correction packet is generated for a data packet, and the error correction packet is transmitted together with the data packet. Therefore, an extra communication band is used. Therefore, the amount of error correction packets that can be generated and transmitted for a finite communication band is limited. Therefore, it is necessary to control application of limited error correction code resources according to the communication status, the importance of data, and the like. For example, Patent Document 1 discloses a method of applying an error correction technique to important data in a system that transmits left and right video data as in the MR system.

JP 2014-27448 A

In the MR system described above, two images corresponding to the left eye and the right eye are observed by two displays corresponding to the left eye and the right eye, respectively. In general, by allocating error correction code resources equally to two video data, it is expected that the right and left video is output normally by improving the tolerance of communication errors between the left and right videos.
However, if the communication situation deteriorates and the error correction code resources assigned to the left and right images alone are insufficient in communication error tolerance, both the left and right images are disturbed or both images are not updated for a long time, The user is greatly discomforted.
The present invention has been made to solve the above-described problems. First, at least two error correction application levels to which error correction code resources are allocated are prepared. Next, the error correction application level for the video frame packet corresponding to the left eye and the right eye is switched in a cycle of at least one frame or more according to a change in communication status. As a result, the communication error tolerance for the left and right video data is periodically switched, and it is possible to reduce the occurrence rate of the situation where the video quality of only one of the left and right videos or the left and right videos is deteriorated for a long time. An object of the present invention is to provide a transmission device and a transmission method.

As one means for achieving the above object, the transmission apparatus of the present invention comprises the following arrangement.
That is, a transmission device that transmits first content data and the second content data to a reception device,
A communication status grasping means for grasping a communication status between the transmitting device and the receiving device;
Based on the communication status, an error correction strength of error correction data used by the receiving device to correct content data that the receiving device has not normally received among the transmitted content data is set to the first content data. Setting means for setting each of the content data and the second content data,
Error correction data generation means for generating first error correction data and second error correction data based on the error correction strength set for each of the first content data and the second content data When,
And transmitting means for transmitting the first content data, the second content data, the first error correction data, and the second error correction data to the receiving device.

  According to the present invention having the above-described configuration, at least two error correction application levels for allocating error correction code resources are prepared, and the error correction application level for video frame packets corresponding to the left eye and the right eye is at least one frame or more. Switch by cycle. As a result, the communication error tolerance for the left and right video data is periodically switched, and it is possible to reduce the occurrence rate of the situation in which the video quality of only one of the left and right videos or only the left and right videos is deteriorated for a long period of time. Become.

1 is a configuration diagram of a wireless MR system according to an embodiment of the present invention. It is a hardware block diagram of HMD concerning the Example of this invention. It is a module block diagram of the packet transmission / reception program concerning the Example of this invention. It is a flowchart which shows the flow of the error correction level switching process concerning the Example of this invention. It is a flowchart which shows the flow of the communication condition determination process in the Example of this invention. This is an example of application of error correction at normal times. This is an example of application of error correction when the communication situation deteriorates. This is an example of application of error correction when the communication status is recovered. This is an example of application of error correction when the communication status further deteriorates. This is an example of application of error correction when the communication situation deteriorates. This is an example of application of error correction to encoded video data using disparity information.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[Example 1]
In the MR system of this embodiment, video data captured by a camera mounted on an HMD (head-mounted display device) and error correction code data for protecting it are transmitted to an external PC, and CG is superimposed on the external PC. The received video data is sent back and displayed on the display mounted on the HMD.

FIG. 1 is a configuration diagram of a wireless MR system described in this embodiment.
A user 101 of the wireless MR system is wearing the HMD 200. A detailed hardware configuration of the HMD 200 will be described later with reference to FIG.
Access point 102 is connected to HMD 200 by wireless communication.
The right eye imaging frame packet group 103 is a packet group of data captured by the right eye image capturing unit 201 in the HMD 200.
The left-eye imaging frame packet group 104 is a packet group of data captured by the left-eye video imaging unit 202 in the HMD 200.

The external PC 105 superimposes an appropriate CG on the right eye imaging frame packet group 103 and the left eye imaging frame packet group 104 received from the HMD 200. The external PC and the access point 102 are connected by a wired LAN (Local Area Network) or the like, but may be wireless or not LAN.
The right eye video frame packet group 106 is a right eye video frame packet group on which CG is superimposed by the external PC 105.
The left-eye video frame packet group 107 is a left-eye video frame packet group on which CG is superimposed by the external PC 105.

FIG. 2 is a hardware configuration block diagram of the HMD 200 in FIG.
The right-eye image capturing unit 201 is a camera that captures a field-of-view image viewed with the right eye of the user 101.
The left-eye image capturing unit 202 is a camera that captures a field-of-view image that is visible with the left eye of the user 101.
The right-eye video imaging unit 201 and the left-eye video imaging unit 202 correspond to video frame acquisition means that acquires first video frame data and second video frame data.

The right-eye video display unit 203 is a display that displays a right-eye video on which CG is superimposed by the CG superposition external PC 105.
The left-eye video display unit 204 is a display that displays a left-eye video on which CG is superimposed by the CG superposition external PC 105.
The network interface 205 is an interface that can be connected to the Internet, public radio, or LAN.

A CPU (Central Processing Unit) 206 comprehensively controls each hardware component.
A ROM (Read Only Memory) 207 stores a control program executed by the CPU 206 and the like.
A RAM (Random Access Memory) 208 functions as a main memory and work area of the CPU 206.

  The encoder 209 converts the video data captured by the right-eye video imaging unit 201 and the left-eye video imaging unit 202 into a specific encoding method such as JPEG (Joint Photographic Expert Group) or H.264. H.264 / AVC (ISO / IEC 14496-10 MPEG-4 Part 10 Advanced Video Coding), H.264 H.264 / MVC (Multi-view Video Coding) and the like are compressed. In the present embodiment, description will be made assuming that encoding is performed using the JPEG system, but the present invention can be applied to other systems. H. Example 2 will be described as an example of an encoding method using left and right disparity information and motion vectors such as H.264 / MVC. In addition, video data captured by each imaging unit may be packetized without being compressed and transmitted to the CG superimposed external PC 105. In that case, the encoder 209 is not included in the system or is not used. The encoder 209 may be implemented by software.

  The encoder 209 corresponds to encoding means, video frame packet generation means, and error correction packet generation means. The encoding means encodes each of the first and second video frame data to obtain first and second video frame encoded data. The video frame packet generation means packetizes each of the first and second video frame encoded data to generate first and second video frame packets. The error correction packet generation means generates a first error correction packet and a second error correction packet with the error correction strength selected for each of the first and second video frame packet groups.

The decoder 210 performs a process of expanding the right-eye video frame packet 106 and the left-eye video frame packet 107 received from the CG superimposed external PC 105 according to a specific decoding method. The decoding method here depends on the encoding method used for compression by the CG superimposed external PC 105, but may be the same method as the encoding method used by the encoder 209 or a different method. Similarly to the encoder 209, the received video frame data may be uncompressed RAW data. In this case, the decoder 210 is not included in the system or is not used. The decoder 210 may be implemented by software.
The bus 211 is a transfer path for various data.

FIG. 3 is a module configuration diagram of the packet transmission / reception program.
The packet generator 301 packetizes the video encoded data input from the encoder 209 according to the communication protocol. Since the wireless MR system has a high demand for real-time performance, a communication protocol such as RTP (A Transport Protocol for Real-Time Applications, RFC3550, IETF) / UDP (User Datagram Protocol) is preferable. However, it may be a proprietary protocol. The packet generation unit 301 divides the encoded content data into a size suitable for communication, and further adds a header necessary for communication to generate an RTP data packet.
In this embodiment, the case where the content data is moving image data will be described. However, the present invention can be applied to data other than that (for example, audio data). In this embodiment, an example in which data is transmitted using the RTP protocol will be mainly described. However, data may be transmitted using another protocol.

The packet transmission unit 302 transmits the packet received from the packet generation unit 301 to the CG superimposed external PC 105 via the network interface 205 using a predetermined communication protocol.
The right-eye video frame packet buffer 303 is a buffer that temporarily stores the right-eye video frame packet packetized by the packet generation unit 301.
The left-eye video frame packet buffer 304 is a buffer that temporarily stores the left-eye video frame packet packetized by the packet generation unit 301.

  Error correction coding sections 305, 306, and 307 generate a restoration packet (FEC packet) according to the packet data generated by packet generation section 301. The error correction coding unit is an example of error correction data generation means. The FEC packet is data for restoration used by the external PC 105 to restore content data that has not been normally received by the external PC 105 (receiving device). The restoration data is an example of error correction data. The error correction encoding units 305, 306, and 307 generate an FEC packet by, for example, an exclusive logical loop operation on content data. The error correction encoding units 305, 306, and 307 have different error correction application levels. The error correction encoders 305, 306, and 307 generate error correction codes from the video frame packet group input from the right eye video frame packet buffer 303 or the left eye video frame packet buffer 304 according to a specific error correction encoding method. To do. In this embodiment, the level can be set in three stages, but it may be two stages or four or more stages. Here, description will be made assuming that the error correction strength is strong in the order of level 1, level 2, and level 3. Error correction coding methods include, for example, a parity method based on exclusive OR operation, a Reed-Solomon coding method, a low density parity check coding method, etc., but any coding method that can set the error correction strength Any method may be used. The error correction coding unit also includes a packet generation unit corresponding to the communication protocol. The packetized error correction packet is transmitted to the CG superimposed external PC 105 by the packet transmission unit 302.

The packet receiving unit 308 receives a packet from the CG superimposed external PC 105 via the network interface 205. The packets received here are roughly divided into two types, one is a video frame packet in which CG is superimposed on the HMD captured video, and the other is a report packet for grasping the communication status. The report packet may use a protocol capable of grasping error rate and delay information such as RTCP (Real-time Transport Control Protocol) Receiver Report and Sender Report, or may be a unique protocol.
In the present embodiment, the description will be made assuming that the result of error correction by the error correction packet generated by the error correction encoding units 305, 306, and 307 is notified, but the present invention is not limited to this.
The packet receiving unit 308 corresponds to a report packet receiving unit that receives a report packet that notifies the communication status from the receiving device, and a video frame packet receiving unit that receives a video frame packet from the receiving device.

  Also, the CG superimposed external PC 105 may generate and transmit an error correction packet. In this case, the error correction packet is also received by this function. The video frame packet on which the CG is superimposed is output to the decoder 210, and the right-eye video frame as a decoding result is output to the right-eye video display unit 203, and the left-eye video frame is output to the left-eye video display unit 204. A report packet for grasping the communication status is output to the communication status grasping unit 309.

The communication status grasping unit 309 grasps the communication status from the contents (for example, communication error information) of the report packet input from the packet reception function 308. Although the report packet is used in this embodiment, the communication status may be grasped from the decoding result in the decoder 210. A specific example of the method for grasping the communication status will be described later with reference to FIG.
The error correction level switching unit 310 switches the level of the error correction encoding function that is the output destination of the right-eye video frame packet buffer 303 and the left-eye video frame packet buffer 304 according to the communication status input from the communication status grasping unit 309. A specific example of the level switching method will be described later with reference to FIG.
The error correction level switching unit 310 corresponds to strength selection means for selecting a strength to be applied from among the error correction strengths set for each of the first video frame packet group and the second video frame packet group. The error correction level switching unit 310 switches the error correction strength applied to the first video frame packet group and the second video frame packet group at a predetermined interval (a period of at least one frame or more).

FIG. 4 is a flowchart showing a flow of error correction level switching processing in the error correction level switching unit 310.
In S401, the communication status is determined. This is a determination process in the communication status grasping unit 309, and details will be described with reference to FIG.
In S402, the process branches according to the communication status determination processing result in S401. When it is determined that the communication state is good, the process proceeds to S403, and when it is determined that the communication state is bad, the process proceeds to S404.

In S403, level 2 error correction strength is applied. In this case, regardless of whether the packet is a right-eye video frame packet or a left-eye video frame packet, the output destination of both packet buffers is the level 2 error correction encoding unit 306, and the error correction level switching process is terminated.
In S404, branching is performed depending on whether the video frame packet is a right-eye video or a left-eye video. If it is a right-eye video, the process proceeds to S405. If it is a left-eye video, the process proceeds to S406.

In S405, the process branches depending on whether the frame number is even or odd. If the frame number is an even number, the process proceeds to S407, and if the frame number is an odd number, the process proceeds to S408.
In S406, as in S405, the process branches depending on whether the frame number is even or odd. If the frame number is an even number, the process proceeds to S408, and if the frame number is an odd number, the process proceeds to S407.

  In S405 and S406, branch processing is performed depending on whether the frame number is an even number or an odd number, but this is merely an implementation example in the present embodiment in which the error correction level is switched in units of one frame. For example, when the error correction level is switched in units of 5 frames, the branch process may be performed depending on whether the frame number / 5 (truncated after the decimal point) is an even number or an odd number.

In S407, the level 3 error correction strength is applied, and the error correction level switching process is terminated.
In S408, the level 1 error correction strength is applied, and the error correction level switching process is terminated.

  The error correction level switching process is performed in units of frames whose levels are switched when it is determined that the communication status is bad. In this embodiment, since switching is performed in units of frames, switching processing occurs each time the frame number of the frame packet output from the right-eye video frame packet buffer 303 and the left-eye video frame packet buffer 304 changes. In this embodiment, switching is performed in units of one frame, but switching may be performed in units of two frames or more. When switching in units of two frames or more, the determination method in S405 and S406 changes. Furthermore, the frame unit to be switched may be dynamically changed according to a change in communication status, a frame rate, or the like.

FIG. 5 is a flowchart showing a flow of communication status determination processing in the communication status grasping unit 309.
In S501, the process branches depending on whether there is an error correction failure. Here, whether or not error correction is possible is determined by whether or not an error correction impossible notification packet has been received from the CG superimposed external PC 105 between the last time this determination process and the present time. If there is no error correction, the process proceeds to S502. If there is no error correction, the process proceeds to S504. The error correction impossibility notification packet is an example of a report packet including recovery possibility information by the error correction packet, and is a packet notifying that the error correction packet cannot be recovered.

In S502, the error correction enable counter that is statically held in the determination process is initialized to 0, and the process proceeds to S503.
In S503, it is determined that the communication status is bad, and the communication status determination processing is terminated.
In S504, the process branches according to the communication status determined when the final determination process is performed. If the previous communication status was good, the process proceeds to S505, and if the previous communication status was bad, the process proceeds to S506.
In S505, it is determined that the communication status is good, and the communication status determination processing is terminated.
In S506, the error correctable counter is incremented, and the process proceeds to S507.

In S507, the process branches depending on whether the error correctable counter is larger than the threshold value N. If the error correctable counter is larger than the threshold N, the process proceeds to S508, and if the error correctable counter is equal to or smaller than the threshold N, the process proceeds to S503. The value of the error correctable counter means the number of times that there was no error correction impossible determination continuously while it was determined that the communication status was bad, and if there was no error correction impossible determination for N + 1 consecutive times, the communication status Will be judged to have improved. The threshold value N may be a predetermined fixed value, or may be dynamically changed according to the communication status, frame rate, and the like.
In S508, the error correctable counter is initialized to 0, and the process proceeds to S505.

  FIG. 6 is a diagram illustrating an application example of error correction when it is determined that the communication status is good. The video frame here is a video frame transmitted from the HMD 200, and the communication error indicates an error that has occurred in the communication path from the HMD 200 to the CG superimposed external PC 105. In order to simplify the description, it is assumed that no error has occurred in the video frame packet sent back from the CG superimposed external PC 105 to the HMD 200. If an error occurs in the video frame packet sent back from the CG superimposed external PC 105 to the HMD 200, the same determination as in the case where there was an error correction failure may be made in S501. When an error correction packet is also added to the video frame packet sent back from the CG superimposed external PC 105, the error correction processing result may be reflected in the error correction impossibility determination in S501.

  There are right-eye and left-eye video frames from frame numbers (1) to (6), and error correction packets of level 2 error correction strength are added respectively. Here, the frames (1), (2), (3), (5), (6) of the right eye image and the frames (1), (3), (4), (5), (6) of the left eye image. , A communication error has occurred, but all can be recovered with an error correction strength of level 2. Therefore, all video frames can be displayed, and it is determined that error correction is impossible in the determination processing of S501, and the processing is continued assuming that there is no change in the communication status.

FIG. 7 is a diagram illustrating an example of application of error correction when the communication status deteriorates from the state illustrated in FIG.
As described above, both the right-eye video frame (7) and the left-eye video frame (7) are video frames transmitted from the HMD 200 to the CG superimposed external PC 105, and the level 2 error correction strength is applied. The left-eye video frame (7) also has 2 communication errors. Therefore, it is determined that the HMD 200 has not received an error correction impossibility notification packet from the CG superimposed external PC 105 between the last communication status determination process and the present time, and step S501 branches to No.

Since it is determined that the communication status is good when the communication status determination process is performed for the frame (7), step S504 branches to No and it is determined that the communication status is good (step S505).
Therefore, step S402 in FIG. 4 branches to No, and the error correction strength of level 2 is applied to the next video frame, that is, the right-eye video frame (8) and the left-eye video frame (8) (step S403).

  Since it is determined that there is no change in the communication status up to the frame (8), an error correction packet with an error correction strength of level 2 is added. In the right-eye video frame (8), the error correction strength of level 2 is An unrecoverable communication error has occurred. Thereby, the error correction impossibility notification packet in the frame (8) is transmitted from the CG superimposed external PC 105 to the HMD 200. The HMD 200 detects a change in the communication status at the timing when the error correction impossible notification packet is received, and changes to an error correction application process when it is determined that the communication status is bad from the next frame (9).

  The timing at which a change in communication status can be detected may be delayed depending on the frame rate, the communication distance between the HMD 200 and the CG superimposed external PC 105, etc., and it is not always possible to change from the next frame (9) to a new error correction application process. In this case, the error correction application process is also changed at the timing when the communication status change can be detected.

  An error correction impossibility notification packet in frame (8) is transmitted to HMD 200, HMD 200 detects a change in communication status, and performs error correction application processing when it is determined from the next frame (9) that the communication status is bad. The case where it changes is demonstrated concretely.

<About frame (9)>
When the HMD 200 receives the error correction impossible notification packet in the frame (8), the step S501 branches to Yes, the error correction possible counter is initialized to 0 (step S502), and it is determined that the communication state is bad (step S503). ).
As a result, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (9), step S404 branches to Yes, step S405 branches to No, and level 1 error correction strength is applied in step S408.
For the left-eye video frame (9), step S404 branches to No, step S406 also branches to No, and level 3 error correction strength is applied in step S407.

In the right-eye video frame (9), since the number of errors is 1 and the error correction strength is level 1, no communication error that cannot be recovered has occurred.
In the left-eye video frame (9), since the number of errors is 2 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.

<About frame (10)>
Since there is no communication error that cannot be recovered in both the right-eye video frame (9) and the left-eye video frame (9), it is determined that there is no error correction, and step S501 in FIG. 5 branches to No.
Since it is determined that the communication status is bad when the communication status determination process is performed for the frame (9), the process branches to Yes in step S504, and the error correction enable counter is incremented in step S506. The value of the error correctable counter is 1.
In this embodiment, the threshold value N in FIG. 5 is set to 2, and if error correction is possible or no error continues for three consecutive frames, the error correction enable counter is initialized to 0 (step S508), and the communication status is recovered. It is determined that it has been done (step S505).
In step S507, since the value of the error correctable counter is 1, the process branches to No, and it is determined in step S503 that the communication status is bad.

Therefore, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (10), step S404 branches to Yes, step S405 also branches to Yes, and the level 3 error correction strength is applied in step S407.
For the left-eye video frame (10), step S404 branches to No, step S406 branches to Yes, and the level 1 error correction strength is applied in step S408.

In the right-eye video frame (10), since the number of errors is 3 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.
In the left-eye video frame (10), since the number of errors is 4 and the error correction strength is level 1, a communication error that cannot be recovered has occurred.

<About frame (11)>
It is determined that there is no error correction, and step S501 in FIG. 5 branches to Yes.
In step S502, the error correctable counter is initialized to 0, and in step S503, it is determined that the communication status is bad.

Therefore, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (11), step S404 branches to Yes, step S405 branches to No, and level 1 error correction strength is applied in step S408.
For the left-eye video frame (11), step S404 branches to No, step S406 also branches to No, and level 3 error correction strength is applied in step S407.

In the right-eye video frame (11), since the number of errors is 3 and the error correction strength is level 1, a communication error that cannot be recovered has occurred.
In the left-eye video frame (11), since the number of errors is 3 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.

<About frame (12)>
It is determined that there is no error correction, and step S501 in FIG. 5 branches to Yes.
In step S502, the error correctable counter is initialized to 0, and in step S503, it is determined that the communication status is bad.

Therefore, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (12), step S404 branches to Yes, step S405 also branches to Yes, and the level 3 error correction strength is applied in step S407.
For the left-eye video frame (12), step S404 branches to No, step S406 branches to Yes, and the level 1 error correction strength is applied in step S408.

In the right-eye video frame (12), since the number of errors is 3 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.
In the left-eye video frame (12), since the number of errors is 3 and the error correction strength is level 1, a communication error that cannot be recovered has occurred.

FIG. 8 is a diagram illustrating an application example of error correction when the communication state is recovered from the state illustrated in FIG.
In the present embodiment, as described above, the threshold value N in FIG. 5 is set to 2, and if error correction is possible or no error continues for three consecutive frames, the error correction enable counter is initialized to 0 (step S508). It is determined that the communication status has been recovered (step S505).

<About frame (13)>
As described above, since an unrecoverable communication error has occurred in the left-eye video frame (12), it is determined that there is no error correction, and step S501 in FIG. 5 branches to Yes.
In step S502, the error correctable counter is initialized to 0, and in step S503, it is determined that the communication status is bad.

Therefore, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (13), step S404 branches to Yes, step S405 branches to No, and level 1 error correction strength is applied in step S408.
For the left-eye video frame (13), step S404 branches to No, step S406 also branches to No, and level 3 error correction strength is applied in step S407.

In the right-eye video frame (13), since the number of errors is 1 and the error correction strength is level 1, no communication error that cannot be recovered has occurred.
In the left-eye video frame (13), since the number of errors is 1 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.

<About frame (14)>
Since there is no communication error that cannot be recovered in both the right-eye video frame (13) and the left-eye video frame (13), it is determined that there is no error correction, and step S501 in FIG. 5 branches to No.
Since it is determined that the communication state (previous communication state) determined when performing the communication state determination process for the frame (13) is bad, step S504 branches to Yes, and the error correction enable counter is incremented in step S506. The value of the error correctable counter is 1.
In step S507, since the value of the error correctable counter is 1, the process branches to No, and it is determined in step S503 that the communication status is bad.

Therefore, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (14), step S404 branches to Yes, step S405 also branches to Yes, and the level 3 error correction strength is applied in step S407.
For the left-eye video frame (14), step S404 branches to No, step S406 branches to Yes, and the level 1 error correction strength is applied in step S408.

In the right-eye video frame (14), since the number of errors is 1 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.
In the left-eye video frame (14), since the number of errors is 0 and the error correction strength is level 1, no communication error that cannot be recovered has occurred.

<About frame (15)>
Since there is no communication error that cannot be recovered in both the right-eye video frame (14) and the left-eye video frame (14), it is determined that there is no error correction, and step S501 in FIG. 5 branches to No.
Since it is determined that the communication state (previous communication state) determined when the communication state determination process of the frame (14) is performed, step S504 branches to Yes, and the error correction enable counter is incremented in step S506. The value of the error correctable counter is 2.
In step S507, since the value of the error correction enable counter is 2, the process branches to No, and in step S503, it is determined that the communication status is bad.

Therefore, step S402 in FIG. 4 branches to Yes.
For the right-eye video frame (15), the level 1 error correction strength is applied, as in the right-eye video frame (13).
For the left-eye video frame (15), the level 3 error correction strength is applied in the same manner as the left-eye video frame (13).

In the right-eye video frame (15), since the number of errors is 0 and the error correction strength is level 1, no unrecoverable communication error has occurred.
In the left-eye video frame (15), since the number of errors is 1 and the error correction strength is level 3, no communication error that cannot be recovered has occurred.

<About frame (16)>
Since there is no communication error that cannot be recovered in both the right-eye video frame (15) and the left-eye video frame (15), it is determined that there is no error correction, and step S501 in FIG. 5 branches to No.
Since it is determined that the communication state (previous communication state) determined when performing the communication state determination process of the frame (15) is bad, step S504 branches to Yes, and the error correction enable counter is incremented in step S506. The value of the error correction enable counter is 3.
In step S507, since the value of the error correctable counter is 3, the process branches to Yes. In step S508, the error correctable counter is initialized to 0. In step S505, it is determined that the communication status is good.

Therefore, step S402 in FIG. 4 branches to No.
In step S403, the level 2 error correction strength is applied to both the right-eye video frame (16) and the left-eye video frame (16).

  As described above, the right-eye and left-eye video frames can be error-corrected or error-free for three consecutive frames (13) to (15). As a result, the HMD 200 detects a change in the communication status at the timing when the error correction enable notification packet of the frame (15) is transmitted from the CG superimposed external PC 105 to the HMD 200. From the subsequent frame (16), an error correction application process when it is determined that the communication status is good, that is, an error correction packet of level 2 error correction strength is generated and transmitted.

In this embodiment, the strength of each error correction level is fixed, but may be changed dynamically according to changes in the communication status.
For example, normally, the correction strength is set to 2 for both the error correction packet for the left-eye video frame packet group and the error correction packet for the right-eye video frame packet group.
When the communication status deteriorates, the correction strength of either the error correction packet group for the left-eye video frame packet group or the error correction packet for the right-eye video frame packet group is set to 1, and the other correction strength is set to 3. .
If the communication status further deteriorates, the correction strength of either the error correction packet group for the left-eye video frame packet group or the error correction packet for the right-eye video frame packet group is set to 2, and the other correction strength is set to 6. May be.

FIG. 9 shows an application example of error correction when the communication situation further deteriorates. In the example shown in the figure, further deterioration of the communication status is detected,
The level 1 error correction strength applied to the right eye video frame (25) is 2, and the level 3 error correction strength applied to the left eye video frame (25) is 6.
The level 3 error correction strength applied to the right-eye video frame (26) is 6, and the level 1 error correction strength applied to the left-eye video frame (26) is 2.

In addition, the option of not applying error correction is also included as an error correction strength option.
For example, in normal times, the right and left error correction strength is 2 (can recover up to 2 errors), and when the communication situation deteriorates, either the left or right error correction strength is 4 (can recover up to 4 errors). The other error correction strength is 0. When the error correction strength is 0, the case where no error correction packet is generated is included.

FIG. 10 shows an example in which error correction is not applied to a video frame packet group corresponding to one of the left eye and the right eye when the communication status deteriorates. In the example shown in the figure, the deterioration of the communication status is detected,
No error correction is applied to the right eye video frame (34), and the error correction strength applied to the left eye video frame (34) is 4,
The error correction strength applied to the right eye video frame (35) is 4, and no error correction is applied to the left eye video frame (34).

<When the strength setting means sets two levels of error correction strength>
When the communication status is good,
An error correction packet (correction strength 2) for the left eye video frame packet group and a left eye video frame packet group, and an error correction packet (correction strength 2) for the right eye video frame packet group and the right eye video frame packet group are generated and transmitted. ,
When the communication situation is bad,
A left-eye video frame packet group and an error correction packet (correction strength 4) for the left-eye video frame packet group are generated and transmitted.
Neither the right-eye video frame packet group nor the error correction packet for the right-eye video frame packet group is generated and may not be transmitted.
In this case, the strength setting means sets the “two” stage error correction strength.

  Further, in the present embodiment, the description has been given of the form in which the error correction packet is generated and transmitted from the HMD 200, but the present invention can also be implemented in the form in which the error correction packet is generated and transmitted in the CG superimposed external PC 105. Further, the error correction packet may be generated and transmitted only from the CG superimposed external PC 105 without generating and transmitting the error correction packet from the HMD 200. In that case, the present invention is applied to the CG superimposed external PC 105. Furthermore, although the wireless MR system has been described as an example in the present embodiment, the present invention can be applied to all use cases in which left and right stereo video data is transmitted via a network.

[Example 2]
In the present embodiment, the video frame data corresponding to the left eye and the right eye is H.264. An example in which the present invention is applied to video frame data compressed by a video encoding method based on inter-frame prediction using disparity information and motion vectors as in H.264 / MVC will be described.
Since the system configuration (FIG. 1), hardware configuration (FIG. 2), and module configuration (FIG. 3) are the same as those in the first embodiment, description thereof will be omitted.
Further, the error correction level switching process (FIG. 4) and the communication status determination process (FIG. 5) are the same as those in the first embodiment, and thus the description thereof is omitted.
The case where the communication state grasping unit 309 determines that the communication state is good (FIG. 6) is the same as that in the first embodiment, and thus the description thereof is omitted.

FIG. 11A is an example of video frame data compressed by a reference video coding method using disparity information and motion vectors. In the figure, I, P, and B mean an I (Intra-coded) frame, a P (Predicted) frame, and a B (Bi-directional Predicted) frame, respectively.
The I frame is encoded using an intra-frame encoding method, and the P frame and the B frame are encoded using an inter-frame encoding method. The B frame is encoded using the correlation between the left and right images. When encoding two videos on the left and right with the inter-frame prediction method as in the MR system, an I frame that does not refer to other frames is included in either the left or right video.

In MPEG (Moving Picture Experts Group), a group of frames starting from an I frame is defined as GOP (Group Of Pictures).
In the present embodiment, the left and right images for inserting the I frame are switched for each GOP. Assuming that 901 is GOP (1), 902 is GOP (2), and 903 is GOP (3), GOP (1) 901 applies the I frame to the right-eye video, and GOP (2) 902 applies the I frame to the left-eye video. Apply, GOP (3) 903 applies the I frame to the right-eye video again.
In this embodiment, the GOP length of each GOP is 4 frames. For this reason, the switching cycle of the left and right images for inserting the I frame is 4 frames, but the switching cycle of the left and right images for inserting the I frame is not limited to 4 frames, and dynamically depends on the communication status and the frame rate. It may be changed.

  FIG. 11B is an example in which error correction is applied when the communication status grasping unit 309 determines that the communication status is bad. When it is determined that the communication status is bad, the error correction level is switched between the left and right images according to the timing, as in the first embodiment. Here, the switching timing is in GOP units. Further, in applying the error correction level, a level having a high error correction strength is applied to a video frame group to which the I frame is applied.

Let 911-919 be GOP (11) -GOP (19). At the start of transmission of GOP (11) to GOP (13), it is determined that the communication status is not bad, and error correction level 2 is applied to both the left and right images for GOP (11) to GOP (13). However, it is assumed that the communication status is determined to be bad before the transmission of GOP (14) 914 starts.
In GOP (14) 914, since an I frame is applied to the right eye video, error correction level 3 is applied to the right eye video frame group, and error correction level 1 is applied to the left eye video frame group.
In the subsequent GOP (15) 915, since the I frame is applied to the left-eye video image, the error correction level 3 is applied to the left-eye video frame group, and the error correction level 1 is applied to the right-eye video frame group.

In GOP (16) 916, since the I frame is applied to the right eye video, the error correction level 3 is applied to the right eye video frame group, and the error correction level 1 is applied to the left eye video frame group.
In GOP (17) 917, since the I frame is applied to the left-eye video, error correction level 3 is applied to the left-eye video frame group, and error correction level 1 is applied to the right-eye video frame group.

  Then, it is determined that the communication status is not bad before the start of transmission of GOP (18) 918. In GOP (18) 918 and GOP (19) 919, error correction level 2 is applied to both the left and right images.

In this embodiment, the communication status grasping process and the error correction application level switching process are processed in units of 1 GOP, but may be processed in units of 2 GOP or more.
Further, as in the first embodiment, the error correction strength and the threshold value for determining the communication status change can be dynamically changed.
In the present embodiment, the frame is not referred to in the future direction on the time axis, but the present invention can also be implemented in a reference form.

[Other Examples]
Although the embodiment has been described in detail above, the present invention can take an embodiment as a system, apparatus, method, program, recording medium (storage medium), or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to a device composed of a single device. good.
The present invention also provides a software program that implements the functions of the above-described embodiments directly or remotely to a system or apparatus, and the system or apparatus computer reads out and executes the supplied program code. Achieved. The program in this case is a computer-readable program that corresponds to the flowchart shown in the drawing in the embodiment.

Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.
In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Recording media for supplying the program include the following media. For example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As a program supply method, the following method is also possible. That is, the browser of the client computer is connected to a homepage on the Internet, and the computer program itself (or a compressed file including an automatic installation function) of the present invention is downloaded to a recording medium such as a hard disk. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to make it. That is, the user can execute the encrypted program by using the key information and install it on the computer.

Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.
Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, and then executed, so that the program of the above-described embodiment can be obtained. Function is realized. That is, based on the instructions of the program, the CPU provided in the function expansion board or function expansion unit can perform part or all of the actual processing.

101 MR system user 102 Wireless router 103 Right-eye imaging frame packet group 104 Left-eye imaging frame packet group 105 CG superimposed external PC
106 CG superimposed right-eye video frame packet group 107 CG superimposed left-eye video frame packet group 200 HMD (head-mounted display device)
211 Data bus 300 Packet transmission / reception program 901 GOP (1)
902 GOP (2)
903 GOP (3)

Claims (13)

A transmitting device that transmits first content data and second content data to a receiving device,
A communication status grasping means for grasping a communication status between the transmitting device and the receiving device;
Based on the communication status, an error correction strength of error correction data used by the receiving device to correct content data that the receiving device has not normally received among the transmitted content data is set to the first content data. Setting means for setting each of the content data and the second content data,
Error correction data generation means for generating first error correction data and second error correction data based on the error correction strength set for each of the first content data and the second content data When,
Transmission means for transmitting the first content data, the second content data, the first error correction data, and the second error correction data to the receiving device. Transmitter device. The communication status grasping means determines whether the communication status satisfies a predetermined condition,
The setting means includes an error larger than the second error correction data in the first error correction data when the communication condition grasping means determines that the communication condition satisfies a predetermined condition. 2. The correction strength is set, and the error correction strength is set to be switched between the first error correction data and the second error correction data at predetermined intervals. The transmitting device according to 1. The transmitting means transmits each of the first content data and the second content data divided into packets,
The setting means sets the error correction strength to switch between the first error correction data and the second error correction data for each of the divided packets. 2. The transmission device according to 2.   The transmission apparatus according to claim 1, further comprising encoding means for encoding each of the first content data and the second content data. Report packet receiving means for receiving a report packet informing the communication status from the receiving device, further comprising:
The transmission apparatus according to claim 1, wherein the communication status grasping unit grasps a communication status from the content of the received report packet.   The transmission device according to claim 5, wherein the report packet includes communication error information.   The transmission device according to claim 5 or 6, wherein the report packet includes recovery possibility information based on the error correction data. Receiving means for receiving content data from the receiving device;
The transmission apparatus according to claim 1, wherein the communication status grasping unit grasps a communication status from a result of reception by the receiving unit.   The setting unit sets the same error correction strength for each of the first content data and the second content data when the communication status determining unit determines that the communication status does not satisfy a predetermined condition. The transmission apparatus according to claim 1, wherein:   Either the transmitting device or the receiving device is a head-mounted display device, and each of the first content data and the second content data is a user wearing the head-mounted display device. The transmission apparatus according to claim 1, wherein the transmission apparatus is video data captured corresponding to left and right eyes.   The transmission apparatus according to claim 1, wherein the content data is data including CG. A transmission method of transmitting first content data and second content data from a transmission device to a reception device,
A communication status grasping step for grasping a communication status between the transmitting device and the receiving device;
Based on the communication status, an error correction strength of error correction data used by the receiving device to correct content data that the receiving device has not normally received among the transmitted content data is set to the first content data. Setting for each of the content data and the second content data,
Error correction data generation step for generating first error correction data and second error correction data based on the error correction strength set for each of the first content data and the second content data When,
A transmission step of transmitting the first content data, the second content data, the first error correction data, and the second error correction data to the receiving device. Transmission method. The program for functioning a computer as each means which the transmission apparatus of any one of Claims 1 thru | or 11 has.

Images