JP4582329B2 - Virtual video signal generation method and transmission / reception terminal - Google Patents

Description

  The present invention relates to a video transmission / reception method and a transmission / reception terminal that perform transmission / reception of video, and in particular, detects a direction and intensity of vibration with respect to a direction in which an image is captured by a sensor that detects vibration, along with video data in a mobile terminal on a reception side. The present invention relates to a video transmission / reception method and a transmission / reception terminal in which vibration information is transmitted and a mobile terminal on the receiving side vibrates a vibrating body in accordance with the video based on the information.

  In a conventional portable terminal, video and audio have been transmitted by a TV phone function or the like. The implications of transmitting video and audio are to inform the user of the receiving terminal of the situation where the transmitting mobile terminal is placed, but with the video and audio alone, the transmitting side shoots the video in any situation. It was not enough for the user of the receiving terminal to experience it.

  In movies and the like, it is common to add vibration to increase the presence of video and audio. A configuration shown in FIG. 3 is an example of a conventional transmission / reception terminal that can reproduce the situation by applying vibration to video and audio.

  FIG. 5 is a block diagram showing a configuration of a conventional transmission / reception terminal capable of reproducing a vibration state, and a transmission side terminal 1 for capturing a video and a communication path 2 for transmitting a video captured by the transmission side terminal 1. And the receiving side terminal 3 that receives and reproduces the transmitted video.

  The transmission side terminal 1 includes a camera 11 and a microphone 12, a video signal encoding unit 13, a vibration sensor 14, a vibration detection unit 15, and a data transmission unit 16.

  The camera 11 captures an image, the microphone 12 records audio, and transmits each data to the video signal encoding unit 13. The video signal encoding unit 13 encodes the image data captured by the camera 11 and the audio data recorded by the microphone 12 in association with each other to generate video data.

  The vibration sensor 14 measures the direction and strength of vibration with respect to the shooting direction in the transmission side terminal 1, and the vibration detection unit 15 generates vibration data indicating the direction and strength of vibration measured by the vibration sensor 14.

  The data transmission unit 16 adds the vibration data from the vibration detection unit 15 to the video data from the video signal encoding unit 13, converts the video data into a signal format corresponding to the communication channel 2, and transmits the data to the communication channel 2.

  The receiving side terminal 3 includes a data receiving unit 31, a video signal decoding unit 32, a display device 33, a speaker 34, a vibration reproduction unit 35, and a vibrating body 36.

  Of the signals received from the transmission path 2, the data reception unit 31 transmits video data to the video signal decoding unit 32 and transmits vibration data to the vibration reproduction unit 35.

  The video signal decoding unit 32 performs decoding based on the video data acquired from the data receiving unit 31, sends image data to the display device 33, and the display device 33 displays the corresponding image. At the same time, the video signal decoding unit 32 sends audio data to the speaker 34, and the speaker 34 reproduces the corresponding audio.

  The vibration reproducing unit 35 transmits control data based on the direction and magnitude of vibration based on the vibration data acquired from the data receiving unit 31 to the vibrating body 36, and the vibrating body 36 performs corresponding vibration.

  Next, the overall operation of the conventional example will be described in detail with reference to the flowchart of FIG. 6 showing the operation of the conventional example.

  First, in the transmission side terminal 1, an image at a certain time is taken by the camera 11, and sound is recorded by the microphone 12 (step A01). The captured image and the recorded audio are encoded by the video signal encoding unit 13 (step A02). On the other hand, the vibration data of the terminal at the same point is measured by the vibration sensor 14 (step A03). The vibration information measured by the vibration sensor is converted into data as the direction and strength of the vibration with respect to the direction in which the image is captured by the vibration detector 15 (step A04).

  These video data and vibration data at a certain point of time are combined by the data transmission unit 16, converted into a data format corresponding to the communication path 2, and transmitted (step A06).

  The transmitted data is received by the data receiver 31 of the receiving terminal 3 via the communication path 2. Since the received data is combined with the video information and the vibration information, first, the data reception unit 31 separates the received data into the video information and the vibration information (steps A07 and 08).

  Of the separated data, the video information is decoded by the video signal decoding unit 32 (step A09), the image is displayed on the display device 33, and the sound is reproduced by the speaker 34 (step A10).

  The vibration data is calculated by the vibration reproduction unit 35 to reproduce the direction and magnitude of the vibration, and the vibration synchronized with the image is reproduced by sending a control signal to the vibrating body 36 to vibrate.

  In the conventional example configured and operated as described above, the vibration information detected using the vibration sensor attached to the transmission side terminal is added to the video information and transmitted, and the reception side terminal receives the received data. By reproducing the decoded video and vibrating the vibrating body based on the vibration information, it is possible to reproduce the video with vibration.

  In addition, although the transmission side terminal 1 and the reception side terminal 3 described above have been described as separate units, these are actually provided in the same casing, and the video and audio by the signal sent from the counterpart terminal are received. The video imaged by the own terminal, the recorded voice, and the signal indicating the vibration status of the own terminal are sent to the partner terminal while watching and hearing and experiencing the vibration.

  If the signal detected by the vibration sensor is sent to the partner terminal as it is as a signal indicating the vibration status of the terminal itself, it will include vibration due to the signal sent from the partner terminal, and it will be accurate as the vibration status of the terminal itself. It will not be.

As a conventional example for obtaining an accurate signal indicating the vibration status of the terminal itself, there is one disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2000-49956). Patent Document 1 discloses that a signal indicating the vibration status of the terminal is obtained by subtracting a signal indicating vibration transmitted from the counterpart terminal from the signal detected by the vibration sensor of the terminal.
JP 2000-49956 A

  Among the conventional technologies described above, when the signal detected by the vibration sensor is sent as it is to the counterpart terminal as a signal indicating the vibration status of the own terminal, it does not indicate an accurate one as the vibration status of the own terminal. There is a problem.

  In the invention disclosed in Patent Document 1, an accurate vibration state of the terminal is shown. However, there is a problem in that the shot video becomes a video shot in a situation where vibration is caused by a signal sent from the partner terminal, and does not accurately indicate the situation where the terminal is placed. There is.

  The present invention has been made in view of the problems of the prior art as described above, and it is possible to generate a video signal for virtual video that can accurately reproduce video and vibrations that accurately indicate the situation where the terminal is placed. The object is to realize a method and a transmitting and receiving terminal.

A transmission / reception terminal according to the present invention includes a camera, a display device, a vibration sensor that detects vibration, and a vibrating body that vibrates according to vibration data indicating vibration, and video data based on an image captured by the camera, and Transmitting vibration data based on vibration detected by the vibration sensor, the display device displays a video based on the received video data, and the vibrator is a transmission / reception terminal that vibrates according to the received vibration data,
An image correction unit that corrects an image captured by the camera using the received vibration data in order to remove blurring due to vibration of the vibrating body in the image captured by the camera.

  In this case, the vibration correction for correcting the detected vibration data indicating the vibration detected by the vibration sensor using the received vibration data in order to remove the influence of the vibration of the vibrating body on the vibration detected by the vibration sensor. It may have a part.

A virtual video signal generation method according to the present invention includes a camera, a display device, a vibration sensor that detects vibration, and a vibrating body that vibrates according to vibration data indicating vibration, and an image captured by the camera. And transmitting and receiving vibration data based on the vibration detected by the vibration sensor, the display device displays an image based on the received video data, and the vibrator is a transmission / reception terminal that vibrates according to the received vibration data. A virtual video signal generation method performed,
Correcting the image captured by the camera using the received vibration data in order to remove blur due to vibration of the vibrating body in the image captured by the camera;
Correcting the detected vibration data indicating the vibration detected by the vibration sensor using the received vibration data in order to remove the influence of the vibration of the vibrating body on the vibration detected by the vibration sensor;
It is characterized by having.

  In the present invention configured as described above, an image taken in a state where vibration is performed by the vibration signal from the counterpart terminal, and the influence of the vibration due to the vibration signal from the counterpart terminal from the detected vibration. Is removed and used as a video signal or a vibration signal to the destination terminal, so that it is possible to obtain a video signal and a vibration signal that accurately indicate the situation where the terminal is placed.

  Next, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention. The transmission / reception terminal 104 shown in FIG. 1 performs transmission / reception with another transmission / reception terminal (not shown) via the communication path 103. The transmission block 101 having the function of the transmission-side terminal 1 shown in FIG. The receiving block 102 is provided with the function of the terminal 3.

  The transmission block 101 includes a camera 111, a microphone 112, a video signal encoding unit 113, a vibration sensor 114, a vibration detection unit 115, a data transmission unit 116, an image correction unit 117, and a vibration correction unit 118.

  The reception block 103 includes a data reception unit 131, a video signal decoding unit 132, a display device 133, a speaker 134, a vibration reproduction unit 135, and a vibrating body 136.

  The operation of the present embodiment will be described first from the operation of the reception block 103.

  Of the signals received from the communication path 103, the data reception unit 131 transmits video data to the video signal decoding unit 132 and transmits vibration data to the vibration reproduction unit 135.

  The video signal decoding unit 132 performs decoding based on the video data acquired from the data receiving unit 131, sends the image data to the display device 133, and the display device 133 displays the corresponding image. At the same time, the video signal decoding unit 132 sends audio data to the speaker 134, and the speaker 134 reproduces the corresponding audio.

  The vibration reproducing unit 135 transmits control data based on the direction and magnitude of vibration based on the vibration data acquired from the data receiving unit 131 to the vibrating body 136, and the vibrating body 136 performs corresponding vibration.

  Next, the operation of the transmission block 101 will be described.

  The camera 111 captures an image and transmits the video data to the image correction unit 117. The microphone 112 records audio and transmits the audio data to the video signal encoding unit 113.

  The above video data is taken under a situation where the vibrating body 136 is vibrating due to vibration data from the counterpart terminal, and includes vibration due to vibration.

  The image correction unit 117 is provided to remove image blur due to the vibrating body 136. In addition to video data, the image correction unit 117 receives vibration data from the destination terminal from the data reception unit 131. It is coming. The image correcting unit 117 corrects the influence of image blurring or the like caused by the vibration of the vibrating body 136 on the image captured by the camera 111 based on the vibration data from the data receiving unit 131, and the video signal The data is transmitted to the encoding unit 113. As a technique for correcting the influence of image blurring or the like performed here, a motion of the camera 111 is predicted based on vibration data, and the entire captured image is moved so that the blurring of the captured image due to the motion is canceled. Various methods used for camera shake correction of an electronic camera can be used, but the present invention is not limited to this.

  The video signal encoding unit 113 encodes the image data photographed by the camera 111 and corrected by the image correction unit 117 and the audio data recorded by the microphone 12 in association with each other, and the data transmission unit 116 as video data. To send.

  The vibration sensor 114 measures the direction and strength of vibration with respect to the shooting direction in the transmission / reception terminal 104 and sends the vibration direction and strength to the vibration detection unit 115. The vibration detection unit 115 vibrates indicating the direction and strength of vibration measured by the vibration sensor 114. Data is generated and sent to the vibration correction unit 118.

  The above vibration data is measured under the condition that the vibration body 136 is vibrating by vibration data from the counterpart terminal, and includes vibrations by the vibration body 136.

  Similarly, when the vibration body 136 of the own terminal vibrates corresponding to the data received from the counterpart terminal, the vibration correction unit 118 adds the vibration data measured by the vibration sensor 114 to the terminal externally. Since the vibration of the vibrating body 136 is added to the vibration, it is provided to remove this vibration. In addition to the video data, vibration data from the counterpart terminal is sent from the data receiving unit 131 to the vibration correcting unit 118.

  The vibration correcting unit 118 removes the vibration data obtained by removing the data added by the vibration of the vibrating body 136 based on the vibration data transmitted from the data receiving unit 131 with respect to the vibration information measured by the vibration detecting unit 115. The data is transmitted to the data transmission unit 116. Thereby, for example, when only the vibrating body 136 is vibrating while the transmission / reception terminal 104 is not externally vibrated, the vibration data sent from the vibration correction unit 118 to the data transmission unit 116 is not moving. (0) indicating

  The data transmission unit 116 adds the vibration data from the vibration correction unit 118 to the video data from the video signal encoding unit 113, converts the video data into a signal format corresponding to the communication channel 103, and transmits it to the communication channel 103.

  Next, the overall operation of this embodiment will be described in detail with reference to the flowchart of FIG. 2 showing the operation of this embodiment.

  First, in the transmission side terminal 101, an image at a certain time is taken by the camera 111, and sound is recorded by the microphone 112 (step A101). The captured image is converted into an image from which vibration due to vibration data sent from the counterpart terminal is removed by the image correction unit 117, and is encoded by the video signal encoding unit 113 together with the recorded audio (step). A102).

  On the other hand, the vibration data of the terminal at the same point is measured by the vibration sensor 114 (step A103). The vibration information measured by the vibration sensor 114 is converted into data as the direction and strength of the vibration with respect to the direction imaged by the vibration detection unit 115, and the vibration data sent from the counterpart terminal by the vibration correction unit 118 is converted into data. It is removed (step A104).

  These video data and vibration data at a certain point in time are combined by the data transmission unit 116, converted into a data format corresponding to the communication path 103, and transmitted (step A106).

  The transmitted data is received by the data receiving unit 131 of the receiving terminal 103 via the communication path 103. Since the received data is combined with the video information and the vibration information, the data reception unit 131 first separates the received data into the video information and the vibration information (steps A107 and 108).

  Of the separated data, the video information is decoded by the video signal decoding unit 32 (step A109), the image is displayed on the display device 133, and the sound is reproduced by the speaker 134 (step A110).

  The vibration data is calculated by the vibration reproduction unit 135 to reproduce the direction and magnitude of the vibration, and a vibration is synchronized with the image by sending a control signal to the vibrating body 136 to vibrate.

Next, when the transmission / reception terminal 104 described using a specific example is a mobile phone, for example, the communication path 103 is a mobile phone network.

  The camera 111 captures an image at a certain point in time, and the sound is recorded by the microphone 112. This data is encoded by the video signal encoding unit 113 using MPEG4, AMR, or the like. This image is not affected by the vibration of the vibrating body 136 due to the vibration data sent from the counterpart terminal.

  On the other hand, at the same time, the vibration sensor 114 attached to the transmission-side mobile phone measures the direction and strength of vibration relative to the shooting direction, and is converted into data by the vibration detection unit 115. This data is not affected by the vibration of the vibrating body 136 due to the vibration data sent from the counterpart terminal.

  The format of the data is, for example, the measurement of acceleration in each direction in the xyz coordinate system with the shooting direction as the xy coordinate and the vertical direction of the shooting as the z coordinate. When the state shown in FIG. , Y, z) = (2, 1, 1) <unit is represented in the form of G>.

  In the data transmission unit 116, video data and vibration data at a certain point in time are combined, further converted into a radio signal, and transmitted to a mobile phone network (communication path 103) which is a transmission path.

  The transmitted wireless signal is received by the data receiving unit 131 of the receiving-side mobile phone (receiving-side terminal 103) and separated into video information and vibration information, respectively.

  The video information is decoded by the video signal decoding unit using MPEG4, AMR, or the like, the image is displayed on the display device 133 such as an LCD, and the sound is reproduced from the speaker 134 such as an external speaker.

  The vibration information is calculated so as to reproduce the situation on the receiving side by the vibration reproducing unit 135. For example, when information as shown in FIG. 3 is transmitted, the receiving terminal generates vibration as shown in FIG. The parameter of the control signal for generating the vibration is calculated, and the control signal is sent to the vibrating body 136 so that the vibration synchronized with the image is reproduced.

  In the conventional example configured and operated as described above, the vibration information detected using the vibration sensor attached to the transmission side terminal is added to the video information and transmitted, and the reception side terminal receives the received data. By reproducing the decoded video and vibrating the vibrating body based on the vibration information, it is possible to reproduce the video with vibration.

  As a result, for example, a user of a receiving terminal that has received a video shot in a car that is turning a curve at high speed can experience vibration that is turning a curve at high speed, similar to the transmission side. it can.

  In addition, the vibration information is transmitted not only by the presence or absence of vibration, but also by transmitting the vibration direction and strength with respect to the imaged direction and controlling the vibrating body to reproduce it on the receiving side (vibration ( For example, when it is detected that the transmitting side vibrates upward, the receiving side vibration body vibrates upward), so that the user of the receiving side terminal can experience more faithful vibration.

  Since the video data and vibration data in this embodiment are not affected by the vibration caused by the vibration data sent from the counterpart terminal, the video data to which the vibration information of the own terminal transmitted to the counterpart side is added. And vibration data can be accurate.

It is a block diagram which shows the structure of one Example of this invention. It is a flowchart which shows operation | movement of the Example shown in FIG. It is a figure for demonstrating the signal which shows the vibration used by this invention. It is a figure for demonstrating the signal which shows the vibration used by this invention. It is a block diagram which shows the structure of a prior art example. It is a flowchart which shows operation | movement of the prior art example shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Transmission block 102 Reception block 103 Communication path 104 Transmission / reception terminal 111 Camera 112 Microphone 113 Video signal encoding part 114 Vibration sensor 115 Vibration detection part 116 Data transmission / reception part 117 Image correction part 118 Vibration correction part 131 Data transmission / reception part 132 Video signal decoding 133 Display device 134 Speaker 135 Vibration reproduction unit 136 Vibrating body

Claims (3)

A camera, a display device, a vibration sensor that detects vibration, and a vibrating body that vibrates according to vibration data indicating vibration, and is detected by the image data based on an image captured by the camera and the vibration sensor Vibration data based on vibration is transmitted, the display device displays a video based on the received video data, and the vibrator is a transmission / reception terminal that vibrates according to the received vibration data,
A transmission / reception terminal comprising: an image correction unit that corrects an image captured by the camera using the received vibration data in order to remove blurring due to vibration of the vibrating body in an image captured by the camera. . The transmission / reception terminal according to claim 1,
A vibration correction unit configured to correct the detected vibration data indicating the vibration detected by the vibration sensor using the received vibration data in order to remove the influence of the vibration of the vibrating body on the vibration detected by the vibration sensor; A transmission / reception terminal characterized by that. A camera, a display device, a vibration sensor that detects vibration, and a vibrating body that vibrates according to vibration data indicating vibration, and is detected by the image data based on an image captured by the camera and the vibration sensor A method for generating a virtual video signal performed by a transmission / reception terminal that transmits vibration data based on vibration, the display device displays a video based on the received video data, and the vibrator is vibrated by the received vibration data. ,
Correcting the image captured by the camera using the received vibration data in order to remove blur due to vibration of the vibrating body in the image captured by the camera;
Correcting the detected vibration data indicating the vibration detected by the vibration sensor using the received vibration data in order to remove the influence of the vibration of the vibrating body on the vibration detected by the vibration sensor;
A method for generating a signal for virtual video, comprising:

Images