WO2019135289A1 - Information communication system and information communication method - Google Patents
Information communication system and information communication method Download PDFInfo
- Publication number
- WO2019135289A1 WO2019135289A1 PCT/JP2018/000135 JP2018000135W WO2019135289A1 WO 2019135289 A1 WO2019135289 A1 WO 2019135289A1 JP 2018000135 W JP2018000135 W JP 2018000135W WO 2019135289 A1 WO2019135289 A1 WO 2019135289A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information processing
- processing apparatus
- data
- information
- communication system
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
Definitions
- the present invention relates to an information communication system and an information communication method, and more particularly to data communication using an omnidirectional camera.
- an omnidirectional camera cameras capable of capturing an image in the omnidirectional direction (that is, in the range of 360 degrees) (hereinafter referred to as an omnidirectional camera) have appeared and are mounted on various devices.
- Patent Document 1 discloses a technique for raising the image quality and improving the visibility of an image.
- an object of the present invention is to provide a technology for performing real-time information communication at high speed using an omnidirectional camera.
- the inventor of the present invention has attempted to speed up communication by a three-step method. That is, the inventors have been able to transmit data for transmitting video data of the omnidirectional camera with low delay, data reduction technology for transmitting data of the video of the omnidirectional camera with low capacity, and We have invented a technology that restores and renders the video data of an astronomical camera as an all-astronomical image with low delay, and further realizes real-time information communication at high speed by combining these technologies alone or in combination.
- the present invention is based on such findings.
- a first information processing apparatus including an omnidirectional camera, and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD)
- An information communication system including an apparatus;
- the omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
- the first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
- the second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
- An information communication system is obtained.
- FIG. 7 is a diagram illustrating another process of the system of FIG. 1;
- FIG. 7 is a diagram illustrating another process of the system of FIG. 1;
- An information communication system and an information communication method have the following configuration.
- An information communication system including: a first information processing apparatus including an omnidirectional camera; and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD).
- the omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
- the first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
- the second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
- Information communication system including: a first information processing apparatus including an omnidirectional camera; and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD).
- HMD head mounted display
- the display unit is a head mounted display (HMD), and displays visual field information in a virtual space using a pseudo sphere object and a virtual camera disposed at the center of the pseudo sphere object.
- the second information processing apparatus configures the target space in the virtual space by arranging the texture data inside the pseudo sphere object.
- Information communication system [Item 3] An information communication system according to item 2;
- the second information processing apparatus acquires inclination information of the HMD, and transmits the acquired information to the second information processing apparatus.
- the first information processing apparatus converts partial video data corresponding to the tilt information in the video data into partial binary data and transmits the partial binary data to the second information processing apparatus.
- Information communication system is a head mounted display (HMD), and displays visual field information in a virtual space using a pseudo sphere object and a virtual camera disposed at the center of the pseudo sphere object.
- the second information processing apparatus configures the target space in the virtual space by arranging the texture data inside the pseudo sphere object.
- Information communication system [Item 3]
- UDP User Datagram Protocol
- Information communication using a first information processing apparatus including an omnidirectional camera and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD) Method The omnidirectional camera acquiring video data of a target space and transmitting the video data to the first information processing apparatus; The first information processing apparatus converts the video data into binary data that can be restored according to a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus; The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
- HMD head mounted display
- An information communication system is a technology for synchronizing an image acquired by an omnidirectional camera with ultra-low delay and stable synchronization via a wireless Internet circuit (Wifi / Mobile network).
- the information communication system can be mainly culture in the following three configurations. That is, data transfer technology for transmitting video data of the omnidirectional camera with low delay, data amount reduction technology for transmitting video data of the omnidirectional camera with low capacity, and It is a technology that restores and renders video data as an omnidirectional video with low delay.
- FIG. 1 is a diagram showing the configuration of an information communication system according to an embodiment of the present invention. Note that this is an example, and other elements may be added.
- the information communication system 1 can be divided into two components of a remote device for remote control by a user at a remote location and an operation device for user control.
- the operation device and the remote device are connected via a network (described later).
- the remote device includes an omnidirectional camera, a first information processing terminal (PC1), and a body.
- the omnidirectional camera is an imaging device capable of taking 360-degree panoramic photographs of all directions in the vertical and horizontal directions, and may also be called an omnidirectional camera.
- the body is a device capable of expanding a part of the user's body, and can be exemplified by, for example, a device such as a robot arm, but is not limited thereto.
- the operation device includes a second information processing terminal (PC2) and a head mount display (HMD).
- PC2 second information processing terminal
- HMD head mount display
- FIG. 2 is a block diagram showing the first information processing terminal 100.
- the first information processing terminal 100 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14 and the like, which are electrically connected to one another through a bus 15. Ru.
- the input / output unit is further communicably connected to the omnidirectional camera 130, the body control unit 132, and the body 134.
- the first information processing terminal 100 is implemented in a remote device, it is preferable to employ a relatively small device.
- the illustrated functional blocks may be omitted as appropriate.
- the processor 10 is an arithmetic device that controls the overall operation of the first information processing apparatus 100, controls transmission and reception of data between elements, and performs information processing and the like necessary for executing an application.
- the processor 10 is a CPU (Central Processing Unit), and executes a program or the like stored in the storage 12 and expanded in the memory 11 to perform each information processing.
- CPU Central Processing Unit
- the memory 11 includes a main storage constituted by a volatile storage device such as a dynamic random access memory (DRAM), and an auxiliary storage constituted by a nonvolatile storage device such as a flash memory or a hard disc drive (HDD). .
- the memory 11 is used as a work area or the like of the processor 10, and stores a BIOS (Basic Input / Output System) executed when the computer terminal 120 is started, various setting information, and the like.
- BIOS Basic Input / Output System
- the storage 12 stores various programs such as application programs.
- a database storing data used for each process may be constructed in the storage 22.
- the transmission / reception unit 13 connects the computer terminal 120 to a network.
- the transmission / reception unit 13 may include a short range communication interface such as Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).
- an information input device such as a keyboard or a mouse and an output device such as a display may be connected to the input / output unit 14.
- the bus 15 is commonly connected to the above-described elements, and transmits, for example, an address signal, a data signal, and various control signals.
- FIG. 3 is a block diagram showing the second information processing terminal 200.
- the second information processing terminal 200 includes at least a processor 20, a memory 21, a storage 22, a transmitting / receiving unit 23, an input / output unit 24 and the like, which are electrically connected to each other through a bus 25. Ru.
- the input / output unit 24 is communicably connected to the operation unit 242 and the HMD 244.
- the HMD 244 has a display and a sensor not shown.
- the display is a non-transmissive display device configured to completely cover the user's field of view, and the user can view only the screen displayed on the display. Then, since the user wearing the non-transmissive HMD 244 loses all the visibility of the outside world, the display mode completely immerses in the virtual space displayed by the executed application.
- the user can configure the omnidirectional image acquired by the omnidirectional camera 130 (see FIG. 2) in the HMD 244 to give the experience as if the view was synchronized with the remote device.
- a sensor provided in the HMD 244 is fixed near the display.
- the sensor includes a geomagnetic sensor, an acceleration sensor, and / or a tilt (angular velocity, gyro) sensor, and can detect various movements of the HMD 244 mounted on the user's head through one or more of these.
- angular velocity sensor in particular, according to the movement of the HMD 244, the angular velocity around three axes of the HMD 244 can be detected over time, and the time change of the angle (tilt) around each axis can be determined.
- the operation unit 242 has, for example, a glove type that can be worn on the user's hand, and has a mechanism that can limit the amount of movement of each finger.
- the operation unit 242 includes a motion detection unit that detects the movement of the user's finger, and a tactile sense presentation unit that transmits the tactile signal generated by the second information processing apparatus 200 to the user's finger.
- FIG. 4 is a diagram showing the flow of information processing performed among the omnidirectional camera 130, the first information processing apparatus 100, the second information processing apparatus 200, and the HMD 244 regarding the processing of the information communication system according to the present embodiment. It is.
- the omnidirectional camera 130 acquires video data.
- the acquired video data is transmitted to the first information processing apparatus 100 (SQ101).
- the first information processing apparatus 100 converts the received video data into predetermined binary data (SQ102).
- the first information processing apparatus 100 transmits binary data to the second information processing apparatus 200 (SQ 103).
- binary data is transmitted to the second information processing apparatus 200 using User Datagram Protocol (UDP).
- UDP User Datagram Protocol
- Communication between the first information processing apparatus 100 and the second information processing apparatus 200 in the present embodiment uses a unique protocol based on UDP.
- a unique protocol based on UDP.
- RTP guarantee of data order
- the second information processing apparatus 200 converts the received binary data into a predetermined texture for each frame of the moving image (SQ 104).
- the converted texture data is transmitted to the HMD 244 (SQ 105).
- the virtual space output and displayed on the HMD 244 according to the present embodiment is, as shown in FIG. 5, a virtual camera in which the visual field information in the virtual space is arranged at the center of the pseudo sphere object V and the pseudo sphere object V. It is displayed using Vc.
- the second information processing apparatus 200 configures a virtual space by pasting texture data inside the pseudo sphere object V.
- the information communication technology according to the present invention further includes a data amount reduction technology for transmitting image data of the omnidirectional camera with a low capacity, and an omnidirectional camera with low latency data of the image of the omnidirectional camera. It also provides technology to restore and render as video.
- FIG. 6 is a diagram showing the process flow of a further embodiment of the information communication system according to the present invention.
- the same processing as that of the embodiment described above is denoted by the same reference numeral to simplify the description.
- the video data acquired by the omnidirectional camera is transmitted to the first information processing apparatus 100 (SQ 101).
- the tilt information of the HMD 244 (i.e., the tilt of the head of the user: the viewing direction) is further used.
- the HMD 244 detects tilt information (SQ601), and the tilt information is transmitted to the second information processing apparatus 200 (SQ602).
- the tilt information is further transmitted to the first information processing apparatus 100 (SQ 603).
- the first information processing apparatus generates partial binary data (partial binary data) for video data corresponding to tilt information among video data acquired from the omnidirectional camera (SQ 604).
- the video data acquired from the omnidirectional camera is divided into six regions, and partial binary data is generated for the corresponding region according to the tilt information.
- Back side hemisphere Region 4: (X, Y, Z) (0, -1, 0) as a vertex, (0, 0, 1), (0, -1, 0),
- binary data is transmitted to the second information processing apparatus via UDP (SQ 603).
- the second information processing apparatus 200 converts the received partial binary data into a predetermined texture for each frame of a moving image (SQ 104).
- the converted texture data is transmitted to the HMD 244 (SQ 105).
- the second information processing apparatus 200 constructs a virtual space by attaching the texture data converted from the partial binary data to a part of the inside of the pseudo sphere object V.
- the virtual space output and displayed on the HMD 244 an image converted from partial binary data corresponding to the above-described six areas is displayed (SQ106).
- the video data from the omnidirectional camera only video data of only the area viewed by the user among the above six areas is to be transmitted based on the inclination of the HMD. It becomes. Accordingly, the amount of data to be transmitted can be reduced.
- ⁇ Differential binary data> The technology according to the embodiment to be described subsequently is intended to transmit only the difference of video data, and it is possible to realize further reduction of transmission data.
- FIG. 7 is a diagram showing the flow of such processing. Also in FIG. 7, the same processing as that of the above-described embodiment is denoted by the same reference numeral, and the description thereof will be simplified.
- the video data acquired by the omnidirectional camera is transmitted to the first information processing apparatus 100 (SQ 101).
- the tilt information of the HMD 244 (that is, the tilt of the head of the user: the direction of the line of sight) is used further, the tilt information is detected by the HMD 244 (SQ601), and the second information processing apparatus 200 is used. Transmission to the first information processing apparatus 100 (SQ 602, SQ 603).
- the first information processing apparatus 100 detects and specifies a portion (difference area) having a change in the video based on the change of the RGB data value in the above-described video data (SQ 701).
- the first information processing apparatus 100 generates binary data (difference binary data) only for the difference area (SQ 702), and transmits the binary data to the second information processing apparatus 200 via UDP (SQ 703).
- the second information processing apparatus 200 updates only the part corresponding to the difference area in the texture data (SQ 704), and transmits the updated data to the HMD (SQ 705).
- the difference may be detected after processing for removing noise that enters video data (that is, an image for each frame).
- a plurality of points (noises) of a color different from that of a normal object or landscape may be included in the image for each frame. It can happen. At this time, if it is determined that the noise (that is, equivalent to the above-described “portion with change”) is a difference region to be transmitted, unnecessary difference processing data is generated.
- the following three steps are performed to remove the noise. That is, (1) Binarize the image for each frame constituting the video (process to convert the image into two gradations of white and black) (2) enlarge the image (dpi) by N times (for example, twice) (3) Reduce the magnified image (dpi) to M times (M> N: 3 times, for example)
- the form of connecting each element is not limited to this. It may be in a form of being connected by a wireless network, or it may be in a form in which the information processing apparatus is included in the HMD or the operation unit. Alternatively, the computing process may be performed by cloud computing.
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- User Interface Of Digital Computer (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
[Problem] To provide a technology for performing real-time information communication at high speed by using an entire celestial sphere camera. [Solution] The present invention is an information communication system including a first information processing device provided with an entire celestial sphere camera, and a second information processing device that is communicably connected to the first information processing device and that is provided with a head mounted display (HMD). The entire celestial sphere camera acquires video image data of a target space and transmits the data to the first information processing device. The first information processing device converts the video image data to binary data that can be restored for each frame by a predetermined video image format, and transmits the binary data to the second information processing device. The second information processing device converts the binary data to predetermined texture data and displays the texture data on a display unit.
Description
本発明は、情報通信システム及び情報通信方法に関し、特に全天球カメラを利用したデータ通信に関する。
The present invention relates to an information communication system and an information communication method, and more particularly to data communication using an omnidirectional camera.
近年、全天球方向（つまり、３６０度の範囲）の映像を撮像可能なカメラ（以下、全天球カメラという）が登場し、様々なデバイスに搭載されている。
In recent years, cameras capable of capturing an image in the omnidirectional direction (that is, in the range of 360 degrees) (hereinafter referred to as an omnidirectional camera) have appeared and are mounted on various devices.
このような全天球カメラにより撮像された映像をヘッドマウント・ディスプレイ（ＨＭＤ）において見る際に、ユーザの視線方向に合わせて動的に映像の明るさを調整し、撮像時のダイナミックレンジを疑似的に上げ、映像の視認性を向上する技術が開示されている（特許文献１）。
When viewing an image captured by such an omnidirectional camera on a head mounted display (HMD), the brightness of the image is dynamically adjusted according to the user's gaze direction, and the dynamic range at the time of imaging is simulated. Patent Document 1 discloses a technique for raising the image quality and improving the visibility of an image.
一方、近年、遠隔地にあるデバイスをリアルタイムで操作・同期させるテレプレゼンス（Ｔｅｌｅｐｒｅｓｅｎｃｅ）技術の研究が進み、遠隔地に存在する操作対象としてのデバイスの「眼」として全天球カメラを適用することも提案されている。
On the other hand, in recent years, research on telepresence (Telepresence) technology for operating and synchronizing devices in remote locations in real time has progressed, and applying the omnidirectional camera as the “eye” of the device as an operation object located in remote locations Is also proposed.
しかしながら、全天球カメラにより得られる情報量は大きいことから、特に、ユーザと遠隔地デバイス（ロボットアーム等）をリアルタイムで同期させることに課題が存在している。
However, since the amount of information obtained by the omnidirectional camera is large, there is a problem, in particular, in real time synchronizing the user and the remote device (robot arm etc.).
そこで、本発明は、全天球カメラを利用したリアルタイム情報通信を高速で行うための技術を提供することを一つの目的とする。
Therefore, an object of the present invention is to provide a technology for performing real-time information communication at high speed using an omnidirectional camera.
上述した目的を達成するために、本発明の発明者は、３段階の方法によって、通信の高速化を図った。即ち、発明者は、全天球カメラの映像のデータを低遅延で送信するためのデータ転送技術、全天球カメラの映像のデータを低容量で送信するためのデータ量削減技術、そして、全天球カメラの映像のデータを低遅延で全天球映像として復元・描画する技術を発明し、更に、これらの技術を単独で又は組み合わせることによって、リアルタイム情報通信を高速で行うことを実現した。本発明はかかる知見に基づくものである。
In order to achieve the above-mentioned purpose, the inventor of the present invention has attempted to speed up communication by a three-step method. That is, the inventors have been able to transmit data for transmitting video data of the omnidirectional camera with low delay, data reduction technology for transmitting data of the video of the omnidirectional camera with low capacity, and We have invented a technology that restores and renders the video data of an astronomical camera as an all-astronomical image with low delay, and further realizes real-time information communication at high speed by combining these technologies alone or in combination. The present invention is based on such findings.
即ち、本発明によれば
全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを含む情報通信システムであって、
前記全天球カメラは、対象空間の映像データを取得して、前記第１情報処理装置に送信し、
前記第１情報処理装置は、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示する、
情報通信システムが得られる。 That is, according to the present invention, a first information processing apparatus including an omnidirectional camera, and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD) An information communication system including an apparatus;
The omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
The first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
An information communication system is obtained.
全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを含む情報通信システムであって、
前記全天球カメラは、対象空間の映像データを取得して、前記第１情報処理装置に送信し、
前記第１情報処理装置は、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示する、
情報通信システムが得られる。 That is, according to the present invention, a first information processing apparatus including an omnidirectional camera, and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD) An information communication system including an apparatus;
The omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
The first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
An information communication system is obtained.
本発明によれば全天球カメラを利用したリアルタイム情報通信を高速で行うためのフィードバック生成方法を提供することができる。
According to the present invention, it is possible to provide a feedback generation method for performing high-speed real-time information communication using an omnidirectional camera.
本発明の実施形態の内容を列記して説明する。本発明の実施の形態による情報通信システム及び情報通信方法は、以下のような構成を備える。
［項目１］
全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを含む情報通信システムであって、
前記全天球カメラは、対象空間の映像データを取得して、前記第１情報処理装置に送信し、
前記第１情報処理装置は、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示する、
情報通信システム。
［項目２］
項目１に記載の情報通信システムであって、
前記表示部は、ヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）であり、仮想空間内における視界情報を疑似球体オブジェクトと当該疑似球体オブジェクトの中心に配置された仮想カメラとを利用して表示するものであり、
前記第２情報処理装置は、前記テクスチャデータを前記疑似球体オブジェクトの内側に配置することにより、前記仮想空間内に前記対象空間を構成する、
情報通信システム。
［項目３］
項目２に記載の情報通信システムであって、
前記第２情報処理装置は、前記ＨＭＤの傾き情報を取得して前記第２情報処理装置に送信し、
前記第１情報処理装置は、前記映像データのうち前記傾き情報に対応する部分映像データについて部分バイナリデータに変換して前記第２情報処理装置に送信する、
情報通信システム。
［項目４］
項目３に記載の情報通信システムであって、
前記第１情報処理装置は、前記部分映像データ内において変化のある部分を検知し、当該変化のある部分のみに対応する差分映像データについて差分バイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記テクスチャデータのうち、前記差分バイナリデータに対応する部分のみを更新する、
情報通信システム。
［項目５］
項目４に記載の情報通信システムであって、
前記第１情報処理装置は、前記部分映像データ内におけるＲＧＢデータ値の変化に基づいて前記検知を行う、
情報通信システム。
［項目６］
項目３乃至項目５のいずれかに記載の情報通信システムであって、
前記第１情報処理装置は、前記映像データを６つの領域に分割し、前記傾き情報に対応する領域の前記部分映像データについて部分バイナリデータに変換して前記第２情報処理装置に送信する、
情報通信システム。
［項目７］
項目１又は項目２に記載の情報通信システムであって、
前記第１情報処理装置は、前記バイナリデータをユーザ・データグラム・プロトコル（Ｕｓｅｒ Ｄａｔａｇｒａｍ Ｐｒｏｔｏｃｏｌ：ＵＤＰ）を利用して、前記第２情報処理装置に送信する、
情報通信システム。
［項目８］
全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを利用した情報通信方法であって、
前記全天球カメラが、対象空間の映像データを取得して、前記第１情報処理装置に送信するステップと、
前記第１情報処理装置が、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信するステップと、
前記第２情報処理装置が、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示するステップと、を含む
情報通信方法。 The contents of the embodiment of the present invention will be listed and described. An information communication system and an information communication method according to an embodiment of the present invention have the following configuration.
[Item 1]
An information communication system including: a first information processing apparatus including an omnidirectional camera; and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD). And
The omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
The first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
Information communication system.
[Item 2]
An information communication system according toitem 1;
The display unit is a head mounted display (HMD), and displays visual field information in a virtual space using a pseudo sphere object and a virtual camera disposed at the center of the pseudo sphere object. And
The second information processing apparatus configures the target space in the virtual space by arranging the texture data inside the pseudo sphere object.
Information communication system.
[Item 3]
An information communication system according to item 2;
The second information processing apparatus acquires inclination information of the HMD, and transmits the acquired information to the second information processing apparatus.
The first information processing apparatus converts partial video data corresponding to the tilt information in the video data into partial binary data and transmits the partial binary data to the second information processing apparatus.
Information communication system.
[Item 4]
An information communication system according to item 3;
The first information processing apparatus detects a portion having a change in the partial video data, converts difference image data corresponding to only the portion having the change into difference binary data, and transmits the difference to the second information processing device And
The second information processing apparatus updates only a portion corresponding to the difference binary data in the texture data.
Information communication system.
[Item 5]
An information communication system according to item 4;
The first information processing apparatus performs the detection based on changes in RGB data values in the partial video data.
Information communication system.
[Item 6]
An information communication system according to any one of items 3 to 5;
The first information processing apparatus divides the video data into six areas, converts the partial video data of the area corresponding to the tilt information into partial binary data, and transmits the partial binary data to the second information processing apparatus.
Information communication system.
[Item 7]
An information communication system according toitem 1 or 2;
The first information processing apparatus transmits the binary data to the second information processing apparatus using a User Datagram Protocol (UDP).
Information communication system.
[Item 8]
Information communication using a first information processing apparatus including an omnidirectional camera and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD) Method,
The omnidirectional camera acquiring video data of a target space and transmitting the video data to the first information processing apparatus;
The first information processing apparatus converts the video data into binary data that can be restored according to a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus;
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
［項目１］
全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを含む情報通信システムであって、
前記全天球カメラは、対象空間の映像データを取得して、前記第１情報処理装置に送信し、
前記第１情報処理装置は、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示する、
情報通信システム。
［項目２］
項目１に記載の情報通信システムであって、
前記表示部は、ヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）であり、仮想空間内における視界情報を疑似球体オブジェクトと当該疑似球体オブジェクトの中心に配置された仮想カメラとを利用して表示するものであり、
前記第２情報処理装置は、前記テクスチャデータを前記疑似球体オブジェクトの内側に配置することにより、前記仮想空間内に前記対象空間を構成する、
情報通信システム。
［項目３］
項目２に記載の情報通信システムであって、
前記第２情報処理装置は、前記ＨＭＤの傾き情報を取得して前記第２情報処理装置に送信し、
前記第１情報処理装置は、前記映像データのうち前記傾き情報に対応する部分映像データについて部分バイナリデータに変換して前記第２情報処理装置に送信する、
情報通信システム。
［項目４］
項目３に記載の情報通信システムであって、
前記第１情報処理装置は、前記部分映像データ内において変化のある部分を検知し、当該変化のある部分のみに対応する差分映像データについて差分バイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記テクスチャデータのうち、前記差分バイナリデータに対応する部分のみを更新する、
情報通信システム。
［項目５］
項目４に記載の情報通信システムであって、
前記第１情報処理装置は、前記部分映像データ内におけるＲＧＢデータ値の変化に基づいて前記検知を行う、
情報通信システム。
［項目６］
項目３乃至項目５のいずれかに記載の情報通信システムであって、
前記第１情報処理装置は、前記映像データを６つの領域に分割し、前記傾き情報に対応する領域の前記部分映像データについて部分バイナリデータに変換して前記第２情報処理装置に送信する、
情報通信システム。
［項目７］
項目１又は項目２に記載の情報通信システムであって、
前記第１情報処理装置は、前記バイナリデータをユーザ・データグラム・プロトコル（Ｕｓｅｒ Ｄａｔａｇｒａｍ Ｐｒｏｔｏｃｏｌ：ＵＤＰ）を利用して、前記第２情報処理装置に送信する、
情報通信システム。
［項目８］
全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを利用した情報通信方法であって、
前記全天球カメラが、対象空間の映像データを取得して、前記第１情報処理装置に送信するステップと、
前記第１情報処理装置が、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信するステップと、
前記第２情報処理装置が、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示するステップと、を含む
情報通信方法。 The contents of the embodiment of the present invention will be listed and described. An information communication system and an information communication method according to an embodiment of the present invention have the following configuration.
[Item 1]
An information communication system including: a first information processing apparatus including an omnidirectional camera; and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD). And
The omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
The first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
Information communication system.
[Item 2]
An information communication system according to
The display unit is a head mounted display (HMD), and displays visual field information in a virtual space using a pseudo sphere object and a virtual camera disposed at the center of the pseudo sphere object. And
The second information processing apparatus configures the target space in the virtual space by arranging the texture data inside the pseudo sphere object.
Information communication system.
[Item 3]
An information communication system according to item 2;
The second information processing apparatus acquires inclination information of the HMD, and transmits the acquired information to the second information processing apparatus.
The first information processing apparatus converts partial video data corresponding to the tilt information in the video data into partial binary data and transmits the partial binary data to the second information processing apparatus.
Information communication system.
[Item 4]
An information communication system according to item 3;
The first information processing apparatus detects a portion having a change in the partial video data, converts difference image data corresponding to only the portion having the change into difference binary data, and transmits the difference to the second information processing device And
The second information processing apparatus updates only a portion corresponding to the difference binary data in the texture data.
Information communication system.
[Item 5]
An information communication system according to item 4;
The first information processing apparatus performs the detection based on changes in RGB data values in the partial video data.
Information communication system.
[Item 6]
An information communication system according to any one of items 3 to 5;
The first information processing apparatus divides the video data into six areas, converts the partial video data of the area corresponding to the tilt information into partial binary data, and transmits the partial binary data to the second information processing apparatus.
Information communication system.
[Item 7]
An information communication system according to
The first information processing apparatus transmits the binary data to the second information processing apparatus using a User Datagram Protocol (UDP).
Information communication system.
[Item 8]
Information communication using a first information processing apparatus including an omnidirectional camera and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD) Method,
The omnidirectional camera acquiring video data of a target space and transmitting the video data to the first information processing apparatus;
The first information processing apparatus converts the video data into binary data that can be restored according to a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus;
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
＜実施の形態の詳細＞
以下、本発明の実施の形態による情報通信システム及び情報通信方法について、図面を参照しながら説明する。 <Details of Embodiment>
Hereinafter, an information communication system and an information communication method according to an embodiment of the present invention will be described with reference to the drawings.
以下、本発明の実施の形態による情報通信システム及び情報通信方法について、図面を参照しながら説明する。 <Details of Embodiment>
Hereinafter, an information communication system and an information communication method according to an embodiment of the present invention will be described with reference to the drawings.
＜概要＞
本発明の実施の形態による情報通信システムは、全天球カメラにより取得した映像を無線インターネット回線（Ｗｉｆｉ／Ｍｏｂｉｌｅ ｎｅｔｗｏｒｋ）経由でも超低遅延かつ安定して同期するための技術である。 <Overview>
An information communication system according to an embodiment of the present invention is a technology for synchronizing an image acquired by an omnidirectional camera with ultra-low delay and stable synchronization via a wireless Internet circuit (Wifi / Mobile network).
本発明の実施の形態による情報通信システムは、全天球カメラにより取得した映像を無線インターネット回線（Ｗｉｆｉ／Ｍｏｂｉｌｅ ｎｅｔｗｏｒｋ）経由でも超低遅延かつ安定して同期するための技術である。 <Overview>
An information communication system according to an embodiment of the present invention is a technology for synchronizing an image acquired by an omnidirectional camera with ultra-low delay and stable synchronization via a wireless Internet circuit (Wifi / Mobile network).
情報通信システムは、主として以下の３つの構成に文化手することができる。即ち、全天球カメラの映像のデータを低遅延で送信するためのデータ転送技術、全天球カメラの映像のデータを低容量で送信するためのデータ量削減技術、そして、全天球カメラの映像のデータを低遅延で全天球映像として復元・描画する技術である。
The information communication system can be mainly culture in the following three configurations. That is, data transfer technology for transmitting video data of the omnidirectional camera with low delay, data amount reduction technology for transmitting video data of the omnidirectional camera with low capacity, and It is a technology that restores and renders video data as an omnidirectional video with low delay.
＜構成＞
図１は、本発明の実施の形態による情報通信システムの構成を示す図である。なお、では一例であり、これ以外の要素が追加されていてもよい。 <Configuration>
FIG. 1 is a diagram showing the configuration of an information communication system according to an embodiment of the present invention. Note that this is an example, and other elements may be added.
図１は、本発明の実施の形態による情報通信システムの構成を示す図である。なお、では一例であり、これ以外の要素が追加されていてもよい。 <Configuration>
FIG. 1 is a diagram showing the configuration of an information communication system according to an embodiment of the present invention. Note that this is an example, and other elements may be added.
図１に示されるように、情報通信システム１は、遠隔地においてユーザが遠隔操作するための遠隔デバイスと、ユーザが操作するための操作デバイスとの２つの要素に分解することができる。操作デバイスと遠隔デバイスとはネットワーク（後述する）を介して接続されている。
As shown in FIG. 1, the information communication system 1 can be divided into two components of a remote device for remote control by a user at a remote location and an operation device for user control. The operation device and the remote device are connected via a network (described later).
遠隔デバイスは、全天球カメラと、第１情報処理端末（ＰＣ１）と、ボディとを含む。全天球カメラは、上下左右全方位の３６０度パノラマ写真が撮れる撮影装置であり、全方位カメラとも呼ばれることがある。ボディは、ユーザの身体の一部を拡張し得る装置であり、例えば、ロボットアーム等の装置が例示できるがこれに限られない。
The remote device includes an omnidirectional camera, a first information processing terminal (PC1), and a body. The omnidirectional camera is an imaging device capable of taking 360-degree panoramic photographs of all directions in the vertical and horizontal directions, and may also be called an omnidirectional camera. The body is a device capable of expanding a part of the user's body, and can be exemplified by, for example, a device such as a robot arm, but is not limited thereto.
操作デバイスは、第２情報処理端末（ＰＣ２）と、ヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔ Ｄｉｓｐｌａｙ：以下、ＨＭＤ）とを含んでいる。
The operation device includes a second information processing terminal (PC2) and a head mount display (HMD).
＜第１情報処理装置１００＞
図２は、第１情報処理端末１００を示すブロック図である。図に示されるように、第１情報処理端末１００は、少なくとも、プロセッサ１０、メモリ１１、ストレージ１２、送受信部１３、入出力部１４等を備え、これらはバス１５を通じて相互に電気的に接続される。 <First Information Processing Apparatus 100>
FIG. 2 is a block diagram showing the first information processing terminal 100. As shown in FIG. As shown in the figure, the first information processing terminal 100 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14 and the like, which are electrically connected to one another through a bus 15. Ru.
図２は、第１情報処理端末１００を示すブロック図である。図に示されるように、第１情報処理端末１００は、少なくとも、プロセッサ１０、メモリ１１、ストレージ１２、送受信部１３、入出力部１４等を備え、これらはバス１５を通じて相互に電気的に接続される。 <First Information Processing Apparatus 100>
FIG. 2 is a block diagram showing the first information processing terminal 100. As shown in FIG. As shown in the figure, the first information processing terminal 100 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14 and the like, which are electrically connected to one another through a bus 15. Ru.
入出力部は、更に、全天球カメラ１３０と、ボディ制御部１３２と、ボディ１３４と通信可能に接続されている。
The input / output unit is further communicably connected to the omnidirectional camera 130, the body control unit 132, and the body 134.
第１情報処理端末１００は、遠隔デバイス内に実装されるため、比較的小型の装置を採用することが好ましい。また、図示されている機能ブロックについては、適宜省略することもできる。
Since the first information processing terminal 100 is implemented in a remote device, it is preferable to employ a relatively small device. The illustrated functional blocks may be omitted as appropriate.
プロセッサ１０は、第１情報処理装置１００全体の動作を制御し、各要素間におけるデータの送受信の制御、及びアプリケーションの実行に必要な情報処理等を行う演算装置である。例えばプロセッサ１０はＣＰＵ（Ｃｅｎｔｒａｌ Ｐｒｏｃｅｓｓｉｎｇ Ｕｎｉｔ）であり、ストレージ１２に格納されメモリ１１に展開されたプログラム等を実行して各情報処理を実施する。
The processor 10 is an arithmetic device that controls the overall operation of the first information processing apparatus 100, controls transmission and reception of data between elements, and performs information processing and the like necessary for executing an application. For example, the processor 10 is a CPU (Central Processing Unit), and executes a program or the like stored in the storage 12 and expanded in the memory 11 to perform each information processing.
メモリ１１は、ＤＲＡＭ（Ｄｙｎａｍｉｃ Ｒａｎｄｏｍ Ａｃｃｅｓｓ Ｍｅｍｏｒｙ）等の揮発性記憶装置で構成される主記憶と、フラッシュメモリやＨＤＤ（Ｈａｒｄ Ｄｉｓｃ Ｄｒｉｖｅ）等の不揮発性記憶装置で構成される補助記憶と、を含む。メモリ１１は、プロセッサ１０のワークエリア等として使用され、また、コンピュータ端末１２０の起動時に実行されるＢＩＯＳ（Ｂａｓｉｃ Ｉｎｐｕｔ ／ Ｏｕｔｐｕｔ Ｓｙｓｔｅｍ）、及び各種設定情報等を格納する。
The memory 11 includes a main storage constituted by a volatile storage device such as a dynamic random access memory (DRAM), and an auxiliary storage constituted by a nonvolatile storage device such as a flash memory or a hard disc drive (HDD). . The memory 11 is used as a work area or the like of the processor 10, and stores a BIOS (Basic Input / Output System) executed when the computer terminal 120 is started, various setting information, and the like.
ストレージ１２は、アプリケーション・プログラム等の各種プログラムを格納する。各処理に用いられるデータを格納したデータベースがストレージ２２に構築されていてもよい。
The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 22.
送受信部１３は、コンピュータ端末１２０をネットワークに接続する。なお、送受信部１３は、Ｂｌｕｅｔｏｏｔｈ（登録商標）及びＢＬＥ（Ｂｌｕｅｔｏｏｔｈ Ｌｏｗ Ｅｎｅｒｇｙ）などの近距離通信インタフェースを備えていてもよい。
The transmission / reception unit 13 connects the computer terminal 120 to a network. The transmission / reception unit 13 may include a short range communication interface such as Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).
入出力部１４には、必要に応じて、更にキーボード・マウス類等の情報入力機器、及びディスプレイ等の出力機器が接続されていてもよい。
If necessary, an information input device such as a keyboard or a mouse and an output device such as a display may be connected to the input / output unit 14.
バス１５は、上記各要素に共通に接続され、例えば、アドレス信号、データ信号及び各種制御信号を伝達する。
The bus 15 is commonly connected to the above-described elements, and transmits, for example, an address signal, a data signal, and various control signals.
＜第２情報処理装置２００＞
図３は、第２情報処理端末２００を示すブロック図である。図に示されるように、第２情報処理端末２００は、少なくとも、プロセッサ２０、メモリ２１、ストレージ２２、送受信部２３、入出力部２４等を備え、これらはバス２５を通じて相互に電気的に接続される。 <Second Information Processing Device 200>
FIG. 3 is a block diagram showing the second information processing terminal 200. As shown in FIG. As shown in the figure, the second information processing terminal 200 includes at least a processor 20, a memory 21, a storage 22, a transmitting / receiving unit 23, an input / output unit 24 and the like, which are electrically connected to each other through a bus 25. Ru.
図３は、第２情報処理端末２００を示すブロック図である。図に示されるように、第２情報処理端末２００は、少なくとも、プロセッサ２０、メモリ２１、ストレージ２２、送受信部２３、入出力部２４等を備え、これらはバス２５を通じて相互に電気的に接続される。 <Second Information Processing Device 200>
FIG. 3 is a block diagram showing the second information processing terminal 200. As shown in FIG. As shown in the figure, the second information processing terminal 200 includes at least a processor 20, a memory 21, a storage 22, a transmitting / receiving unit 23, an input / output unit 24 and the like, which are electrically connected to each other through a bus 25. Ru.
入出力部２４には、更に、操作部２４２と、ＨＭＤ２４４と通信可能に接続されている。
Further, the input / output unit 24 is communicably connected to the operation unit 242 and the HMD 244.
ＨＭＤ２４４は、図示しないディスプレイおよびセンサを具備する。ディスプレイは、ユーザの視界を完全に覆うよう構成された非透過型の表示装置であり、ユーザはディスプレイに表示される画面のみを観察することができる。そして、非透過型のＨＭＤ２４４を装着したユーザは、外界の視界を全て失うため、実行されるアプリケーションにより表示される仮想空間に完全に没入する表示態様となる。
The HMD 244 has a display and a sensor not shown. The display is a non-transmissive display device configured to completely cover the user's field of view, and the user can view only the screen displayed on the display. Then, since the user wearing the non-transmissive HMD 244 loses all the visibility of the outside world, the display mode completely immerses in the virtual space displayed by the executed application.
ユーザは、全天球カメラ１３０（図２参照）によって取得された全天球映像をＨＭＤ２４４内に構成することにより、あたかも、遠隔デバイスに視界が同期したかのような体験をすることができる。
The user can configure the omnidirectional image acquired by the omnidirectional camera 130 (see FIG. 2) in the HMD 244 to give the experience as if the view was synchronized with the remote device.
なお、例えば、操作部２４２を利用して、ボディ１３４（図２参照）を遠隔操作することも可能である。
Note that, for example, it is also possible to remotely control the body 134 (see FIG. 2) using the operation unit 242.
ＨＭＤ２４４が具備するセンサは、ディスプレイ近辺に固定される。センサは、地磁気センサ、加速度センサ、および／または傾き（角速度、ジャイロ）センサを含み、これらの１つ以上を通じて、ユーザの頭部に装着されたＨＭＤ２４４の各種動きを検知することができる。特に角速度センサの場合には、ＨＭＤ２４４の動きに応じて、ＨＭＤ２４４の３軸回りの角速度を経時的に検知し、各軸回りの角度（傾き）の時間変化を決定することができる。
A sensor provided in the HMD 244 is fixed near the display. The sensor includes a geomagnetic sensor, an acceleration sensor, and / or a tilt (angular velocity, gyro) sensor, and can detect various movements of the HMD 244 mounted on the user's head through one or more of these. In the case of an angular velocity sensor, in particular, according to the movement of the HMD 244, the angular velocity around three axes of the HMD 244 can be detected over time, and the time change of the angle (tilt) around each axis can be determined.
操作部２４２は、例えば、ユーザの手に装着可能なグローブ型を有しており、各指の動作量を制限可能な機構を有している。操作部２４２は、ユーザの指の動きを検知する動き検知部と、第２情報処理装置２００で生成した触覚信号をユーザの指に伝達する触覚提示部とを含んでいる。
The operation unit 242 has, for example, a glove type that can be worn on the user's hand, and has a mechanism that can limit the amount of movement of each finger. The operation unit 242 includes a motion detection unit that detects the movement of the user's finger, and a tactile sense presentation unit that transmits the tactile signal generated by the second information processing apparatus 200 to the user's finger.
＜処理の流れ＞
続いて、本実施の形態による情報通信システムの処理の流れを説明する。図４は、本実施の形態による情報通信システムの処理に関して、全天球カメラ１３０、第１情報処理装置１００、第２情報処理装置２００、及びＨＭＤ２４４の間で行われる情報処理の流れを示す図である。 <Flow of processing>
Subsequently, the flow of processing of the information communication system according to the present embodiment will be described. FIG. 4 is a diagram showing the flow of information processing performed among theomnidirectional camera 130, the first information processing apparatus 100, the second information processing apparatus 200, and the HMD 244 regarding the processing of the information communication system according to the present embodiment. It is.
続いて、本実施の形態による情報通信システムの処理の流れを説明する。図４は、本実施の形態による情報通信システムの処理に関して、全天球カメラ１３０、第１情報処理装置１００、第２情報処理装置２００、及びＨＭＤ２４４の間で行われる情報処理の流れを示す図である。 <Flow of processing>
Subsequently, the flow of processing of the information communication system according to the present embodiment will be described. FIG. 4 is a diagram showing the flow of information processing performed among the
図示されるように、全天球カメラ１３０は、映像データを取得する。取得した映像データは、第１情報処理装置１００に送信される（ＳＱ１０１）。第１情報処理装置１００は、受信した映像データを所定のバイナリデータに変換する（ＳＱ１０２）。
As illustrated, the omnidirectional camera 130 acquires video data. The acquired video data is transmitted to the first information processing apparatus 100 (SQ101). The first information processing apparatus 100 converts the received video data into predetermined binary data (SQ102).
本実施の形態においては、映像データの解像度がＨＤＴＶ（１２８０×７２０）以上の場合はＪＰＥＧ形式で復元可能なバイナリデータを使用している。一方、映像データの解像度がＨＤＴＶ（１２８０×７２０）未満の場合はＷｅｂＰ形式で復元可能なバイナリデータに変換する。なお、画像形式は一例であり、これ以外の形式を採用してもよい。
In this embodiment, when the resolution of video data is HDTV (1280 × 720) or more, binary data that can be restored in JPEG format is used. On the other hand, if the resolution of the video data is less than HDTV (1280 × 720), it is converted into binary data that can be restored in WebP format. Note that the image format is an example, and other formats may be adopted.
続いて、第１情報処理装置１００は、バイナリデータを第２情報処理装置２００に送信する（ＳＱ１０３）。本実施の形態においては、バイナリデータは、ユーザ・データグラム・プロトコル（Ｕｓｅｒ Ｄａｔａｇｒａｍ Ｐｒｏｔｏｃｏｌ：ＵＤＰ）を利用して、第２情報処理装置２００に送信される。
Subsequently, the first information processing apparatus 100 transmits binary data to the second information processing apparatus 200 (SQ 103). In the present embodiment, binary data is transmitted to the second information processing apparatus 200 using User Datagram Protocol (UDP).
本実施の形態における第１情報処理装置１００と第２情報処理装置２００との通信は、ＵＤＰをベースとした独自プロトコルを用いることとしている。ＵＤＰのメインデータ部分の先頭に独自のシーケンス（順序）番号を付与することで、ＲＴＰと同じ機能（＝データの順番の保証）をＲＴＰよりはるかに軽量なヘッダーデータ量で実現させる独自プロトコルを定義している。
Communication between the first information processing apparatus 100 and the second information processing apparatus 200 in the present embodiment uses a unique protocol based on UDP. By assigning a unique sequence (order) number at the beginning of the main data part of UDP, a proprietary protocol is realized that achieves the same function as RTP (= guarantee of data order) with a much lighter amount of header data than RTP. doing.
第２情報処理装置２００は、受信したバイナリデータを動画のフレーム毎に所定のテクスチャに変換する（ＳＱ１０４）。変換されたテクスチャデータは、ＨＭＤ２４４に送信される（ＳＱ１０５）。
The second information processing apparatus 200 converts the received binary data into a predetermined texture for each frame of the moving image (SQ 104). The converted texture data is transmitted to the HMD 244 (SQ 105).
本実施の形態によるＨＭＤ２４４に出力され表示される仮想空間は、図５に示されるように、当該仮想空間内における視界情報を疑似球体オブジェクトＶと当該疑似球体オブジェクトＶの中心に配置された仮想カメラＶｃとを利用して表示するものである。
The virtual space output and displayed on the HMD 244 according to the present embodiment is, as shown in FIG. 5, a virtual camera in which the visual field information in the virtual space is arranged at the center of the pseudo sphere object V and the pseudo sphere object V. It is displayed using Vc.
第２情報処理装置２００は、テクスチャデータを疑似球体オブジェクトＶの内側に貼り付けることによって仮想空間を構成する。
The second information processing apparatus 200 configures a virtual space by pasting texture data inside the pseudo sphere object V.
以上のように、本実施の形態によれば、映像データから変換されたバイナリデータをＵＤＰ経由で送信することとしているため、全天球カメラから得られた全天球映像データを高速かつ低遅延で送信することができる。
As described above, according to the present embodiment, since binary data converted from video data is transmitted via UDP, high speed and low delay of all celestial sphere video data obtained from all celestial sphere cameras is achieved. Can be sent.
＜部分バイナリ変換＞
上述した実施の形態は、主に通信速度を改善するための技術である転送技術を説明した。しかしながら、本発明による情報通信技術は、更に、全天球カメラの映像のデータを低容量で送信するためのデータ量削減技術、そして、全天球カメラの映像のデータを低遅延で全天球映像として復元・描画する技術も提供するものである。 <Partial binary conversion>
The above-described embodiment has mainly described the transfer technology, which is a technology for improving the communication speed. However, the information communication technology according to the present invention further includes a data amount reduction technology for transmitting image data of the omnidirectional camera with a low capacity, and an omnidirectional camera with low latency data of the image of the omnidirectional camera. It also provides technology to restore and render as video.
上述した実施の形態は、主に通信速度を改善するための技術である転送技術を説明した。しかしながら、本発明による情報通信技術は、更に、全天球カメラの映像のデータを低容量で送信するためのデータ量削減技術、そして、全天球カメラの映像のデータを低遅延で全天球映像として復元・描画する技術も提供するものである。 <Partial binary conversion>
The above-described embodiment has mainly described the transfer technology, which is a technology for improving the communication speed. However, the information communication technology according to the present invention further includes a data amount reduction technology for transmitting image data of the omnidirectional camera with a low capacity, and an omnidirectional camera with low latency data of the image of the omnidirectional camera. It also provides technology to restore and render as video.
図６は、本発明による情報通信システムの更なる実施の形態の処理の流れを示す図である。なお、図において上述した実施の形態と同様の処理については同一の参照符号を付しその説明を簡略化する。
FIG. 6 is a diagram showing the process flow of a further embodiment of the information communication system according to the present invention. In the figure, the same processing as that of the embodiment described above is denoted by the same reference numeral to simplify the description.
上述した実施の形態と同様に、全天球カメラによって取得された映像データは、第１情報処理装置１００に送信される（ＳＱ１０１）。
As in the embodiment described above, the video data acquired by the omnidirectional camera is transmitted to the first information processing apparatus 100 (SQ 101).
本実施の形態においては、ＨＭＤ２４４の傾き情報（即ち、ユーザの頭部の傾き：視線方向）を更に利用することとしている。ＨＭＤ２４４は傾き情報を検知し（ＳＱ６０１）、当該傾き情報は第２情報処理装置２００に送信される（ＳＱ６０２）。
In the present embodiment, the tilt information of the HMD 244 (i.e., the tilt of the head of the user: the viewing direction) is further used. The HMD 244 detects tilt information (SQ601), and the tilt information is transmitted to the second information processing apparatus 200 (SQ602).
傾き情報は、第１情報処理装置１００に更に送信される（ＳＱ６０３）。第１情報処理装置は、全天球カメラから取得した映像データのうち傾き情報に対応する映像データについて部分的なバイナリデータ（部分バイナリデータ）を生成する（ＳＱ６０４）。
The tilt information is further transmitted to the first information processing apparatus 100 (SQ 603). The first information processing apparatus generates partial binary data (partial binary data) for video data corresponding to tilt information among video data acquired from the omnidirectional camera (SQ 604).
本実施の形態においては、全天球カメラから取得される映像データを６つの領域に分割し、傾き情報に応じてそれに対応する領域について部分的なバイナリデータを生成する。
In the present embodiment, the video data acquired from the omnidirectional camera is divided into six regions, and partial binary data is generated for the corresponding region according to the tilt information.
図５を参照して、分割される領域は、以下の６つの領域である。
領域１：（Ｘ，Ｙ，Ｚ）＝（１，０，０）を頂点として、（０，０，１）、（０，－１，０）、（０，０，－１）及び（０，１，０）を通る仮想円を含む半球：図の右側半球
領域２：（Ｘ，Ｙ，Ｚ）＝（－１，０，０）を頂点として、（０，０，１）、（０，－１，０）、（０，０，－１）及び（０，１，０）を通る仮想円を含む半球：図の左側半球
領域３：（Ｘ，Ｙ，Ｚ）＝（０，１，０）を頂点として、（０，０，１）、（－１，０，０）、（０，０，－１）及び（１，０，０）を通る仮想円を含む半球：図の奥側半球
領域４：（Ｘ，Ｙ，Ｚ）＝（０，－１，０）を頂点として、（０，０，１）、（－１，０，０）、（０，０，－１）及び（１，０，０）を通る仮想円を含む半球：図の手前側半球
領域５：（Ｘ，Ｙ，Ｚ）＝（０，０，１）を頂点として、（１，０，０）、（０，－１，０）、（１，０，０）及び（０，１，０）を通る仮想円を含む半球：図の上側半球
領域６：（Ｘ，Ｙ，Ｚ）＝（０，０，－１）を頂点として、（１，０，０）、（０，－１，０）、（１，０，０）及び（０，１，０）を通る仮想円を含む半球：図の下側半球 Referring to FIG. 5, the divided areas are the following six areas.
Region 1: (X, Y, Z) = (1, 0, 0) as a vertex, (0, 0, 1), (0, -1, 0), (0, 0, -1) and (0 , 1, 0): A hemisphere including an imaginary circle passing through: right hemisphere in the figure Region 2: (X, Y, Z) = (-1, 0, 0) as a vertex, (0, 0, 1), (0 , -1, 0), (0, 0, -1) and (0, 1, 0) including a phantom circle: the left hemisphere in the figure Region 3: (X, Y, Z) = (0, 1) , 0) as vertices, and a hemisphere including an imaginary circle passing through (0, 0, 1), (-1, 0, 0), (0, 0, -1) and (1, 0, 0): Back side hemisphere Region 4: (X, Y, Z) = (0, -1, 0) as a vertex, (0, 0, 1), (-1, 0, 0), (0, 0, -1) And a hemisphere including a virtual circle passing through (1, 0, 0): the front hemisphere of the figure Region 5: (X, Y, Z) = (0, 0, 1) A hemisphere including a virtual circle passing through (1, 0, 0), (0, -1, 0), (1, 0, 0) and (0, 1, 0) with the vertex at the top: the upper hemisphere of the figure Region 6: (X, Y, Z) = (0, 0, -1) as a vertex, (1, 0, 0), (0, -1, 0), (1, 0, 0) and (0) , 1, 0) including the imaginary circle passing through: the lower hemisphere of the figure
領域１：（Ｘ，Ｙ，Ｚ）＝（１，０，０）を頂点として、（０，０，１）、（０，－１，０）、（０，０，－１）及び（０，１，０）を通る仮想円を含む半球：図の右側半球
領域２：（Ｘ，Ｙ，Ｚ）＝（－１，０，０）を頂点として、（０，０，１）、（０，－１，０）、（０，０，－１）及び（０，１，０）を通る仮想円を含む半球：図の左側半球
領域３：（Ｘ，Ｙ，Ｚ）＝（０，１，０）を頂点として、（０，０，１）、（－１，０，０）、（０，０，－１）及び（１，０，０）を通る仮想円を含む半球：図の奥側半球
領域４：（Ｘ，Ｙ，Ｚ）＝（０，－１，０）を頂点として、（０，０，１）、（－１，０，０）、（０，０，－１）及び（１，０，０）を通る仮想円を含む半球：図の手前側半球
領域５：（Ｘ，Ｙ，Ｚ）＝（０，０，１）を頂点として、（１，０，０）、（０，－１，０）、（１，０，０）及び（０，１，０）を通る仮想円を含む半球：図の上側半球
領域６：（Ｘ，Ｙ，Ｚ）＝（０，０，－１）を頂点として、（１，０，０）、（０，－１，０）、（１，０，０）及び（０，１，０）を通る仮想円を含む半球：図の下側半球 Referring to FIG. 5, the divided areas are the following six areas.
Region 1: (X, Y, Z) = (1, 0, 0) as a vertex, (0, 0, 1), (0, -1, 0), (0, 0, -1) and (0 , 1, 0): A hemisphere including an imaginary circle passing through: right hemisphere in the figure Region 2: (X, Y, Z) = (-1, 0, 0) as a vertex, (0, 0, 1), (0 , -1, 0), (0, 0, -1) and (0, 1, 0) including a phantom circle: the left hemisphere in the figure Region 3: (X, Y, Z) = (0, 1) , 0) as vertices, and a hemisphere including an imaginary circle passing through (0, 0, 1), (-1, 0, 0), (0, 0, -1) and (1, 0, 0): Back side hemisphere Region 4: (X, Y, Z) = (0, -1, 0) as a vertex, (0, 0, 1), (-1, 0, 0), (0, 0, -1) And a hemisphere including a virtual circle passing through (1, 0, 0): the front hemisphere of the figure Region 5: (X, Y, Z) = (0, 0, 1) A hemisphere including a virtual circle passing through (1, 0, 0), (0, -1, 0), (1, 0, 0) and (0, 1, 0) with the vertex at the top: the upper hemisphere of the figure Region 6: (X, Y, Z) = (0, 0, -1) as a vertex, (1, 0, 0), (0, -1, 0), (1, 0, 0) and (0) , 1, 0) including the imaginary circle passing through: the lower hemisphere of the figure
上述した領域の数やその分割方法は例示であり、適宜変更することが可能である。
The number of areas described above and the method of dividing the area are examples and can be changed as appropriate.
本実施の形態においても、バイナリデータは、ＵＤＰ経由で第２情報処理装置に送信される（ＳＱ６０３）。
Also in the present embodiment, binary data is transmitted to the second information processing apparatus via UDP (SQ 603).
第２情報処理装置２００は、受信した部分バイナリデータを動画のフレーム毎に所定のテクスチャに変換する（ＳＱ１０４）。変換されたテクスチャデータは、ＨＭＤ２４４に送信される（ＳＱ１０５）。
The second information processing apparatus 200 converts the received partial binary data into a predetermined texture for each frame of a moving image (SQ 104). The converted texture data is transmitted to the HMD 244 (SQ 105).
第２情報処理装置２００は、部分バイナリデータから変換されたテクスチャデータを疑似球体オブジェクトＶの内側の一部に貼り付けることによって仮想空間を構成する。
ＨＭＤ２４４に出力され表示される仮想空間には、上述した６つの領域に分割した領域に対応する部分バイナリデータから変換された映像が表示される（ＳＱ１０６）。 The second information processing apparatus 200 constructs a virtual space by attaching the texture data converted from the partial binary data to a part of the inside of the pseudo sphere object V.
In the virtual space output and displayed on theHMD 244, an image converted from partial binary data corresponding to the above-described six areas is displayed (SQ106).
ＨＭＤ２４４に出力され表示される仮想空間には、上述した６つの領域に分割した領域に対応する部分バイナリデータから変換された映像が表示される（ＳＱ１０６）。 The second information processing apparatus 200 constructs a virtual space by attaching the texture data converted from the partial binary data to a part of the inside of the pseudo sphere object V.
In the virtual space output and displayed on the
かかる実施の形態によれば、全天球カメラからの映像データのうち、ＨＭＤの傾きに基づいて、 上述した6つの領域のうちからユーザの見ている範囲の領域のみの映像データのみが送信対象となる。従って、送信対象となるデータ量を削減することができる。
According to this embodiment, of the video data from the omnidirectional camera, only video data of only the area viewed by the user among the above six areas is to be transmitted based on the inclination of the HMD. It becomes. Accordingly, the amount of data to be transmitted can be reduced.
＜差分バイナリデータ＞
続いて説明する実施の形態による技術は、映像データの差分のみを送信対象とするものであり、更なる送信データの削減を実現することが可能となる。 <Differential binary data>
The technology according to the embodiment to be described subsequently is intended to transmit only the difference of video data, and it is possible to realize further reduction of transmission data.
続いて説明する実施の形態による技術は、映像データの差分のみを送信対象とするものであり、更なる送信データの削減を実現することが可能となる。 <Differential binary data>
The technology according to the embodiment to be described subsequently is intended to transmit only the difference of video data, and it is possible to realize further reduction of transmission data.
図７は、かかる処理の流れを示す図である。なお、図７においても、上述した実施の形態と同様の処理については同一の参照符号を付しその説明を簡略化する。
FIG. 7 is a diagram showing the flow of such processing. Also in FIG. 7, the same processing as that of the above-described embodiment is denoted by the same reference numeral, and the description thereof will be simplified.
上述した実施の形態と同様に、全天球カメラによって取得された映像データは、第１情報処理装置１００に送信される（ＳＱ１０１）。
As in the embodiment described above, the video data acquired by the omnidirectional camera is transmitted to the first information processing apparatus 100 (SQ 101).
本実施の形態においても、ＨＭＤ２４４の傾き情報（即ち、ユーザの頭部の傾き：視線方向）を更に利用することとし、ＨＭＤ２４４によって傾き情報を検知し（ＳＱ６０１）、第２情報処理装置２００を介して第１情報処理装置１００に送信することとしている（ＳＱ６０２、ＳＱ６０３）。
Also in the present embodiment, the tilt information of the HMD 244 (that is, the tilt of the head of the user: the direction of the line of sight) is used further, the tilt information is detected by the HMD 244 (SQ601), and the second information processing apparatus 200 is used. Transmission to the first information processing apparatus 100 (SQ 602, SQ 603).
第１情報処理装置１００は、上述した映像データ内においてＲＧＢデータ値の変化に基づいて映像内の変化のある部分（差分領域）を検知・特定する（ＳＱ７０１）。第１情報処理装置１００は、当該差分領域についてのみバイナリデータ（差分バイナリデータ）を生成し（ＳＱ７０２）、ＵＤＰ経由で第２情報処理装置２００に送信する（ＳＱ７０３）。
The first information processing apparatus 100 detects and specifies a portion (difference area) having a change in the video based on the change of the RGB data value in the above-described video data (SQ 701). The first information processing apparatus 100 generates binary data (difference binary data) only for the difference area (SQ 702), and transmits the binary data to the second information processing apparatus 200 via UDP (SQ 703).
第２情報処理装置２００では、テクスチャデータのうち、差分領域に対応する部分のみを更新し（ＳＱ７０４）、当該更新データをＨＭＤに送信する（ＳＱ７０５）。
The second information processing apparatus 200 updates only the part corresponding to the difference area in the texture data (SQ 704), and transmits the updated data to the HMD (SQ 705).
かかる構成によれば、映像データのうち、変化のある部分のみが送信対象となるため、送信データの更なる削減を行うことができる。
According to this configuration, only the portion of the video data that has a change is to be transmitted, so that it is possible to further reduce the transmission data.
なお、この際、映像データ（即ち、フレーム毎の画像）に入るノイズを除去する処理を行った後に差分を検知することとしてもよい。
At this time, the difference may be detected after processing for removing noise that enters video data (that is, an image for each frame).
即ち、例えば、全天球カメラ側の一般的な問題(仕様)等によっては、1フレーム毎の画像には、通常の物体や風景とは異なる色の点(ノイズ)が複数入り込んでしまうことが起こり得る。このとき、当該ノイズ（即ち、上述した「変化のある部分」に相当）を送信対象の差分領域であると判定してしまうと、不必要な差分処理データが発生することとなる。
That is, for example, depending on a general problem (specification) or the like on the omnidirectional camera side, a plurality of points (noises) of a color different from that of a normal object or landscape may be included in the image for each frame. It can happen. At this time, if it is determined that the noise (that is, equivalent to the above-described “portion with change”) is a difference region to be transmitted, unnecessary difference processing data is generated.
そこで、本実施の形態においては、このノイズを除去するために、以下の3ステップの処理を行うこととしている。
即ち、
（１）映像を構成するフレーム毎の画像を2値化(画像を白と黒の2階調に変換する処理)する
（２）当該画像(dpi)をＮ倍（例えば２倍）に拡大する
（３）拡大した画像(dpi)をＭ倍（Ｍ＞Ｎ：例えば３倍）に縮小する Therefore, in the present embodiment, the following three steps are performed to remove the noise.
That is,
(1) Binarize the image for each frame constituting the video (process to convert the image into two gradations of white and black) (2) enlarge the image (dpi) by N times (for example, twice) (3) Reduce the magnified image (dpi) to M times (M> N: 3 times, for example)
即ち、
（１）映像を構成するフレーム毎の画像を2値化(画像を白と黒の2階調に変換する処理)する
（２）当該画像(dpi)をＮ倍（例えば２倍）に拡大する
（３）拡大した画像(dpi)をＭ倍（Ｍ＞Ｎ：例えば３倍）に縮小する Therefore, in the present embodiment, the following three steps are performed to remove the noise.
That is,
(1) Binarize the image for each frame constituting the video (process to convert the image into two gradations of white and black) (2) enlarge the image (dpi) by N times (for example, twice) (3) Reduce the magnified image (dpi) to M times (M> N: 3 times, for example)
かかるステップを行うことにより、ノイズである通常の物体や風景とは異なる色の点は画像から除去できることから、通常の物体や風景(色の集合体)のみが画像内に残る。差
By performing such steps, it is possible to remove from the image noise that is different from normal objects and scenery that are noises, so that only ordinary objects and scenery (a collection of colors) remain in the image. difference
上述した実施の形態における情報処理装置は物理的にHMDや操作部に接続されていることとしたが、各要素を接続する形態はこれに限られるものではない。無線ネットワークによって接続されている形態でもよいし、情報処理装置がHMDや操作部に内包される形態でもよい。または演算処理をクラウドコンピューティングに実施させる形態でもよい。
Although the information processing apparatus in the embodiment described above is physically connected to the HMD or the operation unit, the form of connecting each element is not limited to this. It may be in a form of being connected by a wireless network, or it may be in a form in which the information processing apparatus is included in the HMD or the operation unit. Alternatively, the computing process may be performed by cloud computing.
上述した実施の形態は、本発明の理解を容易にするための例示に過ぎず、本発明を限定して解釈するためのものではない。本発明は、その趣旨を逸脱することなく、変更、改良することができると共に、本発明にはその均等物が含まれることは言うまでもない。
The embodiments described above are merely examples for facilitating the understanding of the present invention, and are not intended to limit and interpret the present invention. It goes without saying that the present invention can be modified and improved without departing from the gist thereof, and that the present invention includes equivalents thereof.
１ 情報通信システム
１００ 第１情報処理システム
２００ 第２情報処理システム 1 information communication system 100 first information processing system 200 second information processing system
１００ 第１情報処理システム
２００ 第２情報処理システム 1 information communication system 100 first information processing system 200 second information processing system
Claims (8)
- 全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを含む情報通信システムであって、
前記全天球カメラは、対象空間の映像データを取得して、前記第１情報処理装置に送信し、
前記第１情報処理装置は、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示する、
情報通信システム。 An information communication system including: a first information processing apparatus including an omnidirectional camera; and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD). And
The omnidirectional camera acquires video data of a target space and transmits the video data to the first information processing apparatus,
The first information processing apparatus converts the video data into binary data that can be restored in a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus.
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
Information communication system. - 請求項１に記載の情報通信システムであって、
前記表示部は、ヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）であり、仮想空間内における視界情報を疑似球体オブジェクトと当該疑似球体オブジェクトの中心に配置された仮想カメラとを利用して表示するものであり、
前記第２情報処理装置は、前記テクスチャデータを前記疑似球体オブジェクトの内側に配置することにより、前記仮想空間内に前記対象空間を構成する、
情報通信システム。 The information communication system according to claim 1, wherein
The display unit is a head mounted display (HMD), and displays visual field information in a virtual space using a pseudo sphere object and a virtual camera disposed at the center of the pseudo sphere object. And
The second information processing apparatus configures the target space in the virtual space by arranging the texture data inside the pseudo sphere object.
Information communication system. - 請求項２に記載の情報通信システムであって、
前記第２情報処理装置は、前記ＨＭＤの傾き情報を取得して前記第２情報処理装置に送信し、
前記第１情報処理装置は、前記映像データのうち前記傾き情報に対応する部分映像データについて部分バイナリデータに変換して前記第２情報処理装置に送信する、
情報通信システム。 The information communication system according to claim 2, wherein
The second information processing apparatus acquires inclination information of the HMD, and transmits the acquired information to the second information processing apparatus.
The first information processing apparatus converts partial video data corresponding to the tilt information in the video data into partial binary data and transmits the partial binary data to the second information processing apparatus.
Information communication system. - 請求項３に記載の情報通信システムであって、
前記第１情報処理装置は、前記部分映像データ内において変化のある部分を検知し、当該変化のある部分のみに対応する差分映像データについて差分バイナリデータに変換して前記第２情報処理装置に送信し、
前記第２情報処理装置は、前記テクスチャデータのうち、前記差分バイナリデータに対応する部分のみを更新する、
情報通信システム。 The information communication system according to claim 3, wherein
The first information processing apparatus detects a portion having a change in the partial video data, converts difference image data corresponding to only the portion having the change into difference binary data, and transmits the difference to the second information processing device And
The second information processing apparatus updates only a portion corresponding to the difference binary data in the texture data.
Information communication system. - 請求項４に記載の情報通信システムであって、
前記第１情報処理装置は、前記部分映像データ内におけるＲＧＢデータ値の変化に基づいて前記検知を行う、
情報通信システム。 The information communication system according to claim 4, wherein
The first information processing apparatus performs the detection based on changes in RGB data values in the partial video data.
Information communication system. - 請求項３乃至請求項５のいずれかに記載の情報通信システムであって、
前記第１情報処理装置は、前記映像データを６つの領域に分割し、前記傾き情報に対応する領域の前記部分映像データについて部分バイナリデータに変換して前記第２情報処理装置に送信する、
情報通信システム。 The information communication system according to any one of claims 3 to 5, wherein
The first information processing apparatus divides the video data into six areas, converts the partial video data of the area corresponding to the tilt information into partial binary data, and transmits the partial binary data to the second information processing apparatus.
Information communication system. - 請求項１乃至請求項６のいずれかに記載の情報通信システムであって、
前記第１情報処理装置は、前記バイナリデータをユーザ・データグラム・プロトコル（Ｕｓｅｒ Ｄａｔａｇｒａｍ Ｐｒｏｔｏｃｏｌ：ＵＤＰ）を利用して、前記第２情報処理装置に送信する、
情報通信システム。 The information communication system according to any one of claims 1 to 6, wherein
The first information processing apparatus transmits the binary data to the second information processing apparatus using a User Datagram Protocol (UDP).
Information communication system. - 全天球カメラを備える第１情報処理装置と、当該第１情報処理装置と通信可能に接続され且つヘッドマウント・ディスプレイ（Ｈｅａｄ Ｍｏｕｎｔｅｄ Ｄｉｓｐｌａｙ：ＨＭＤ）を備える第２情報処理装置とを利用した情報通信方法であって、
前記全天球カメラが、対象空間の映像データを取得して、前記第１情報処理装置に送信するステップと、
前記第１情報処理装置が、前記映像データを少なくとも１フレーム毎に所定映像形式によって復元可能なバイナリデータに変換して前記第２情報処理装置に送信するステップと、
前記第２情報処理装置が、前記バイナリデータを所定のテクスチャデータに変換し、前記表示部に表示するステップと、を含む
情報通信方法。
Information communication using a first information processing apparatus including an omnidirectional camera and a second information processing apparatus communicably connected to the first information processing apparatus and including a head mounted display (HMD) Method,
The omnidirectional camera acquiring video data of a target space and transmitting the video data to the first information processing apparatus;
The first information processing apparatus converts the video data into binary data that can be restored according to a predetermined video format at least every one frame, and transmits the binary data to the second information processing apparatus;
The second information processing apparatus converts the binary data into predetermined texture data and displays the texture data on the display unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2018/000135 WO2019135289A1 (en) | 2018-01-07 | 2018-01-07 | Information communication system and information communication method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2018/000135 WO2019135289A1 (en) | 2018-01-07 | 2018-01-07 | Information communication system and information communication method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2019135289A1 true WO2019135289A1 (en) | 2019-07-11 |
Family
ID=67144112
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2018/000135 WO2019135289A1 (en) | 2018-01-07 | 2018-01-07 | Information communication system and information communication method |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2019135289A1 (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001298652A (en) * | 2000-04-17 | 2001-10-26 | Sony Corp | Method and device for compressing image and software storage medium |
| JP2004023748A (en) * | 2002-06-20 | 2004-01-22 | Ricoh Co Ltd | Device for recording image, method of recording image, program, and information recording medium |
| JP2008171292A (en) * | 2007-01-15 | 2008-07-24 | Toshiba Corp | Image forming device, display terminal, communication system, communication method, and program |
| JP2017123503A (en) * | 2016-01-04 | 2017-07-13 | 日本電信電話株式会社 | Video distribution apparatus, video distribution method and computer program |
-
2018
- 2018-01-07 WO PCT/JP2018/000135 patent/WO2019135289A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001298652A (en) * | 2000-04-17 | 2001-10-26 | Sony Corp | Method and device for compressing image and software storage medium |
| JP2004023748A (en) * | 2002-06-20 | 2004-01-22 | Ricoh Co Ltd | Device for recording image, method of recording image, program, and information recording medium |
| JP2008171292A (en) * | 2007-01-15 | 2008-07-24 | Toshiba Corp | Image forming device, display terminal, communication system, communication method, and program |
| JP2017123503A (en) * | 2016-01-04 | 2017-07-13 | 日本電信電話株式会社 | Video distribution apparatus, video distribution method and computer program |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11303875B2 (en) | Split rendering between a head-mounted display (HMD) and a host computer | |
| CN110488977B (en) | Virtual reality display method, device and system and storage medium | |
| US10585283B2 (en) | Head mounted display and control method therefor | |
| WO2018051815A1 (en) | Information processing apparatus and method and program | |
| WO2019026765A1 (en) | Rendering device, head-mounted display, image transmission method, and image correction method | |
| JP2017097122A (en) | Information processing device and image generation method | |
| WO2020140758A1 (en) | Image display method, image processing method, and related devices | |
| JP7192592B2 (en) | Imaging device, image communication system, image processing method, program | |
| JP2020150330A (en) | Image communication system, imaging apparatus, mode switching method, program | |
| US11120632B2 (en) | Image generating apparatus, image generating system, image generating method, and program | |
| JP5743016B2 (en) | Apparatus and method for generating images | |
| US10572764B1 (en) | Adaptive stereo rendering to reduce motion sickness | |
| US11006042B2 (en) | Imaging device and image processing method | |
| KR20140141419A (en) | Display apparatus and control method thereof | |
| JP6518645B2 (en) | INFORMATION PROCESSING APPARATUS AND IMAGE GENERATION METHOD | |
| JP2007102462A (en) | Image composition method, system, terminal and image composition program | |
| US20230071690A1 (en) | Remote control system, remote operation apparatus, video image processing apparatus, and computer-readable medium | |
| WO2019135289A1 (en) | Information communication system and information communication method | |
| JP2020145590A (en) | Image processing system, image processing method, and program | |
| CN107580157B (en) | Image processing apparatus, image processing method, and program | |
| JP2018078422A (en) | Imaging apparatus | |
| US10186016B2 (en) | Image processing device, image display device, image processing system, and image processing method | |
| JP2020109955A (en) | Photographing device, photographing system, image processing method, and program | |
| JP2018109740A (en) | Display device, program, and method for display | |
| US20220165021A1 (en) | Apparatus, system, method, and non-transitory medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18898654 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 18898654 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: JP |