JP2017010120A - Information processing device, video processing device, control method for those, and video processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the amount of video data transmitted from an information processing device to a video processing device while restraining reduction in the detection accuracy of the position and posture of a movable information processing device (for example, an HMD).SOLUTION: A video processing system includes: a transportable information processing device, and a video processing device for superposing a virtual image on video received from the information processing device. The information processing device transmits video used for determining the position and posture of the information processing device to the video processing device via a communication unit, and the video processing device determines the position and posture of the information processing device on the basis of the received video. The video includes multiple pieces of video corresponding to different spaces, and the ratio of the amounts of data of the multiple pieces of video transmitted from the information processing device to the video processing device is controlled on the basis of a communication band usable at the communication unit and the determined position and posture.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an information processing apparatus, a video processing apparatus, a control method thereof, and a video processing system that realize, for example, mixed reality technology.

  Mixed reality (MR) technology is known as a technology that seamlessly fuses the real world and the virtual world in real time. One of the methods for realizing the MR system is to use a video see-through type HMD (Head Mounted Display). In this MR system, a field of view of an HMD user is imaged by a camera, and an image obtained by synthesizing CG (Computer Graphics) with the captured image is presented to an HMD wearer (hereinafter referred to as an HMD user). In Patent Document 1, as an MR system using a video see-through HMD, a first camera that captures an HMD user's field of view and a second camera that captures an image for detecting the position and orientation of the HMD are provided. A technique using the provided HMD has been proposed.

  Since the processing for acquiring the position and orientation of the HMD from the captured video and the processing for generating the CG according to the position and orientation of the HMD have a high calculation processing cost, these processes are performed on a video processing apparatus as an external device connected to the HMD. It is running. In this case, a video processing device such as a PC as an external device and the HMD are connected, and the captured video is transmitted from the HMD to the video processing device. Then, the position and orientation of the HMD is detected on the video processing device, the CG is generated according to the HMD position and orientation, and the CG is synthesized with the captured image. The captured image with the synthesized CG is sent back to the HMD and displayed. The In such an MR system, it is desirable that the HMD and the video processing apparatus be connected by wireless communication so that the HMD user can experience the MR space while freely moving. However, generally, wireless communication has a narrower communication band than wired communication, and thus it is necessary to reduce the amount of data to be transmitted wirelessly. In addition, since the HMD is driven by a battery or the like, it is desirable to reduce the amount of transmission data from the viewpoint of power consumption.

  As a technique for reducing the amount of video data to be transmitted, a method of compressing data using a video encoding technique is known. Patent Document 2 proposes a technique for calculating an effective communication band from a packet loss rate / delay time measured by a receiving apparatus and controlling a transmission rate of video coding according to the effective communication band.

JP 2004-205711 A JP 2006-128997 A

  In the video see-through MR system described above, the video processing apparatus detects the position and orientation of the HMD by extracting an index such as a marker or a natural feature point from the captured video received from the HMD by image analysis. In this detection process, high-definition video data is required, and if the video data is significantly compressed as in Patent Document 2, sufficient and sufficient video quality cannot be obtained, and the detection accuracy of the position and orientation of the HMD is high. There is a problem that it decreases.

  The present invention has been made in view of the above-described problems. For example, in an image processing system such as an MR system, the information processing apparatus (for example, an HMD) can be controlled while suppressing a decrease in position and orientation detection accuracy. The object is to reduce the amount of video data transmitted from the processing device to the video processing device.

In order to achieve the above object, a video processing system according to an aspect of the present invention has the following arrangement. That is,
A video processing system comprising a portable information processing device and a video processing device that synthesizes a virtual image with video received from the information processing device,
First transmission means for transmitting a first video used for determining the position and orientation of the information processing apparatus from the information processing apparatus to the video processing apparatus via a communication means; and the first video is , Including multiple videos corresponding to different spaces,
In the video processing device, determination means for determining a position and orientation of the information processing device based on the first video;
Based on the communication band that can be used by the first transmission unit of the communication unit and the position and orientation determined by the determination unit, the data amount of the plurality of videos transmitted by the first transmission unit Control means for controlling the ratio.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to reduce the data amount of the image | video transmitted from an information processing apparatus to a video processing apparatus, suppressing the fall of the detection accuracy of the position and orientation of a movable information processing apparatus.

The block diagram of MR system by embodiment. The figure which shows the example of the natural feature point which can be utilized as a parameter | index in MR system. The figure which shows the example of arrangement | positioning of the marker which can be utilized as a parameter | index, and the example of the imaging range of a sub camera in MR system. The block diagram which shows the function structural example of HMD by 1st Embodiment. The block diagram which shows the function structural example of PC by 1st Embodiment. The figure which shows the production | generation timing of video data and motion information. The figure which shows the structure of a communication frame. The figure which shows the structure of a communication frame. The sequence diagram which shows the communication operation between HMD and PC by 1st Embodiment. The flowchart which shows operation | movement of HMD by 1st Embodiment. The flowchart which shows operation | movement of PC by 1st Embodiment. The block diagram which shows the function structural example of HMD by 2nd Embodiment. The block diagram which shows the function structural example of PC by 2nd Embodiment. The sequence diagram which shows the communication operation between HMD and PC by 2nd Embodiment. The flowchart which shows operation | movement of HMD by 2nd Embodiment. The flowchart which shows operation | movement of PC by 2nd Embodiment.

  Hereinafter, some preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of an MR system according to the first embodiment. The MR system includes a portable information processing device that can be moved with the user when the user carries it, and a video processing device that synthesizes a virtual image with a video received from the information processing device. System. The MR system of the present embodiment includes a head-mounted display (hereinafter referred to as HMD 101), which is a video see-through head-mounted display device worn by the user, as an information processing apparatus that can move with the user, and an image. A PC 130 as a processing apparatus is provided. The marker 120 is used to detect the three-dimensional position and orientation (hereinafter referred to as “position and orientation”) of the HMD 101 from the video data. The arrangement position of the marker 120 in the real space is known.

  The HMD 101 captures the video image used for detecting the position and orientation, and the right main camera 105 and the left main camera 106 for capturing an image in which the HMD user's field of view is captured and a virtual image is combined. A sub camera 107 and a lower sub camera 108 are provided. In the present embodiment, two upper and lower sub-cameras are used so as to increase the possibility that the marker 120 is imaged by expanding the imaging range. However, the number of sub cameras may be three or more. As described above, a plurality of videos corresponding to different spaces are acquired by the plurality of sub-cameras. Hereinafter, the right main camera 105 and the left main camera 106 are collectively referred to as a main camera, and the upper sub camera 107 and the lower sub camera 108 are collectively referred to as a sub camera. The HMD 101 includes an inertial sensor such as a gyro sensor or an acceleration sensor, and constantly measures the amount of rotation of the HMD 101 at a predetermined sampling frequency and outputs it as motion information.

  The main video data captured by the main camera and the sub camera, the sub video data, and the movement rotation amount information (motion information) measured by the inertial sensor 109 are transmitted to the PC 130 via the communication link 125. Examples of the transmission method of the communication link 125 include, but are not limited to, wired methods such as Gigabit Ethernet (registered trademark) and IEEE 1394, and wireless methods such as IEEE 802.11n and IEEE 802.11ac.

  The PC 130 detects the marker 120 by image analysis using the received sub video data, and detects the position and orientation of the HMD 101. More specifically, the PC 130 detects a marker from the sub video data, and acquires information such as the size and shape of the marker, and a fill pattern. The PC 130 can calculate the relative positional relationship between the position / orientation detection marker 120 and the HMD 101 and the three-dimensional position / orientation information regarding the direction in which the HMD 101 is imaging the marker from the acquired information. In the present embodiment, a plurality of markers 120 are used, the positional relationship between the markers is defined in advance as index arrangement information, and the HMD 101 observes (captures) the markers from their relative positional relationships. The direction is calculated. Therefore, it is possible to use a marker that does not have direction information, such as a color marker or a light emitting element such as an LED, instead of a marker that can be identified by the internal filling pattern.

  In addition, the natural feature points in the image, such as the contour lines of the door 200, the table 205, and the window 210 shown in FIG. However, it is also possible to calculate position and orientation information using them. In FIG. 2B, marks 250 and 255 indicate some of the feature points of the door 200. By using a plurality of the same type of markers, using several types of markers at the same time, or combining information obtained from the markers and information on feature points in the image, it is possible to generate position and orientation information with higher accuracy. Furthermore, since the positional relationship of a plurality of markers and feature points is associated, it is possible to estimate the position of each marker or feature point even if not all markers or feature points are displayed in the image. Is possible.

  Further, the PC 130 estimates the position and orientation using the received motion information (movement rotation amount information) indicating the motion of the HMD 101. For example, even when a marker is not included in the imaging range of the sub camera, the current position and orientation can be obtained by combining the position and orientation information detected by the PC 130 using the sub camera image in the past and the integrated movement rotation amount information. Can be estimated. When detecting or estimating the position and orientation of the HMD 101, the PC 130 generates a CG as a virtual image according to the detected or estimated position and orientation, synthesizes the generated CG on the main camera video, and generates a CG composite video. The PC 130 transmits the generated CG composite video to the HMD 101 via the communication link 135, and the HMD 101 displays the received CG composite video.

  FIG. 3 is a diagram illustrating an example of the position of the HMD 101 and the imaging ranges of the upper sub camera 107 and the lower sub camera 108. In FIG. 3, an imaging range 310 and an imaging range 315 are imaging ranges of the upper sub camera 107 and the lower sub camera when the HMD is located at P1. The imaging range 310 of the upper sub camera 107 includes one marker, and the imaging range 315 of the lower sub camera 108 includes two markers. The imaging range 320 and the imaging range 325 are imaging ranges of the upper sub camera 107 and the lower sub camera 108 when the HMD moves to P2. The imaging range 325 of the lower sub camera 108 includes one marker, and the imaging range 320 of the upper sub camera 107 includes three markers. As described above, as the position and orientation of the HMD 101 changes, the number of markers included in the imaging ranges of the upper sub camera 107 and the lower sub camera 108 changes. The number of markers can be determined from the marker arrangement information and the position and orientation of the HMD 101.

  FIG. 4 is a functional block diagram of the HMD 101 according to the first embodiment. In FIG. 4, the first imaging unit 400 is a main camera for imaging a real space image viewed by the user wearing the HMD 101, and includes a left main camera 106 and a right main camera 105. The first compression unit 405 compresses the left and right main camera images captured by the left main camera 106 and the right main camera 105 of the first imaging unit 400 using a predetermined encoding method. Examples of encoding methods include Motion-JPEG and H.264. H.264 / AVC, etc. are mentioned, but other methods may be used. The left and right main camera images compressed by the first compression unit 405 are transmitted to the PC 130 via the communication unit 406.

  The second imaging unit 410 includes an upper sub camera 107 and a lower sub camera 108 for capturing an image for detecting the position and orientation of the HMD 101, and outputs upper and lower sub camera images. The priority information acquisition unit 415 acquires priority information including the transmission priority of the sub camera video from the PC 130 via the communication unit 406. The priority information will be described later. The band determination unit 420 determines the excess or deficiency of the communication band that can be used for transmission of the sub camera video (video used for position and orientation detection) acquired by the sub camera via the communication unit 406. For example, when wireless communication is used as the communication unit 406, it is assumed that the communication band that can be used by the communication unit 406 greatly changes as the HMD 101 moves. The band determination unit 420 monitors the communication band (data transmission amount that can be communicated) of such wireless communication used by the communication unit 406 and determines whether the communication band is sufficient or insufficient.

  The ratio determination unit 425 determines the video acquired by the upper sub camera 107 and the lower sub camera 108 based on the determination of excess or deficiency of the communication band acquired by the band determination unit 420 and the priority information acquired by the priority information acquisition unit 415. Determine the ratio of the amount of transmitted data. For example, it is assumed that the priority of the video of the upper sub camera 107 is high and it is determined that the available communication band is insufficient. In this case, if the ratio of the video transmission data amount of the upper sub camera 107 and the lower sub camera 108 is set to 1: 0, the entire communication band is allocated to the video transmission of the upper sub camera 107. When there is a sufficient available communication band, the ratio of the transmission data amount is 1: 1, and it is possible to evenly allocate the communication band to the video transmission of both the upper and lower sub cameras.

  The second compression unit 430 compresses the upper and lower sub-camera images captured by the second imaging unit 410 using a predetermined encoding method according to the ratio of the transmission data amount determined by the ratio determination unit 425. In the present embodiment, the first compression unit 405 and the second compression unit 430 are separate blocks, but may be a common block. The motion detection unit 435 generates motion information from a signal received from the inertial sensor 109 such as a gyro sensor or an acceleration sensor. The motion detection unit 435 steadily measures the amount of movement rotation of the HMD 101 using the inertial sensor 109 at a predetermined sampling frequency. The amount of movement rotation measured by the motion detection unit 435 is collected for a predetermined number of samples and transmitted to the PC 130 via the communication unit 406 as motion information. The composite video decompression unit 440 decompresses the compressed CG composite video received from the PC 130 via the communication unit 406. The expanded CG composite image is displayed on the display unit 445 and presented to the user (wearer) of the HMD 101. Note that each functional unit shown in FIG. 4 is implemented in the apparatus as hardware or software. When these functional units are implemented as software, a program for executing these functions is stored in a storage unit (a memory such as a ROM or a RAM) provided in the HMD 101. The CPU provided in the HMD 101 executes the program stored in the storage unit, thereby realizing the functions provided by these functional units.

  FIG. 5 is a functional block diagram of the PC 130. The first decompressing unit 505 decompresses the compressed main camera video (video captured by the left main camera 106 and the right main camera 105) received via the communication unit 500. The second decompression unit 510 decompresses the compressed sub camera video received via the communication unit 500. As described above, the received sub-camera image is an image of either or either of the upper sub-camera 107 and the lower sub-camera 108. In addition, the compression rate may be different between the video of the upper sub camera 107 and the video of the lower sub camera 108. In the present embodiment, the first extension unit 505 and the second extension unit 510 are separate blocks, but may be a common block.

  The index detection unit 515 performs image analysis on the expanded sub-camera image and detects an index such as the marker 120. The marker arrangement information 520 is information defining a plurality of markers and their positional relationships, and is constructed in advance by imaging all the markers before starting the MR system. The index arrangement information 520 may be updated according to the analysis of the sub camera video. The position / orientation detection unit 525 detects the position / orientation of the HMD 101 based on the index arrangement information 520 and the index information detected by the index detection unit 515. The CG generation unit 530 generates a CG based on the detected position and orientation of the HMD. The CG combining unit 535 generates a CG combined image by combining the CG generated by the CG generating unit 530 and the main camera image expanded by the first extending unit 505. The CG synthesized video generated by the CG synthesizing unit 535 is compressed by the synthesized video compressing unit 540 using a predetermined encoding method and transmitted to the HMD 101 via the communication unit 500.

  The motion acquisition unit 550 acquires the motion information (movement rotation amount information) output from the motion detection unit 435 of the HMD 101 via the communication unit 500. The priority information generation unit 545 uses the acquired motion information (movement rotation amount information) and the current position / orientation information detected by the position / orientation detection unit 525 to capture the sub-camera image to be transmitted next. Estimate the position and orientation. As described above, the position and orientation of the HMD 101 in the next frame is estimated by adding the integration of motion information (amount of movement and rotation for one frame) to the current position and orientation determined from an index such as a marker. Can do. Then, the priority information generation unit 545 generates priority information of the next sub-camera image to be transmitted based on the estimated position and orientation of the HMD 101 and the index arrangement information 520. For example, it is assumed that the next estimated shooting position of the HMD 101 is P <b> 2 in FIG. 3, and the shooting ranges of the sub camera based on the position and orientation of the HMD 101 are the shooting range 320 and the shooting range 325. In this case, based on the index arrangement information 520, it can be seen that the upper sub camera 107 video includes three markers and the lower sub camera 108 video includes one marker. Therefore, the priority information generation unit 545 generates priority information in which the video priority of the upper sub camera 107 is increased. The generated priority information is transmitted to the HMD 101 via the communication unit 500. Each function unit shown in FIG. 5 is implemented in the apparatus as hardware or software. When these function units are implemented as software, a program for executing these functions is stored in a storage unit (memory such as ROM or RAM) provided in the PC 130. Then, the CPU provided in the PC 130 executes the program stored in the storage unit, thereby realizing the functions provided by these functional units.

  FIG. 6 is a diagram illustrating data generation timings of the sub camera video, the main camera video, and the motion information (movement rotation amount information) in the HMD 101. A period 600 indicates the generation timing of the video frame (N), and a period 605 indicates the generation timing of the video frame (N + 1). For example, when the frame rate is 60 Hz, a video frame is generated about every 16 ms and transmitted to the PC 130. Upper sub camera video data 615 from the upper sub camera 107 and lower sub camera video data 620 from the lower sub camera 108 are transmitted during the period of each video frame. In each video frame period, left main camera video data 625 from the left main camera 106 and right main camera video data 630 from the right main camera 105 are transmitted. Reference numeral 610 denotes video frame (N) data, and reference numeral 611 denotes motion information (movement rotation amount) data sampled while the video frame (N + 1) data is generated. Compared with video data, the amount of data per sample of motion information is small, and the sampling frequency of motion information is higher than the frame rate of video as shown in FIG. In the present embodiment, as shown in FIG. 7, a plurality of pieces of motion information sampled during the generation of one video frame are collected and stored together with video frame data in a communication frame for transmission. Each function unit shown in FIG. 5 is implemented in the apparatus as hardware or software. When these function units are implemented as software, a program for executing these functions is stored in a storage unit (memory such as ROM or RAM) provided in the PC 130. Then, the CPU provided in the PC 130 executes the program stored in the storage unit, thereby realizing the functions provided by these functional units.

FIG. 7 is a diagram illustrating a configuration of a communication frame transmitted from the HMD 101 to the PC 130. Reference numerals 700, 705, 710, 715, and 720 show examples of communication frames when a communication band is sufficiently provided when transmitting a video frame (N). When the bandwidth determination unit 420 determines that the communication bandwidth is sufficient, the present embodiment includes the following communication frames.
Upper sub camera video data 700 from the upper sub camera 107 and lower sub camera video data 705 from the lower sub camera 108;
Left main camera video data 710 from the left main camera 106 and right main camera video data 715 from the right main camera 105,
A plurality of motion information 720 sampled during the generation of one video frame.

  As described above, when the band determination unit 420 determines that a sufficient communication band is provided, the upper and lower sub-camera images are transmitted. In this case, the position / orientation detection unit 525 of the PC 130 detects the position / orientation of the HMD 101 by performing image analysis on the upper sub-camera video data 700 and the lower sub-camera video data 705. Further, the priority information generation unit 545 uses the detected position and orientation and the motion information 720 (movement rotation amount information), and the position and orientation of the HMD 101 at the timing when the sub camera captures the sub camera image to be transmitted next. Is generated, and priority information is generated and transmitted to the HMD 101.

  On the other hand, reference numerals 725, 730, 735, and 740 in FIG. 7 show examples of communication frames when a communication band is insufficient when transmitting a video frame (N + 1). In this case, based on the priority information transmitted from the PC 130, the HMD 101 transmits the lower sub camera video data from the lower sub camera 108 having a higher priority, and does not transmit the upper sub camera video data from the upper sub camera 107. To control. Accordingly, in the (N + 1) th video frame, the lower sub-camera video data 725, the left main camera video data 730 and the right main camera video data 735, and a plurality of motion information 740 sampled during the generation of the (N + 1) th video frame are transmitted. Is done. In addition, although the structure using time division transmission is shown in the said embodiment, it is clear to those skilled in the art that the structure using other multiplexing systems, such as frequency division transmission, may be sufficient.

FIG. 8 is a diagram illustrating another configuration example of a communication frame transmitted from the HMD 101 to the PC 130. In FIG. 8, unlike FIG. 7, the video transmitted by changing the compression rate when compressing the upper and lower sub-camera video based on the priority information, instead of selecting either the upper or lower sub-camera video. The data code amount is changed. The upper sub-camera video data 800 is compressed data obtained by compressing video data captured by the upper sub-camera 107 indicated by the priority information at a high compression rate. On the other hand, the lower sub-camera video data 805 is compressed data obtained by compressing the video data captured by the lower sub-camera 108 whose priority is indicated by the priority information at a low compression rate. It is. The code amounts of the upper sub-camera video data 800 and the lower sub-camera video data 805 are set according to the communication band shortage state (shortage amount) determined by the band determination unit 420 and priority information. As described above, when it is determined that the communication standby is insufficient, when determining the code amount ratio according to the priority information,
-The total code amount of the sub-camera video is the range that can be transmitted in the usable communication band,
The compression ratio is set so that the ratio of the code amount of the upper sub-camera video and the lower sub-camera video matches the ratio indicated by the priority information.

  FIG. 9 is a sequence diagram showing the contents of data transmission / reception between the HMD 101 and the PC 130 of this embodiment. This sequence diagram shows a situation in which the communication band is sufficient during data transmission of the video frame (N), and the communication band is insufficient during data transmission after the video frame (N + 1). Further, it is assumed that the HMD 101 is positioned at P1 in FIG. 3 when imaging the video frame (N + 1), and that the HMD 101 has moved to P2 in FIG. 3 when imaging the video frame (N + 2).

  First, the HMD 101 transmits the sub-camera image of the captured image frame (N) (S900). At this time, since the communication band is sufficient, both the upper and lower sub-camera images are transmitted. Next, the HMD 101 transmits the main camera video of the video frame (N) (S905). Next, the HMD 101 transmits movement rotation amount information (motion information) sampled at the time of generating the video frame (N) (S910). The PC 130 detects the current position and orientation of the HMD 101 based on the upper and lower sub-camera images, and generates and transmits a CG composite image based on the detected position and orientation (S915). Further, based on the current position and orientation of the HMD 101 and the movement rotation amount information, the position and orientation of the HMD 101 at the timing of shooting the video frame (N + 1) is estimated. The PC 130 recognizes that more markers are arranged in the imaging range 315 of the lower sub camera 108 than the imaging range 310 of the upper sub camera 107 from the estimated position and orientation (P1) of the HMD 101. As a result, the PC 130 transmits priority information in which the priority of the video of the lower sub camera 108 is higher than the priority of the video of the upper sub camera 107 to the HMD 101 (S920).

  Next, the HMD 101 detects that the communication band is insufficient during data transmission of the video frame (N + 1). Therefore, the HMD 101 selects the video of the lower sub camera 108 having a higher priority as a transmission target according to the priority information received from the PC 130, and transmits it as the sub camera video of the video frame (N + 1) (S925). Next, the HMD 101 transmits the main camera video of the video frame (N + 1) (S930). Next, the HMD 101 collectively transmits the movement rotation amount information (motion information) sampled when the video frame (N + 1) is generated (S935). The PC 130 detects the current position and orientation of the HMD 101 based on the video data of the lower sub camera 108 transmitted in S925, and generates and transmits a CG composite video based on the detected position and orientation (S940). In addition, the PC 130 estimates the position and orientation of the HMD 101 at the timing of capturing the next N + 1 frame image based on the current position and orientation of the HMD 101 and the moving rotation amount information. The PC 130 recognizes that more markers are arranged in the imaging range 320 of the upper sub camera 107 than the imaging range 325 of the lower sub camera 108 from the estimated position and orientation (P2) of the HMD 101. Accordingly, the PC 130 generates priority information in which the priority of the video of the upper sub camera 107 is higher than the priority of the video of the lower sub camera 108, and transmits the priority information to the HMD 101 (S945).

  Next, at the time of data transmission of the video frame (N + 2), the HMD 101 detects that the communication band shortage continues. Therefore, the HMD 101 selects the video of the upper sub camera 107 having a higher priority according to the priority information received from the PC 130, and transmits it as the sub camera video of the video frame (N + 2) (S950). The processing of S955 to S970 is the same as the processing of S930 to S970 except that the video data of the upper sub camera 107 is used.

  As described above, each time a video frame and moving rotation amount information are transmitted, the PC 130 side estimates the position of the HMD 101 at the time of transmitting the next video frame, thereby determining the priority of the sub-camera video and suitable for position detection. The sub-camera video is transmitted with priority. Hereinafter, processing of each functional unit shown in FIGS. 4 and 5 for realizing the operations of the HMD 101 and the PC 130 described with reference to FIG. 9 will be described with reference to the flowcharts of FIGS. 10 and 11.

  FIG. 10 is a flowchart showing the operation of the HMD 101 in units of video frames. First, in S1000, the HMD 101 receives the movement rotation amount information (motion information) from the motion detection unit 435 (inertial sensor), the upper and lower sub camera images from the second imaging unit 410 (sub camera), and the main camera image from the first. Obtained from the imaging unit 400 (main camera). In step S <b> 1005, the band determination unit 420 acquires a communication band that can be currently used by the communication unit 406.

  In step S <b> 1006, the band determination unit 420 determines whether the usable communication band is insufficient based on the communication band determined by the band determination unit 420. If it is determined that the communication band is sufficient, the process branches to S1011 and the ratio determination unit 425 equalizes the ratio of the transmission data amount of the upper and lower sub camera images. On the other hand, if it is determined that the communication band is insufficient, the process branches to S1010. In S1010, the ratio determination unit 425 determines the ratio of the transmission data amount of the upper and lower sub-camera videos from the communication band acquired by the band determination unit 420 and the priority information acquired by the priority information acquisition unit 415 during the previous video frame processing. To decide. In that case, a higher priority sub-camera image indicated by the priority information can be transmitted with higher priority by increasing the ratio of the amount of transmission data. When the transmission ratio is 1: 0, one of the sub camera images with high priority is transmitted from the HMD 101 to the PC 130, and the other sub camera image is not transmitted.

  In step S <b> 1015, the second compression unit 430 compresses the sub camera video according to the transmission data amount ratio determined by the ratio determination unit 425. In step S <b> 1020, the first compression unit 405 compresses the main camera video acquired from the first imaging unit 400. In S1025, the communication unit 406 transmits to the PC 130 the movement rotation amount information acquired in S1000, the sub camera video compressed in S1015, and the main camera video compressed in S1020. In step S1030, in step S1025, the communication unit 406 receives from the PC 130 the CG composite video and priority information generated based on the information transmitted to the PC 130. The priority information is used in S1010 during the next video frame processing. In S1035, the composite video decompression unit 440 decompresses the CG composite video received in S1030, and in S1040, the display unit 445 displays the decompressed CG composite video.

  FIG. 11 is a flowchart showing the operation of the PC 130 in units of video frames. First, in S1100, the communication unit 500 receives the movement rotation amount information (motion information), the sub camera video, and the main camera video transmitted by the HMD 101 in S1025. Note that the received sub-camera image may be either upper or lower or one of upper and lower. The main camera video and the sub camera video are supplied to the first decompression unit 505 and the second decompression unit 510 via the communication unit 500. Further, the movement rotation amount information is acquired as motion information by the motion acquisition unit 550 via the communication unit 500. In S1105, the second decompression unit 510 decompresses the sub camera video received by the communication unit 500, and in S1110, the first decompression unit 505 decompresses the main camera video received by the communication unit 500.

  In step S1115, the index detection unit 515 analyzes the sub-camera video expanded in step S1105 and detects an index such as a marker. In S1120, the position / orientation detection unit 525 detects the current position / orientation of the HMD 101 based on the marker detected in S1105 and the index arrangement information 520 that defines the positional relationship between the markers. In S1125, the priority information generation unit 545 determines the priority of the upper and lower sub-camera images based on the current position and orientation of the HMD 101 and the movement rotation amount acquired by the motion acquisition unit 550, and generates priority information. And send. A method for determining the priority of the upper and lower sub-camera images will be described. The priority information generation unit 545 first determines the position and orientation of the HMD 101 at the timing when the sub camera performs imaging in order to obtain the sub camera image to be transmitted next, the current position and orientation of the HMD 101, and the motion acquisition unit 550. Estimated from the acquired movement rotation amount. Then, the priority information generation unit 545 determines the priority of the upper and lower sub camera images from the estimated position and orientation of the HMD 101 and the index arrangement information 520, and generates priority information. In the present embodiment, the priority of the sub camera in which many markers (indexes) are included in the imaging range determined from the estimated position and orientation is set high. Note that the method of determining an image suitable for position and orientation detection is not limited to this. For example, in the case of an MR system that uses natural feature points for position and orientation detection instead of markers as shown in FIG. The priority may be determined by a number.

  In S1130, the CG generation unit 530 generates a CG based on the position and orientation of the HMD detected in S1120. Then, the CG combining unit 535 combines the main camera image expanded in S1110 and the generated CG based on the position and orientation of the HMD detected in S1120 to generate a CG combined image. In step S <b> 1135, the synthesized video compression unit 540 compresses the CG synthesized video generated by the CG synthesis unit 535. In S1140, the communication unit 500 transmits the CG composite video compressed by the composite video compression unit 540 and the priority information generated by the priority information generation unit 545 in S1125 to the HMD 101.

  As described above, according to the first embodiment, the sub-camera image suitable for the position detection of the HMD 101 is preferentially transmitted according to the available communication band, so even if the communication band is insufficient It becomes possible to maintain detection accuracy. Furthermore, by reducing the video data to be transmitted and reducing the data transmission time of the HMD, it becomes possible to suppress the power consumption of the HMD.

Second Embodiment
In the first embodiment, the priority of the sub camera video to be transmitted is determined on the PC 130 side, but the present invention is not limited to this. In the second embodiment, a configuration in which the priority of the sub camera video is determined on the HMD 101 side by transmitting the position and orientation information detected on the PC 130 side to the HMD 101 will be described.

  FIG. 12 is a block diagram illustrating a functional configuration of the HMD 101 in the second embodiment. Compared with the functional configuration of the first embodiment (FIG. 4), the content of the communication unit 406a is changed, and a position / orientation acquisition unit 1205, index placement information 1210, and a priority determination unit 1215 are added. The position / orientation acquisition unit 1205 acquires position / orientation information detected on the PC 130 side via the communication unit 406a. This position and orientation information indicates the position and orientation of the HMD 101 at the time of the previous (immediately preceding) video frame transmission.

  Like the marker placement information 520 (FIG. 5) on the PC 130 side, the marker placement information 1210 is information defining a plurality of markers and their positional relationships, and is constructed in advance by imaging all the markers before the execution of the MR system. Is done. The index arrangement information may be updated according to the analysis of the sub camera video on the PC 130 side and transmitted to the HMD 101 side at a predetermined cycle. The priority determination unit 1215 determines the priority of the video of the sub camera by the same processing as the priority information generation unit 545 (FIG. 5). That is, the priority determination unit 1215 determines the current video based on the motion information (movement rotation amount information) acquired by the motion detection unit 435, the position / orientation information acquired by the position / orientation acquisition unit 1205, and the index arrangement information 1210. Estimate the position and orientation during frame transmission. And the priority determination part 1215 determines the priority of the sub camera image | video transmitted in the present video frame transmission based on the estimated position and orientation. Thus, unlike the first embodiment, the second embodiment is configured to determine the priority of the sub camera video on the HMD 101 side.

  FIG. 13 is a block diagram illustrating a functional configuration example of the PC 130 in the second embodiment. Compared with the functional configuration (FIG. 5) in the first embodiment, the motion acquisition unit 550 and the priority information generation unit 545 are omitted, and the contents of the position / orientation detection unit 525a and the communication unit 500a are changed. After detecting the position and orientation, the position and orientation detection unit 525a transmits the position and orientation information to the HMD 101 via the communication unit 500a. As described above, in the second embodiment, the PC 130 transmits necessary position and orientation information to the HMD 101 so that the priority of the sub camera video is not determined on the PC 130 side but on the HMD 101 side.

  FIG. 14 is a sequence diagram showing data transmission between the HMD 101 and the PC 130 in the second embodiment. This sequence diagram shows a situation in which the communication band is sufficient during data transmission of the video frame (N), and the communication band is insufficient during data transmission after the video frame (N + 1). Further, it is assumed that the HMD 101 is located at P1 in FIG. 3 when imaging the video frame (N + 1), and has moved to P2 in FIG. 3 when imaging the video frame (N + 2).

  The HMD 101 transmits the sub-camera image of the captured image frame (N) to the PC 130 (S1400). At this time, since the communication band is sufficient, both the upper and lower sub-camera images are transmitted. Next, the HMD 101 transmits the main camera video of the video frame (N) to the PC 130 (S1405). The PC 130 detects the position and orientation of the HMD 101 based on the upper and lower sub-camera images, generates a CG composite video based on the detected position and orientation, and transmits the generated CG composite video to the HMD 101 (S1410). Further, the PC 130 transmits the detected position and orientation information of the HMD 101 (S1415).

  Next, the HMD 101 detects that the communication band is insufficient during data transmission of the video frame (N + 1). Therefore, the HMD 101 estimates the current position and orientation of the HMD 101 based on the position and orientation transmitted from the PC 130 in S1415 and the movement rotation amount information accumulated in the HMD 101. The HMD 101 recognizes that the estimated position and orientation is P1, and in P1, more markers are arranged in the imaging range 315 of the lower sub camera 108 than the imaging range 310 of the upper sub camera 107. Increase camera image priority. In the present embodiment, the HMD 101 selects only the video of the lower sub camera 108 and transmits it to the PC 130 as the sub camera video of the video frame (N + 1) (S1420). Next, the HMD 101 transmits the main camera video of the video frame (N + 1) to the PC 130 (S1425). The PC 130 detects the position and orientation of the HMD 101 based on the lower sub-camera image transmitted in S1420, generates a CG composite image based on the detected position and orientation, and transmits this to the HMD 101 (S1430). Further, the PC 130 transmits the detected position and orientation information of the HMD 101 to the HMD 101 (S1435).

  Next, during data transmission of the video frame (N + 2), the HMD 101 detects that a shortage of communication bandwidth continues. Therefore, the HMD 101 estimates the current position and orientation of the HMD 101 based on the position and orientation transmitted from the PC 130 in S1415 and the movement rotation amount information accumulated in the HMD 101. When the HMD 101 recognizes that the current position is P2, the priority of the video of the upper sub camera 107 is higher because more markers are arranged in the imaging range 320 of the upper sub camera 107 than the imaging range 325 of the lower sub camera 108. The degree is increased (S1346). In this embodiment, the HMD 101 selects only the upper sub-camera video and transmits it as a sub-camera video of the video frame (N + 2) (S1440). The processing of S1445 to S1455 is the same as the processing of S1425 to S1435 except that the video data of the upper sub camera 107 is used.

  As described above, each time the HMD 101 of the second embodiment transmits a video frame, the HMD 101 determines its own current position from the previous position information of the HMD 101 received from the PC 130 and the current movement information (movement rotation amount information). presume. Then, the HMD 101 determines the priority of the sub camera video from the estimated current position, and preferentially transmits the sub camera video suitable for position detection.

  Hereinafter, processing of each functional unit shown in FIGS. 12 and 13 for realizing the operations of the HMD 101 and the PC 130 described with reference to FIG. 14 will be described with reference to the flowcharts of FIGS. 15 and 16. FIG. 15 is a flowchart showing the operation of the HMD 101 in units of video frames. In S <b> 1500, the HMD 101 transmits the movement rotation amount information from the motion detection unit 435 (inertial sensor), the upper and lower sub camera images from the second imaging unit 410 (sub camera), and the main camera image from the first imaging unit 400 (main camera). ) In step S1505, the band determination unit 420 acquires a currently available communication band.

  In step S1506, the bandwidth determination unit 420 determines whether the currently available communication bandwidth is insufficient. If it is determined that the communication band is sufficient, the process proceeds to S1511, and the ratio determination unit 425 sets the ratio of the transmission data amount of the upper and lower sub-camera images equally. If it is determined that the communication band is insufficient, the process advances to step S1510, and the priority determination unit 1215 determines the priority of the images of the upper and lower sub cameras. In the priority determination, the priority determination unit 1215 first determines the current position and orientation of the HMD 101 from the movement rotation amount information acquired in S1500 and the position and orientation information of the previous HMD 101 received from the PC during the previous video frame processing. presume. Then, from the estimated position and orientation and the index arrangement information 1210, the priority of the images of the upper and lower sub cameras is determined based on the amount of the marker existing in each imaging range of the upper and lower sub cameras. In step S1515, the ratio determining unit 425 determines the ratio of the transmission data amount of the upper and lower sub-camera videos from the communication band acquired in step S1505 and the priority determined in step S1510.

  Next, in S1520, the second compression unit 430 compresses the sub camera image from the second imaging unit 410 according to the ratio of the transmission data amount determined by the ratio determination unit 425 (S1515). In step S1525, the first compression unit 405 compresses the main camera video from the first imaging unit 400. In step S1530, the communication unit 406a transmits the sub camera video compressed in step S1520 and the main camera video compressed in step S1525 to the PC 130. In S1535, the communication unit 406a receives from the PC 130 the position and orientation information of the HMD 101 and the CG synthesized video determined based on the sub camera video transmitted to the PC 130 (S1530). This position / orientation information is used in S1510 during the next video frame processing. In S1540, the composite video decompression unit 440 decompresses the received CG composite video, and in S1545, the display unit 445 displays the CG composite video.

  FIG. 16 is a flowchart showing the operation of the PC 130 in units of video frames. In S1600, the communication unit 500a receives the sub camera video and the main camera video from the HMD 101. The sub-camera image received here may be either up-down or up-down. In S1605, the second decompression unit 510 decompresses the received sub camera video, and in S1610, the first decompression unit 505 decompresses the received main camera video. In step S1615, the index detection unit 515 analyzes the sub camera video expanded in step S1605 and detects an index such as a marker.

  In S1620, the position and orientation detection unit 525a detects the position and orientation of the HMD 101 from the marker detected in S1615 and the index arrangement information 520 that defines the positional relationship between the markers. In S1625, the CG generation unit 530 generates a CG based on the position and orientation of the HMD 101 detected in S1620, and the CG synthesis unit 535 generates the main camera video and CG expanded in S1610 in the HMD 101 detected in S1620. Compositing based on the position and orientation. In S1630, the composite video compression unit 540 compresses the CG composite video, and in S1635, transmits the CG composite video compressed in S1630 and the position and orientation information detected in S1620 to the HMD 101.

  As described above, in the second embodiment, the video priority of the sub camera is determined on the HMD 101 side. Also in the second embodiment, similarly to the first embodiment, the sub camera video suitable for position detection can be preferentially transmitted to the HMD 101 in accordance with the available communication band. Therefore, it is possible to maintain position detection accuracy even when the communication band is insufficient. Furthermore, by reducing the video data to be transmitted and reducing the data transmission time of the HMD, it becomes possible to suppress the power consumption of the HMD.

  In each of the above embodiments, a configuration has been described in which a plurality of sub camera images are obtained from a plurality of sub cameras. That is, a plurality of sub camera images obtained from a plurality of sub cameras become a plurality of images corresponding to different spaces for detecting the position and orientation of the HMD 101. However, the present invention is not limited to this. For example, a video obtained from one sub camera may be divided into a plurality of partial videos to obtain a plurality of sub camera videos. In this case, the plurality of partial videos become a plurality of videos corresponding to different spaces for detecting the position and orientation of the HMD 101.

  In the above embodiment, the number of markers such as markers and natural features included in the imaging range is used as a condition in order to determine the priority of the sub-camera video, but instead of or in addition to this, the imaging range Other conditions such as the size of the index and the sharpness included in the image may be used as appropriate. Further, the sub-camera image selection process described above is executed when the available communication bandwidth is low, but it may be operated under other conditions such as when the battery of the HMD 101 is low. You may comprise. Alternatively, the HMD 101 may be provided with an operation mode such as a power saving mode, and the operation of this embodiment may be performed only when the HMD wearer selects this mode.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: HMD, 105: Right main camera, 106: Left main camera 106, 107: Upper sub camera, 108: Lower sub camera, 109: Inertial sensor, 120: Marker, 125, 135: Communication link, 130: PC,

Claims (29)

A video processing system comprising a portable information processing device and a video processing device that synthesizes a virtual image with video received from the information processing device,
First transmission means for transmitting a first video used for determining the position and orientation of the information processing apparatus from the information processing apparatus to the video processing apparatus via a communication means; and the first video is , Including multiple videos corresponding to different spaces,
Determining means for determining a position and orientation of the information processing device based on the first video;
Based on the communication band that can be used by the first transmission unit of the communication unit and the position and orientation determined by the determination unit, the data amount of the plurality of videos transmitted by the first transmission unit And a control means for controlling the ratio. The first image includes an index for detecting a position and orientation;
The control means determines the amount of an index included in each of the plurality of videos based on the position and orientation of the information processing apparatus determined by the determination means, and the ratio is determined according to the determined index amount. The video processing system according to claim 1, wherein the video processing system is controlled.   The video processing system according to claim 2, wherein the control unit sets the ratio so that the amount of transmitted data increases as the amount of the index included in the video increases.   The determination unit determines a position and orientation of the information processing apparatus at a timing when the first video transmitted next by the first transmission unit is captured. The video processing system according to claim 1. Further comprising an acquisition means for acquiring movement information indicating a movement rotation amount of the information processing apparatus;
The determination unit determines a position and orientation of the information processing apparatus at the timing based on the first video transmitted by the first transmission unit and the motion information acquired by the acquisition unit. The video processing system according to claim 4, wherein:   6. The video processing system according to claim 1, wherein the information processing apparatus includes a plurality of photographing devices for photographing the plurality of videos included in the first video. 7. .   7. The information processing apparatus according to claim 1, wherein the information processing apparatus obtains the plurality of images by dividing the first image captured by one image capturing device into a plurality of partial images. The video processing system described in 1.   The video processing system according to claim 1, wherein the information processing apparatus is a head mounted display. Second transmission means for transmitting, via the communication means, a second video in which the virtual image is synthesized from the information processing apparatus to the video processing apparatus;
The said video processing apparatus is further provided with the production | generation means which produces | generates the synthetic | combination image | video by which the virtual image was synthesize | combined with the said 2nd image | video based on the position and orientation determined by the said determination means. The video processing system according to any one of 1 to 8. A portable information processing apparatus,
A first imaging unit that captures a first video used to determine a position and orientation of the information processing apparatus; and the first video includes a plurality of videos corresponding to different spaces;
First transmission means for transmitting the first video to an external device via communication means;
Acquisition means for acquiring priority information representing the priority of each of the plurality of videos generated based on the first video from the external device;
Control for controlling a ratio of data amounts of the plurality of videos transmitted by the first transmission unit based on a communication band that can be used by the first transmission unit of the communication unit and the priority information. And an information processing apparatus. A portable information processing apparatus,
A first imaging unit that captures a first video used to determine a position and orientation of the information processing apparatus; and the first video includes a plurality of videos corresponding to different spaces;
First transmission means for transmitting the first video to an external device via communication means;
Obtaining means for obtaining a position and orientation of the information processing apparatus determined based on the first video from the external device;
Generation that generates priority information indicating the priority of each of the plurality of videos based on the position and orientation acquired by the acquisition unit and a communication band that can be used by the first transmission unit of the communication unit Means,
Control for controlling a ratio of data amounts of the plurality of videos transmitted by the first transmission unit based on a communication band that can be used by the first transmission unit of the communication unit and the priority information. And an information processing apparatus. A second photographing means for photographing a second video to be combined with the virtual image;
The information processing apparatus according to claim 10, further comprising: a second transmission unit that transmits the second video to the external device via the communication unit. A detecting means for detecting the amount of rotation of the information processing apparatus;
The information processing apparatus according to claim 10, wherein the first transmission unit further transmits motion information indicating the amount of movement rotation detected by the detection unit to the external device. A detecting means for detecting the amount of rotation of the information processing apparatus;
The generating means includes
Based on the position and orientation acquired by the acquisition unit and the motion information indicating the amount of movement rotation detected by the detection unit, the information processing at a timing when the first video to be transmitted next is captured. Estimate the position and orientation of the device,
The information processing apparatus according to claim 11, wherein the priority information is generated based on the estimated position and orientation and the available communication band.   The information according to any one of claims 10 to 14, wherein the control unit selects a video to be transmitted by the first transmission unit from the plurality of videos based on the priority information. Processing equipment. The control means determines a compression rate for each of the plurality of videos based on the priority information, compresses each of the plurality of videos with the determined compression rate, and obtains compressed data,
The information processing apparatus according to claim 10, wherein the first transmission unit transmits the compressed data.   The information according to any one of claims 10 to 16, wherein the first photographing unit includes a plurality of photographing devices for photographing the plurality of videos included in the first video. Processing equipment.   The information processing apparatus according to any one of claims 10 to 17, wherein the first photographing unit obtains the plurality of videos by dividing the photographed video into a plurality of partial videos. A video processing device for processing video transmitted from an external device,
The first receiving means for receiving the first video used for determining the position and orientation of the external device from the external device via the communication means, and the first video corresponds to different spaces. Including multiple videos,
Determining means for determining a position and orientation of the external device based on the first video;
Based on the communication bandwidth of the communication means that can be used by the first receiving means and the position and orientation determined by the determining means, the priorities of the plurality of videos received by the first receiving means are determined. Generating means for generating priority information to indicate;
Transmitting means for transmitting the priority information to the external device. Second receiving means for receiving, via the communication means, a second video to be combined with a virtual image from the external device;
The video processing according to claim 19, further comprising: a synthesis unit that generates a synthesized video in which a virtual image is synthesized with the second video based on the position and orientation determined by the determination unit. apparatus. The first image includes an index for detecting a position and orientation;
The generation unit determines an amount of an index included in each of the plurality of videos based on the position and orientation of the external device determined by the determination unit, and the priority based on the determined amount of the index 21. The video processing apparatus according to claim 19 or 20, wherein information is generated.   The video processing apparatus according to claim 21, wherein the generation unit generates the priority information so that the priority is higher as the amount of the index included in the video is larger.   The video processing according to any one of claims 19 to 22, wherein the determination unit determines a position and orientation of the external device at a timing when the first video to be transmitted next is captured. apparatus. It further comprises acquisition means for acquiring movement information indicating the movement of the external device,
The determination unit determines a position and orientation of the external device at the timing based on the first video received by the first reception unit and the motion information acquired by the acquisition unit. The video processing apparatus according to claim 23. A control method for a video processing system, comprising: a portable information processing device; and a video processing device that synthesizes a virtual image with video received from the information processing device,
A first transmission step of transmitting a first video used for determining a position and orientation of the information processing device from the information processing device to the video processing device via a communication unit; and , Including multiple videos corresponding to different spaces,
A determination step of determining a position and orientation of the information processing device based on the first video;
Based on the communication bandwidth available in the first transmission step of the communication means and the position and orientation determined in the determination step, the data amount of the plurality of videos transmitted in the first transmission step And a control step for controlling the ratio. A control method for a portable information processing apparatus,
A first shooting step of shooting a first video used to determine a position and orientation of the information processing apparatus; and the first video includes a plurality of videos corresponding to different spaces;
A first transmission step of transmitting the first video to an external device via communication means;
An acquisition step of acquiring priority information representing the priority of each of the plurality of videos generated based on the first video from the external device;
Control for controlling the ratio of the data amount of the plurality of videos transmitted in the first transmission step based on the communication band available in the first transmission step of the communication means and the priority information And a method for controlling the information processing apparatus. A control method for a portable information processing apparatus,
A first shooting step of shooting a first video used to determine a position and orientation of the information processing apparatus; and the first video includes a plurality of videos corresponding to different spaces;
A first transmission step of transmitting the first video to an external device via communication means;
An acquisition step of acquiring the position and orientation of the information processing device determined based on the first video from the external device;
Generation for generating priority information indicating the priority of each of the plurality of videos based on the position and orientation acquired in the acquisition step and the communication band available in the first transmission step of the communication unit Process,
Control for controlling the ratio of the data amount of the plurality of videos transmitted in the first transmission step based on the communication band available in the first transmission step of the communication means and the priority information And a method for controlling the information processing apparatus. A control method of a video processing device for processing video transmitted from an external device,
A first reception step of receiving a first video used for determining a position and orientation of the external device from the external device via a communication unit, and the first video correspond to different spaces. Including multiple videos,
A determination step of determining a position and orientation of the external device based on the first video;
Based on the communication bandwidth of the communication means that can be used in the first reception step and the position and orientation determined in the determination step, the priorities of the plurality of videos received in the first reception step are determined. A generation step for generating priority information to be shown;
And a transmission step of transmitting the priority information to the external device.   A program for causing a computer to execute each step of the control method according to any one of claims 25 to 28.

Images