JP2017055397A - Image processing apparatus, image composing device, image processing system, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an MR (Mixed Reality) device to perform highly accurate position and posture estimation, by the combination of an imaging system, which has a small deterioration with respect to movement, with an imaging system which can image a high definition video though having a large deterioration with respect to movement, enabling high definition video viewing.SOLUTION: An image processing apparatus outputs a first video using first imaging means, having a relatively small image deterioration when a subject moves, and also outputs a second video using second imaging means having a relatively large image deterioration when a subject moves. The image processing apparatus analyzes the first video to generate the position and the posture information of the image processing apparatus. The image processing apparatus then draws a CG object on the second video in a superposed manner at a position determined based on the position and the posture information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an image composition device, an image processing system, an image processing method, and a program.

  In recent years, a technique for estimating a three-dimensional position and orientation from a moving camera image called Visual SLAM (Multiple Localization and Mapping) has been put into practical use. As this application, there is MR (Mixed Reality) / AR (Arranged Reality) that displays a virtually existing three-dimensional computer graphic object on a video based on the position and orientation of the camera. Some techniques for estimating the position and orientation of a camera from an image include those that use markers and those that do not. In both cases, a marker or a natural object is identified between frames and its movement is tracked (hereinafter, this tracking is referred to as tracking) to estimate the position and orientation of the camera in the three-dimensional space. One position / orientation estimation method using markers is disclosed in Patent Document 1. One of the position and orientation estimation methods that do not use a marker is Non-Patent Document 2 (common name: PTAM). In MR, a map that indicates the three-dimensional position of a marker or subject called an environment map is generated from the estimated position and orientation of the camera, the position and orientation of the CG object are determined using this environment map, and the input video Superimpose on. Thereby, it is possible to obtain an image in which a CG object exists in the real space. At this time, whether or not the CG can be superimposed at the correct position depends on the tracking accuracy, and the tracking accuracy greatly depends on the characteristics of each frame image of the video.

  The characteristics of the frame image depend on the sensor and the driving condition of the sensor. For example, when a rolling shutter sensor, which is a method often used in CMOS sensors, is used, distortion called rolling shutter distortion occurs when there is a moving subject in the scene or when the camera pans. This distortion reduces the accuracy of identifying markers and subjects between frames, and as a result, reduces tracking accuracy and position / orientation estimation accuracy. On the other hand, when a global shutter sensor typified by a CCD is used, rolling shutter distortion does not occur. However, in general, a global shutter sensor generally requires a high driving voltage, and it is difficult to achieve high resolution and high frame rate. Further, even with the same rolling shutter sensor, rolling shutter distortion can be greatly reduced by increasing the sensor driving speed.

Hirokazu Kato and Mark Billinghurst, “Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System”, Proceedings. 2nd IEEE and ACM International Workshop on Augmented Reality '99 Georg Klein and David Murray, “Parallel Tracking and Mapping on a Camera Phone”, In Proc. International Symposium on Mixed and Augmented Reality (ISMAR'09, Orlando)

  In the MR apparatus, a rolling shutter distortion occurs in the image of the rolling shutter sensor, and the position / orientation estimation accuracy decreases. When using a global shutter sensor, it is difficult to handle images with sufficient resolution and high frame at low cost.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement. That is,
A first imaging means for outputting a first video and relatively less image degradation caused by movement of the subject;
A second imaging means for outputting a second video, wherein image degradation due to movement of the subject is relatively large;
Estimating means for analyzing the first video and generating position and orientation information of the image processing device;
Drawing means for drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information;
It is characterized by providing.

  Both high-precision position and orientation estimation and high-definition viewing video generation are compatible.

FIG. 3 is a diagram illustrating the configuration of an MR apparatus according to the first embodiment. The figure explaining MR apparatus composition in Embodiment 2. The figure explaining MR apparatus composition in Embodiment 3. FIG. 10 is a diagram for explaining a method of detecting a motion amount in the third embodiment. Diagram explaining geometric transformation of frame image FIG. 10 is a diagram for explaining a modification of the method for detecting a motion amount in the third embodiment. The figure explaining MR apparatus composition in Embodiment 4. FIG. 6 is a diagram for explaining an MR system configuration according to a fifth embodiment. The figure explaining MR apparatus structure in Embodiment 6. FIG. FIG. 10 is a diagram for explaining an MR processing method according to a sixth embodiment. FIG. 10 is a diagram for explaining a modification of the MR processing method according to the sixth embodiment. FIG. 10 is a diagram for explaining an MR system configuration according to a seventh embodiment. FIG. 10 is a diagram illustrating an MR processing method according to a seventh embodiment.

[Embodiment 1]
An MR apparatus 100 that is an image processing apparatus according to the first embodiment of the present invention will be described. In the present embodiment, it is assumed that processing is executed on an MR apparatus including an imaging unit and a display unit. Hereinafter, the configuration of the MR apparatus and the operation of each module will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration of an MR apparatus (for example, a head mounted display) according to the first embodiment.

  The first sensor 101 captures the first video. In the present embodiment, the first sensor 101 is a global shutter sensor capable of imaging 960 pixels vertically, 540 pixels horizontally, and 60 fps. In the present embodiment, a lens unit is connected to the sensor, and a sensor signal can be obtained by capturing an image composed of continuous images. The resolution and frame rate are not limited to these values.

  A first ISP (image signal processor) 102 converts a sensor signal obtained by the first sensor 101 into an image. In the present embodiment, the first ISP 102 is a module having a function of outputting image data or an encoded stream obtained by encoding image data. Specifically, the first ISP 102 has a complex image processing function such as generating an RGB image from the sensor signal, enlarging or reducing the image, and encoding a video. The RAM required for these processes is built in the first ISP 102, but an external RAM may be further connected to the first ISP 102.

  The position / orientation estimation unit 103 estimates the position / orientation of the MR apparatus 100 using the video obtained by the first ISP 102. The position / orientation estimation unit 103 can estimate the position / orientation of the second sensor 104 mounted on the MR apparatus 100, for example. In the present embodiment, the position / orientation estimation unit 103 is a CPU incorporating a ROM (read-only memory) and a RAM (random access memory). The CPU performs position and orientation estimation processing by operating while using the RAM as a work area according to the position and orientation estimation program stored in the ROM. As the position / orientation estimation program, a program according to the method described in Non-Patent Document 1 can be used. Note that the position / orientation estimation unit 103 may be a dedicated HW (hardware). Further, the position / orientation estimation processing is not limited to the one according to the method of Non-Patent Document 1, and can be performed according to the Visual SLAM method as described in Non-Patent Document 2, for example.

  The first sensor 101 captures the second video. In the present embodiment, the second sensor 104 is a rolling shutter sensor that can capture 1920 pixels vertically, 1080 pixels horizontally, and 60 fps. In the present embodiment, a lens unit is connected to the sensor, and a sensor signal can be obtained by capturing an image composed of continuous images. The resolution and frame rate are not limited to these values.

  The second ISP 105 converts the sensor signal obtained by the second sensor 104 into an image. The second ISP 105 can have the same function as the first ISP 102.

  The CG drawing unit 106 generates a composite image using the image output from the second ISP 105. In the present embodiment, the CG drawing unit 106 holds three-dimensional (computer graphic) object information, and generates a composite image by drawing a virtual CG object on the image. For example, the CG drawing unit 106 can render a virtual CG object according to the CG object information and superimpose the obtained virtual CG object on the image. At this time, the superimposed virtual CG object and the position at which the virtual CG object is superimposed are controlled according to the position and orientation of the MR apparatus 100 estimated by the position and orientation estimation unit 103. For example, the CG rendering unit 106 renders a virtual CG object by placing a viewpoint at a position according to the estimated position and orientation of the MR apparatus 100, and superimposes the obtained virtual image on the image output from the second ISP 105. Thus, a composite image can be generated.

  The display unit 107 displays the composite image generated by the CG drawing unit 106. A user of the MR apparatus 100 can view an image via the display unit 107.

  In the present embodiment, it is assumed that the sensors 101 and 102 are fixedly arranged at positions close enough to capture an image with the same angle of view. Further, a half mirror may be inserted in the optical path so that the sensors 101 and 102 can capture images with the same angle of view. It is not essential that the angle of view is the same. When the angle of view is different, for example, the position / orientation estimation unit 103 may correct the difference in the angle of view, or the ISPs 102 and 105 perform geometric correction processing so that the angle of view of the images output from each of them matches. The image may be output after this is done.

  Hereinafter, the operation of the entire MR apparatus 100 will be described. In FIG. 1, arrows indicate the flow of main data. Although control signals and the like can be communicated bidirectionally between the respective units, description of such control signals and the like is omitted.

  A signal from the sensor 101 is output to the ISP 102. The ISP 102 generates an RGB image and outputs the RGB image as a signal to the position / orientation estimation unit 103. In this specification, a combination of a sensor and an ISP is called an imaging system. An imaging system including the sensor 101 and the ISP 102 is referred to as an analytic video imaging system. Furthermore, a video generated by the analysis video imaging system is called an analysis video.

  The position and orientation estimation unit 103 analyzes the analysis image and generates position and orientation information. In the present embodiment, the position / orientation estimation unit 103 generates position / orientation information indicating the position / orientation of the MR apparatus 100 in a three-dimensional space, generates an environment map based on the position / orientation information, Is output to the CG rendering unit 106.

  A signal from the sensor 104 is output to the ISP 105. The ISP 105 generates an RGB image and outputs the RGB image as a signal to the CG drawing unit 106. In this specification, an imaging system including the sensor 104 and the ISP 105 is referred to as a viewing video imaging system. A video generated by the viewing video imaging system is referred to as a viewing video.

  Based on the position and orientation information, the CG drawing unit 106 draws a virtually existing three-dimensional computer graphic object on the viewing video as an image projected onto the viewing video screen. In the present embodiment, the CG drawing unit 106 draws a virtual CG object on the viewing video and outputs the generated composite video to the display unit 107 as a signal. At this time, the CG rendering unit 106 obtains the position / orientation information and the environment map information from the position / orientation estimation unit 103, and the viewing video is as if a CG object exists at a fixed position in the virtual three-dimensional space. A virtual CG object is drawn on top. The display unit 107 displays the video obtained from the CG drawing unit 106. These operations are performed while continuously capturing and processing images.

  In the present embodiment, the MR apparatus 100 includes an analysis video imaging system (101, 102), a viewing video imaging system (104, 105), and a display unit 107. However, the MR apparatus 100 may have these processing units doubly, and such an MR apparatus 100 can be used as an HMD (head mounted display) that captures and displays images of the left eye and the right eye. Further, in FIG. 1, each part is illustrated as being directly connected, but the present invention is not limited to such a configuration, and each part may be connected via a bus. In this case, each part receives a signal via the bus. You can send and receive.

  The conventional MR apparatus has only one imaging system, and the position and orientation of the MR apparatus are estimated based on the video obtained by the imaging system, and the virtual CG object is added to the video obtained by the same imaging system. Was drawn. For this reason, when a rolling shutter is used as an image sensor, rolling shutter distortion occurs, making it difficult to estimate the position and orientation of the MR apparatus. On the other hand, when a global shutter sensor such as a CCD is adopted, there is a problem that power consumption increases remarkably when resolution is increased or a frame rate is increased.

  The MR apparatus 100 according to the present embodiment has two imaging systems, an analysis video imaging system and a viewing video imaging system. Here, the analysis video imaging system including the sensor 101 has relatively low image quality degradation characteristics of moving objects, and the viewing video imaging system including the sensor 104 has relatively high image quality degradation characteristics of moving objects. That is, the analysis video imaging system has less image quality degradation when a moving object is imaged than the viewing video imaging system. On the other hand, in one embodiment, the viewing video imaging system has better image quality than the analytic video imaging system. For example, the viewing video imaging system has a higher resolution than the analytic video imaging system. For example, a global shutter sensor having a lower resolution is used for the analysis video imaging system, and a rolling shutter sensor having a higher resolution is used for the viewing video imaging system. For this reason, it is possible to achieve both the position / orientation estimation performance and the viewing image quality.

  However, a rolling shutter sensor that can be driven at a high speed may be used as the sensor 101 of the analysis video imaging system instead of the global shutter sensor. For example, when a frame image is acquired using a sensor that can be driven with a vertical scanning time of 4 ms (corresponding to 240 fps), the rolling shutter distortion amount is a quarter of that when the vertical scanning time is 16 ms (corresponding to 60 fps). become. As described above, since an analysis image with a small amount of rolling shutter distortion can be obtained as compared with a rolling shutter sensor driven at a low speed, the position / orientation estimation performance can be improved. Since the sensor 101 may have a smaller number of pixels than the sensor 104, it is easy to employ a sensor that can be driven at high speed.

  Further, both the sensor 101 and the sensor 104 may be global shutter sensors. In general, when an image is captured at a high shutter speed, a sharp image with less motion blur (blur due to movement) can be captured. Since motion blur is a factor that makes video analysis difficult, videos with little motion blur are suitable for use in position and orientation estimation. However, since a discontinuity of motion called jerkiness occurs in a video imaged at a high shutter speed, a sense of incongruity is felt when viewing. Therefore, the sensor 101 can be driven at a high shutter speed (for example, an opening time of 4 ms), and the sensor 104 can be driven at a low shutter speed (for example, an opening time of 16 ms). Even with such a configuration, it is possible to achieve both position and orientation estimation performance and viewing image quality. Since the sensor 101 may have fewer pixels than the sensor 104, it can be easily driven at a high shutter speed.

  The analysis video and the viewing video may have different frame rates. In one embodiment, in order to improve the position and orientation estimation performance, the sensor 101 employs high-speed driving and a high-speed shutter. In this case, the frame rate of the analysis video is higher than the frame rate of the viewing video.

  In one embodiment, sensor 101 is a relatively high frame rate sensor and sensor 104 is a relatively low frame rate sensor. In one embodiment, the sensor 101 is a sensor having a relatively high shutter speed, and the sensor 104 is a sensor having a relatively low shutter speed. In one embodiment, the resolution of the analysis video is lower than the resolution of the viewing video. In one embodiment, the frame rate of the analysis video is higher than the frame rate of the viewing video.

  As described above, the image processing apparatus according to the present embodiment outputs the first video, the first imaging unit that relatively reduces image degradation due to the movement of the subject, and the second video. A second imaging unit that is relatively subject to image degradation caused by movement of the subject. Then, the position / orientation estimation unit 103 analyzes the first video to generate position / orientation information of the image processing apparatus, and the CG drawing unit 106 superimposes the position on the position determined based on the position / orientation information. The CG object is drawn on the second video. At this time, if the movement of the image processing apparatus is larger than the predetermined threshold, the first video is analyzed to generate position and orientation information, and if the movement of the image processing apparatus is smaller than the predetermined threshold, the second video is The position and orientation information is generated by analysis.

[Embodiment 2]
An MR apparatus 200 that is an image processing apparatus according to the second embodiment of the present invention will be described with reference to FIG. In comparison with the MR apparatus 100, the MR apparatus 200 includes an inertial sensor 201 and a selector 202. Unless otherwise specified, the operation of the MR apparatus 200 is the same as the operation of the MR apparatus 100 (for example, a head mounted display) according to the first embodiment.

  The inertial sensor 201 outputs speed information of the MR apparatus 200. As the inertial sensor 201, a gyro sensor or an acceleration sensor can be used. These are sensors for detecting acceleration, and the velocity can be calculated from the acceleration by calculation. In this embodiment, the inertial sensor 201 includes a gyro sensor, an acceleration sensor, and a calculation processing unit, and can output speed information of the MR apparatus 200. In the present embodiment, the inertial sensor 201 outputs horizontal and vertical velocities and angular velocities with respect to the sensor surface of the apparatus. A widely known method can be adopted as a method for calculating velocity information from acceleration information, and detailed description thereof is omitted in this specification. The processing for calculating the speed information is not necessarily performed in the inertial sensor 201, and may be performed by an arithmetic unit installed outside.

  The selector 202 has a function of outputting one of the two inputs. The selector 202 has, for example, a programmable calculation unit, and can select and output one input according to the calculation result. The operation of the selector 202 will be described later.

  The operation of the MR apparatus 200 in the second embodiment will be described. In the present embodiment, the ISP 102 enlarges the internally generated RGB image twice vertically and horizontally and then outputs it. The inertial sensor 201 is fixed to the MR apparatus 200 itself, detects speed information of the MR apparatus 200, and outputs it to the selector 202. The selector 202 calculates the motion amount of the MR apparatus 200 from the speed information, and outputs the video signal from the ISP 102 to the position / orientation estimation unit 103 when it is determined that the motion amount is equal to or greater than the threshold value. On the other hand, when it is determined that the motion amount is smaller than the threshold value, the selector 202 outputs the video signal from the ISP 105 to the position / orientation estimation unit 103.

  The threshold value is calculated based on the position and orientation of the video signal from the ISP 102 when the MR apparatus has a large amount of motion such that it is difficult to estimate the position and orientation of the MR apparatus 200 from the viewing video, and the video signal from the ISP 105 otherwise. It can be set to be output to the unit 103. The amount of movement varies depending on the sensor size, resolution, lens focal length, subject distance, and the like. In the present embodiment, the amount of movement of the MR apparatus 200 is represented by the movement of one frame (for example, 16 ms) of a subject located 30 cm from the camera at the center of the image. Therefore, the motion amount pixels can be expressed in units, and when the motion amount is, for example, 4 pixels or more, the input video from the ISP 102 is adopted. The size of 4 pixels is not limited. Since these operations are performed while images are continuously captured and processed, switching by the selector 202 is also performed in units of frame images.

  In the present embodiment, by adding the inertial sensor 201 and the selector 202, when the MR apparatus 200 is stationary or moving small, the position and orientation with higher accuracy can be obtained using a viewing image with higher resolution. Estimation can be performed. In addition, when the MR apparatus 200 is moving, it is possible to perform highly accurate position and orientation estimation using an analysis image with little rolling shutter distortion as described in the first embodiment. In one embodiment, the output destination of a frame image can be switched according to the amount of movement of the MR apparatus 200 at the time of capturing the frame image. However, the output destination of the frame image may be switched in accordance with the amount of motion of the MR apparatus 200 before or after the frame image is captured.

  As described above, the image processing apparatus according to the present embodiment includes the inertial sensor 201 that detects the movement of the image processing apparatus.

[Embodiment 3]
An MR apparatus 300 that is an image processing apparatus according to the third embodiment of the present invention will be described with reference to FIGS. FIG. 3 is a diagram illustrating the configuration of the MR apparatus 300 according to the third embodiment. Compared to the MR apparatus 100, the MR apparatus 300 has a motion detection unit 301 and a selector 302 added thereto. Unless otherwise specified, the operation of the MR apparatus 300 is the same as the operation of the MR apparatus 100 according to the first embodiment.

  The motion detection unit 301 detects a motion between images captured by the sensor 101 and outputs the motion to the selector 302. In the present embodiment, the motion detection unit 301 includes a ROM, a RAM, and a CPU, and the CPU executes a program that performs an operation described below that is incorporated in the ROM while using the RAM as a work area. It is not essential that the motion detection unit 301 includes a CPU, and a dedicated HW having an equivalent function can also be used as the motion detection unit 301.

  The selector 302 has the same function as the selector 202 according to the second embodiment, and the operation will be described later.

  Hereinafter, the overall operation of the MR apparatus 300 (for example, a head mounted display) will be described. The motion detection unit 301 acquires an image from the ISP 102, detects the amount of motion between images, and outputs the detected amount to the selector 302. A method for calculating the amount of movement will be described later. The selector 302 outputs the video signal from the ISP 102 to the position / orientation estimation unit 103 when it is determined that the amount of motion is equal to or greater than the threshold. If the selector 302 is determined to be small, the selector 302 outputs the input video signal from the ISP 105 to the position / orientation estimation unit 103. The threshold value can be set in the same manner as in the second embodiment. Here, it is assumed that the input video from the ISP 102 is employed when the amount of motion is 4 pixels or more.

  Hereinafter, a method of detecting the amount of motion between frames by the motion detection unit 301 will be described with reference to FIG. In step S4010, the motion detection unit 301 detects a motion vector of an object on the image between a plurality of frame images. For example, a plurality of motion vectors for a plurality of objects can be detected using two consecutive frame images output from the ISP 102. In step S4020, the motion detection unit 301 calculates a length for each of the detected plurality of motion vectors. In step S4030, the motion detection unit 301 calculates the average length of the detected plurality of motion vectors. The average value thus obtained is used as the amount of movement.

  The method for detecting the amount of motion is not limited to the method shown in FIG. For example, the amount of motion can be detected using the method shown in FIGS. The following method may be used. FIG. 5 is a diagram for explaining the geometric transformation of the frame image. The rectangle without shading represents the current frame, and the shaded area represents the previous frame in time. From FIG. 5, it can be seen that a tilt movement of the MR apparatus 300 occurs between the immediately preceding frame and the original frame in addition to the horizontal and vertical translations. Such plane movement in the three-dimensional space can be expressed by a homography matrix. Hereinafter, a method for calculating a motion amount after calculating a homography matrix will be described with reference to FIG.

  In step S6010, the motion detection unit 301 detects a plurality of motion vectors in the same manner as in step S4010. In step S6020, the motion detection unit 301 calculates a homography matrix according to the detected motion vector. For the calculation of the homography matrix, robust estimation such as RANSAC or M estimation can be used, but the calculation method is not limited. In step S6030, the motion detection unit 301 projects the start and end points of the motion vectors at the four corners of the screen using the homography matrix. In step S6040, the motion detection unit 301 calculates the length of each motion vector obtained by projection from the start and end points of the four motion vectors obtained by projection. In step S6050, the motion detection unit 301 selects the motion vector having the longest length from among the motion vectors obtained by the projection. In the present embodiment, the length of the motion vector thus selected is used as the motion amount. In this description, the motion vectors of the four corner points of the screen are projected. However, the motion vector of another point may be projected, or the number of points to which the motion vector is projected may be larger or smaller. Good.

  In the present embodiment, the inertial sensor 201 can be omitted by using the motion detection unit 301 that electronically detects motion instead of the inertial sensor 201 used in the second embodiment. In this embodiment, the motion detection unit 301 has a dedicated CPU. However, the processing of a plurality of units such as the motion detection unit 301 and the selector 302 is performed using one CPU, for example, using time division processing or the like. Can also be realized. In this case, a CPU having a computing capacity sufficient to realize processing of a plurality of units is used.

  As described above, the image processing apparatus according to the present embodiment includes the motion detection unit 301 that analyzes the first video and detects the motion of the image processing apparatus.

[Embodiment 4]
An MR apparatus 700 (for example, a head mounted display) that is an image processing apparatus according to Embodiment 4 of the present invention will be described with reference to FIG. In the present embodiment, the position / orientation estimation unit 103 has a function of calculating motion information. Unless otherwise specified, the operation of the MR apparatus 700 is the same as the operation of the MR apparatus 200 according to the second embodiment.

  The overall operation of the MR apparatus 700 will be described below. By using the position / orientation information calculated by the position / orientation estimation unit 103, the relative position of the MR apparatus 700 in the three-dimensional space can be calculated. In the present embodiment, the position / orientation estimation unit 103 calculates the amount of change in the relative position between frames, that is, the amount of motion of the MR apparatus 700. The amount of movement can be expressed in the same manner as in the second embodiment.

  The position / orientation estimation unit 103 feeds back this amount of motion to the selector 302. The selector 302 positions the input video signal from the ISP 102 when it is determined that the amount of motion is so large that video analysis from the viewed video is difficult, and the input video signal from the ISP 105 when it is determined to be small. Output to the posture estimation unit 103. This is the same as the operation 302 in the third embodiment.

  Since these operations are performed while images are continuously captured and processed, switching by the selector is also performed in units of frame images. However, switching by the selector is performed from the frame next to the frame in which the motion has occurred. The same effect as in the second embodiment can be obtained without an additional module such as an inertial sensor.

As described above, the position / orientation estimation unit 103 analyzes the first video to generate position / orientation information and detects the movement of the image processing apparatus. When the detected movement is larger than the predetermined threshold, the position / orientation information The video to be analyzed when generating is switched to the second video.
[Embodiment 5]

  A fifth embodiment relating to the MR system will be described with reference to FIG. In the present embodiment, it is assumed that the HMD and the host computer are connected, and the HMD and the host computer are connected via a network. Unless otherwise specified, the operation is the same as that of the second embodiment described with reference to FIG.

  FIG. 8 is a diagram for explaining an MR apparatus configuration according to the fifth embodiment. A selector 801 controls the presence / absence of signal output. Reference numeral 802 denotes an NW (network system). Assume that the HMD side and the host computer side each have a packetizer circuit, a baseband engine, an RF unit, and an antenna, and each main data can be freely transmitted between the HMD and the host computer. Reference numerals 803, 805, and 811 denote video decoding units. Reference numeral 804 denotes a selector for selecting a signal. Reference numeral 807 denotes a CPU. Reference numeral 808 denotes a RAM. Reference numeral 809 denotes a nonvolatile storage. Reference numeral 807 reads the program in the nonvolatile storage 809 into the RAM 808 and executes the program. Reference numeral 810 denotes a bus. Each module is connected by a bus, and the modules connected by the bus are assumed to exchange data unless there is a special description.

  In this embodiment, unlike the first embodiment, the ISP 102 and the ISP 105 encode and output a video. 102, 105, 803, 805, 806, 811 are handled by H.264. However, the present invention is not limited to this. 102 outputs the encoded stream to 801. When it is determined that the input motion amount is so large that it is difficult to analyze the video from the viewing video, the input video signal is output to 803 via the NW 802, and the output is stopped when it is determined that the input video signal is small. To do. While the output of the video signal is stopped, the selector 801 continues to output a stop signal to the ISP 102. H. In an encoding format using an inter frame such as H.264, a subsequent inter frame can be encoded only after the intra frame is decoded. Therefore, the ISP 102 encodes the video only with the intra frame while receiving the stop signal, and starts inter-frame encoding after the stop signal is received. As a result, high-compression encoding using inter frames is realized, and when transmission of an encoded stream is resumed, an intra frame is transmitted, so that decoding can be resumed promptly.

  803 decodes the encoded stream to generate a video signal and outputs it to 804. The ISP 105 outputs the encoded stream to the video decoder 805 via the NW 802. The video decoder 805 generates a video signal by decoding the encoded stream. The selector 804 outputs the video signal input from 803 to 807 when the data 801 transmits data, and outputs the video signal input from 805 to 807 when the data is not transmitted. The output may be performed via a RAM. The CPU 807 performs position / orientation estimation and outputs position / orientation information and an environment map to the CG drawing unit 106. The CG rendering unit 106 outputs the video generated by performing the operation described in the first embodiment to the video encoding unit 806. The video encoding unit 806 outputs the encoded stream obtained by encoding the video to the video decoding unit 811 via the NW 802. The video decoding unit 811 outputs the video signal generated by decoding the encoded stream to the display unit 107. Since these operations are performed while images are continuously captured and processed, switching by the selector is also performed in units of frame images.

  Thus, according to the present embodiment, both the analysis video and the viewing video are sent from the HMD to the host computer, and the host computer generates the composite video using both the analysis video and the viewing video. The same effect as in the first mode can be obtained. In this embodiment, analysis video transmission control is performed in accordance with the amount of motion. That is, according to the present embodiment, in addition to the effects shown in the second embodiment, NW transmission of the analysis video can be suppressed and the NW band can be reduced when the video motion amount is small. Further, the code amount corresponding to the reduced amount may be assigned to the code amount at the time of encoding the viewing video, and an operation for generating a higher-definition viewing video may be performed. On the other hand, in order to obtain the same effect as in the first embodiment, it is not essential to perform transmission control of the analysis video according to the amount of motion.

  In this embodiment, the HMD functions as an image processing device, and the host computer functions as an image composition device. The HMD and the host computer constitute an image processing system.

[Embodiment 6]
A sixth embodiment relating to the MR system will be described with reference to FIGS. FIG. 9 is a diagram for explaining an MR apparatus configuration according to the sixth embodiment. In FIG. 9, 901 to 904 are added to FIG. When there is no special description, the operation is the same as that of the second embodiment described with reference to FIG.

  Reference numeral 901 denotes a bus. Reference numeral 902 denotes a CPU. Reference numeral 903 denotes a RAM. Reference numeral 904 denotes a nonvolatile storage. The program stored in the non-volatile storage is read into the RAM via the bus and executed by the CPU. In this embodiment, a position / orientation estimation program is executed to generate position / orientation information and environment map information. Further, the motion amount output from the inertial sensor 201 and the video data output from the ISPs 102 and 105 are stored in the RAM 903 via the bus and become program input data. The CG drawing unit 106 acquires the image data stored in the RAM 903 via the bus, draws a computer graphic object, and outputs the computer graphic object to the display unit 107. The CPU 902 controls each module. Unless otherwise specified, modules connected to the bus shall input and output data via the bus.

  The operation of the position / orientation estimation program executed by the CPU 902 will be described below with reference to FIG. FIG. 10 is a diagram for explaining a position and orientation estimation method according to the sixth embodiment. In S10000, the amount of movement is acquired. In this embodiment, the amount of motion output from the inertial sensor 201 is acquired. In S10010, a viewing video is acquired. In S10020, the amount of motion is determined. If it is determined that the amount of motion is so large that video analysis from the viewed video is difficult, S10030 is executed. If it is determined that the value is smaller, S10040 is executed. In S10030, the video A for acquiring the analysis video is used. As described above, the analysis video is a video with less image quality degradation due to movement than the viewing video. In S10035, the analysis video is video A. In S10040, the viewing video is video A. In S10050, using the video A, position and orientation information and an environment map are generated.

  In S10060, a computer graphic object is drawn on the viewing video using the position and orientation information and the environment map. As a result, an image is generated as if a computer graphic object exists at a fixed position in a virtual three-dimensional space. In the present embodiment, the CPU 902 is executed in such a manner as to issue an instruction to the CG drawing unit 106, but the present invention is not limited to this, and the CPU 902 may be responsible for superimposition processing. Since these operations are performed while images are continuously captured and processed, condition determination is also performed in units of frame images.

  Even if it is the structure of this embodiment, the effect similar to Embodiment 2 is acquired. In the present embodiment, the output from the inertial sensor is used as the amount of movement, but as shown in the third and fourth embodiments, a value calculated by software processing may be acquired. In the present embodiment, an example in which the amount of motion is acquired has been described, but a configuration in which the amount of motion is not used may be employed. Hereinafter, this modification will be described with reference to FIG.

  FIG. 11 is a diagram illustrating a modification of the sixth embodiment. In S11000, an analysis video is acquired. In S11010, a viewing video is acquired. In S11020, position and orientation information and an environment map are generated using the analysis video. In S10030, a computer graphic object is drawn on the viewing video using the position and orientation information and the environment map.

  As a result, an image is generated as if a computer graphic object exists at a fixed position in a virtual three-dimensional space. Even if it is such a structure, the effect similar to Embodiment 1 can be acquired.

[Embodiment 7]
A seventh embodiment relating to the MR system will be described with reference to FIGS. FIG. 12 is a diagram for explaining an MR apparatus configuration according to the seventh embodiment. Unless otherwise specified, the operation is the same as that of the fifth embodiment described with reference to FIG.

  FIG. 12 is a diagram illustrating the MR system configuration according to the seventh embodiment. In the fifth embodiment, the configuration of the host computer is shown as a configuration in which a dedicated HW exists and modules are directly connected. In this embodiment, each module is connected to a bus, and the video decoding function and the video encoding function are also performed by the CPU. In FIG. 11, since there is no selector function described in the fifth embodiment, the CPU 807 controls the data flow and performs position and orientation estimation. The control flow at this time will be described with reference to FIG. FIG. 12 is a diagram for explaining a position and orientation estimation method according to the seventh embodiment. FIG. 13 is different from FIG. 10 in that S10010 is replaced with S13000 and S10020 is replaced with S13020. The operation of other steps is the same as that described in FIG.

  In S13000, the status of whether or not the host computer has received the analysis video is acquired. The analysis video reception operation is performed for each frame. In S13020, if an analysis video is received, S10030 is executed, and S10040 is executed. Since these operations are performed while images are continuously captured and processed, condition determination is also performed in units of frame images. Even with the configuration shown in the present embodiment, the same effect as in the fifth embodiment can be obtained.

  According to this embodiment, the image processing apparatus further includes a configuration for determining the reception state of the first video. If the first video is received, the first video is analyzed to generate position / orientation information. If the first video is not received, the second video is analyzed to generate position / orientation information. Is done.

  The head mounted display as an example of the MR apparatus has been described as a video see-through type in which a composite image obtained by superimposing a CG object on a captured image is displayed on a display unit to allow a user to observe. However, the head-mounted display according to the present specification may be an optical see-through type in which a CG object is superimposed and displayed on a display that can be observed through the real space.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 sensor; 102 ISP; 103 position and orientation estimation unit; 104 sensor; 105 ISP; 106 CG rendering unit; 107 display unit; 201 inertia sensor; 209 selector; 301 motion detection unit; 302 selector; 801 selector; 804 selector; 805 video decoding unit; 806 video encoding unit; 807 CPU; 808 RAM; 809 non-volatile storage; 810 bus; 811 video decoding unit; 901 bus; 902 CPU; 903 RAM;

Claims (18)

An image processing apparatus,
A first imaging means for outputting a first video and relatively less image degradation caused by movement of the subject;
A second imaging means for outputting a second video, wherein image degradation due to movement of the subject is relatively large;
Estimating means for analyzing the first video and generating position and orientation information of the image processing device;
Drawing means for drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information;
An image processing apparatus comprising:   When the movement of the image processing apparatus is larger than a predetermined threshold, the estimating means analyzes the first video to generate the position and orientation information, and when the movement of the image processing apparatus is smaller than the predetermined threshold The image processing apparatus according to claim 1, wherein the position and orientation information is generated by analyzing the second video.   The image processing apparatus according to claim 2, further comprising a motion detection unit that detects a motion of the image processing apparatus using an inertial sensor.   The image processing apparatus according to claim 2, further comprising a motion detection unit that analyzes the first video and detects a motion of the image processing apparatus.   The estimation means analyzes the first video to generate the position / orientation information and detects the movement of the image processing apparatus. When the detected movement is greater than a predetermined threshold, 5. The image processing apparatus according to claim 1, wherein an image to be analyzed at the time of generation is switched to the second image. 6.   The image processing apparatus according to claim 1, wherein the first imaging unit is a global shutter sensor, and the second imaging unit is a rolling shutter sensor.   The first imaging means is a sensor with a relatively high frame rate, and the second imaging means is a sensor with a relatively low frame rate. The image processing apparatus according to item.   The first image pickup means is a sensor having a relatively high shutter speed, and the second image pickup means is a sensor having a relatively low shutter speed. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein a resolution of the first video is lower than a resolution of the second video.   The image processing apparatus according to claim 1, wherein a frame rate of the first video is higher than a frame rate of the second video. First acquisition means for receiving, from the image processing apparatus, a first video that has relatively little image degradation due to movement of the subject;
Second acquisition means for receiving, from the image processing device, a second video that has relatively high image degradation due to movement of the subject;
Estimating means for analyzing the first video and generating position and orientation information of the image processing device;
Drawing means for drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information;
An image composition apparatus comprising: A determination unit for determining a reception state of the first video;
The estimation means generates the position and orientation information by analyzing the first video when the first video is received, and generates the position and orientation information when the first video is not received. The image synthesizing apparatus according to claim 11, wherein the position and orientation information is generated by analyzing a video. An image processing system comprising an image processing device and an image composition device,
The image processing apparatus includes:
A first imaging means for outputting a first video and relatively less image degradation caused by movement of the subject;
A second imaging means for outputting a second video, wherein image degradation due to movement of the subject is relatively large;
Transmission means for transmitting the first video and the second video to an image composition device,
The image composition device includes:
First acquisition means for receiving the first video from an image processing device;
Second acquisition means for receiving the second video from the image processing device;
Estimating means for analyzing the first video and generating position and orientation information of the image processing device;
An image processing system comprising: drawing means for drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information.   The transmission means transmits both the first video and the second video when the movement of the image processing apparatus is larger than a predetermined threshold, and the movement of the image processing apparatus is smaller than the predetermined threshold. The image processing system according to claim 13, wherein the second video is transmitted without transmitting the first video. An image processing method performed by an image processing apparatus,
Using the first imaging means to obtain a first video with relatively little image degradation due to movement of the subject;
Using the second imaging means to obtain a second video that has a relatively large image degradation due to movement of the subject;
Analyzing the first video to generate position and orientation information of the image processing device;
Drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information;
An image processing method comprising: An image processing method performed by an image composition device,
Receiving a first image from the image processing apparatus with relatively little image degradation caused by movement of the subject;
Receiving from the image processing device a second video image that is relatively image-degraded due to movement of the subject;
Analyzing the first video to generate position and orientation information of the image processing device;
Drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information;
An image processing method comprising: An image processing method performed by an image processing system including an image processing device and an image composition device,
The image processing apparatus using the first imaging means to obtain a first video that has relatively little image degradation due to movement of the subject;
The image processing apparatus using the second imaging means to acquire a second video image that is relatively largely deteriorated due to movement of the subject;
The image processing device transmitting the first video and the second video to the image synthesis device;
The image synthesizing device receiving the first video from the image processing device;
The image synthesizing device receiving the second video from the image processing device;
The image synthesizing device analyzing the first video to generate position and orientation information of the image processing device;
Drawing the CG object on the second video so that the image synthesizing device is superimposed on the position determined based on the position and orientation information;
An image processing method comprising: On the computer,
Obtaining a first video imaged by the image processing apparatus, with relatively little image degradation due to movement of the subject;
Obtaining a second video imaged by the image processing device, the image of which is relatively deteriorated due to movement of a subject;
Analyzing the first video to generate position and orientation information of the image processing device;
Drawing a CG object on the second video so as to be superimposed on a position determined based on the position and orientation information;
A program for running