JP2009048237A - Image processor, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique for reducing preventing as much as possible a virtual space image and an actual space image from being generated asynchronously with displaying a composite image for the images. <P>SOLUTION: An image processor acquires a time required for calculating position attitude information of an imaging device 1010, a time required for generating a virtual space image, a time required for generating an actual space image, and a time required for whole processing to generate a composite image of the virtual space image and the actual space image. The processor decides an operation order of a position attitude derivation unit 1120, an image input unit 1110, a virtual space image generation unit 1190 and an image composition unit 1200. The processor then operates the derivation unit 1120, input unit 1110, generation unit 1190 and composition unit 1200 in the decided operation order. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for providing mixed reality.

  In recent years, research on mixed reality (MR) for the purpose of seamless connection between real space and virtual space has been actively conducted. An image display device that presents mixed reality is, for example, a device having the following configuration. That is, an image of a virtual space (for example, a virtual object or character information drawn by computer graphics) generated according to the position and orientation of the imaging device on an image of the real space captured by the imaging device such as a video camera. It is an apparatus that displays an image that is superimposed and drawn. For example, an HMD (head mounted display) can be used for such an apparatus.

  Such an image display device is also realized by an optical see-through method in which an image in a virtual space generated according to the position and orientation of the observer's viewpoint is displayed on an optical see-through display mounted on the observer's head. The

  As an application of such an image display device, there are surgical support for superimposing and displaying the state of the body on the patient's body surface, a mixed reality game for fighting virtual enemies floating in the real space, and the like. In addition to this, for example, there is navigation that superimposes and displays names and guidance of famous buildings and the like included in an image of a real space obtained by imaging a city area as a virtual space image. In addition, for example, there is a landscape simulation in which a computer graphics image of a building to be built is superimposed on an image obtained by imaging a building construction site.

  In order to observe the inside of the mixed reality space, it is necessary to perform the following processing. That is, the user's viewpoint position, line-of-sight direction, and the like are measured in real time, and an image obtained by observing the real space from the viewpoint position and line-of-sight direction is acquired by a video camera or the like. Then, a virtual object (CG object) observed from the viewpoint position, line-of-sight direction, etc. of the user is superimposed on the acquired video.

  Here, in order to measure the user's viewpoint position and line-of-sight direction in real time, a magnetic position / orientation sensor or the like is used, and the position / orientation of the imaging apparatus is detected by attaching the sensor to the imaging apparatus. The magnetic position / orientation sensor detects a relative position / orientation between a magnetism generator (transmitter) and a magnetic sensor (receiver), such as a product FASTRAK manufactured by Polhemus, USA. This is a device that detects the three-dimensional position and orientation of a sensor in real time within a specific region. It is possible to observe the inside of the mixed reality space by connecting the measured values of the six degrees of freedom to the position and orientation (camera coordinate system) of the virtual camera in the virtual space.

By the way, it is known that an experienced person will have symptoms such as motion sickness only by observing visual stimuli. This symptom is called video sickness or VR sickness. Although the mechanism is not clear at present, the main factor of video sickness is the mismatch between the input of visual motion and the input of motion from the balanced sense system. Has been. In recent years, research to elucidate the mechanism of motion sickness (Non-Patent Document 1) and research to prevent motion sickness by using an image that has undergone camera shake correction processing as a video to be displayed (Non-Patent Document 2) have been conducted. Yes.
Tree dragon. Relationship between motion vector and biological function of video sickness. ITE Technical Report, Vol. 30, no. 58, 2006. Morita. Technology that prevents the effects of video on the living body. ITE Technical Report, Vol. 30, no. 58, 2006.

  The above-mentioned video sickness is a symptom that develops even when an HMD is worn to experience mixed reality. One reason for this is said to be a time contradiction between the vestibular system and the visual system due to a shift between the body movement of the mixed reality experience person and the timing at which the mixed reality space image is displayed. For example, in the case of a video see-through method, this timing shift is necessary for each process such as capture and rendering of a real space image, derivation of the position and orientation of a camera coordinate system, and rendering of a virtual space image to generate a mixed reality space image. This is the cause.

  In order to reduce this timing shift as much as possible, various methods have been proposed so far for speeding up the process for deriving the position and orientation of the camera coordinate system and speeding up the rendering of the virtual space image. ing. However, no proposal has been made so far regarding a method for synthesizing a real space image and a virtual space image necessary for generating a mixed reality space image.

  The present invention has been made in view of the above problems, and provides a technique for reducing the deviation between the generation timing of a virtual space image and a real space image and the timing of displaying a composite image of these images as much as possible. The purpose is to do.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, a first means for calculating position and orientation information of the imaging device included in the head-mounted display device;
A second means for generating a virtual space image based on the position and orientation information;
Third means for generating a real space image based on a result of imaging by the imaging device;
An image processing apparatus comprising: output means for outputting a composite image of the virtual space image and the real space image to the head-mounted display device,
Time information indicating time required to calculate the position and orientation information; time information indicating time required to generate the virtual space image; time information indicating time required to generate the real space image; Time information indicating the time required for processing for generating an image, and determining means for determining the operation order of the first to third means based on the acquired time information;
And a means for operating the first to third means in the operation order determined by the determining means.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, a first means for calculating position and orientation information of the imaging device included in the head-mounted display device;
A second means for generating a virtual space image based on the position and orientation information;
Third means for generating a real space image based on a result of imaging by the imaging device;
An image processing method performed by an image processing apparatus comprising: output means for outputting a composite image of the virtual space image and the real space image to the head-mounted display device,
Time information indicating time required to calculate the position and orientation information; time information indicating time required to generate the virtual space image; time information indicating time required to generate the real space image; A determination step of acquiring time information indicating a time required for processing for generating an image, and determining an operation order of the first to third means based on the acquired time information;
And a step of operating the first to third means in the operation order determined in the determination step.

  According to the configuration of the present invention, it is possible to reduce the deviation between the generation timing of the virtual space image and the real space image and the timing of displaying the composite image of these images as much as possible.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a block diagram illustrating a functional configuration example of a system according to the present embodiment. As shown in FIG. 1, the system according to the present embodiment includes an image processing device 1100 and an HMD 1000 as an example of a head-mounted display device. The image processing device 1100 and the HMD 1000 communicate with each other. Are connected as possible. Accordingly, the connection between the image processing apparatus 1100 and the HMD 1000 may be either wired or wireless.

  First, the HMD 1000 will be described.

  Reference numeral 1010 denotes an image pickup apparatus such as a video camera, which picks up a moving image in the real space. Then, the imaging apparatus 1010 sequentially sends the images of each frame (real space image) constituting the captured moving image to the image processing apparatus 1100. The imaging device 1010 is provided for the right eye and for the left eye, respectively, and captures a moving image in the real space seen from the right eye and a moving image in the real space seen from the left eye of the observer wearing the HMD 1000 on the head.

  Reference numeral 1020 denotes a display device composed of a liquid crystal screen or the like, and displays an image (a composite image or a mixed reality space image) sent from the image processing device 1100. The display device 1020 is provided for each of the right eye and the left eye, and each is attached to the HMD 1000 so as to be positioned in front of the right eye and the left eye of the observer wearing the HMD 1000 on the head.

  Next, the image processing apparatus 1100 will be described.

  The image input unit 1110 acquires the real space image for the right eye and the real space image for the left eye sent from the imaging device 1010. Each acquired physical space image is stored in the data storage unit 1170.

  The position / orientation deriving unit 1120 (first unit) calculates position / orientation information of the imaging apparatus 1010 using the real space image stored in the data storage unit 1170 by the image input unit 1110. Here, the operation of the position / orientation deriving unit 1120 will be described in more detail.

  In the real space that can be imaged by the imaging apparatus 1010, indices Q1 to Q4 (hereinafter referred to as reference indices) are arranged as shown in FIG. It is assumed that the position of each reference index in the world coordinate system is known. Here, the world coordinate system is a coordinate system in which one point in the real space is defined as the origin, and three axes orthogonal to each other at the origin are the x axis, the y axis, and the z axis. The arrangement position of each reference index is stored in the data storage unit 1170 as data.

  Each reference index is preferably arranged so that three or more images are always captured by the imaging device 1010 within a range in which the HMD 1000 can move. FIG. 1 shows a situation where Q1, Q3, and Q4 are included in the field of view of the imaging device 1010. Note that the number of reference indices arranged in the real space is not limited to four as shown in FIG. 1 and may be three or more.

  Further, the reference index may be constituted by, for example, circular markers each having a different color, or may be constituted by feature points such as natural features each having a different texture feature. It is also possible to use a square index formed by a rectangular single color area having a certain area. That is, if the image coordinates of the projected image of the reference index on the image captured by the imaging device 1010 can be detected and which reference index can be identified by any method, which reference index is Form may be sufficient.

  Therefore, the position / orientation deriving unit 1120 acquires the real space image stored in the data storage unit 1170 and detects the image coordinates of the reference index in the acquired real space image. Then, using the detected image coordinates and the arrangement position of the reference index in the world coordinate system (obtained from the data storage unit 1170), the position and orientation information of the imaging device 1010 is calculated. This calculation process is performed for each of the right-eye imaging device 1010 and the left-eye imaging device 1010. The following document describes the technique for calculating the position and orientation information of the imaging device that captured the image using the image coordinates of the reference index in the image and the arrangement position of the reference index in the world coordinate system. Since it is publicly known, a detailed description thereof will be omitted.

● R. M.M. Haralick, C.I. Lee, K.K. Ottenberg, and M.M. Nole: Review and analysis of solutions of the three point perspective positive estimation, International Journal of Computer Vision, vol. 13, no. 3, pp. 331-356, 1994.
● D. G. Low: Fitting parametrized three-dimensional models to images, IEEE Transactions on PAMI, vol. 13, no. 5, pp. 441-550, 1991.
Note that the method for obtaining the position and orientation information of the imaging device 1010 is not limited to this, and various methods are conceivable. For example, a method using a position and orientation sensor can be considered. As the position and orientation sensor, for example, there is a product FASTRAK manufactured by Polhemus, USA. A method for calculating the position and orientation information of the imaging apparatus 1010 using the position and orientation sensor will be briefly described. After the magnetic transmitter is placed at a predetermined position in the real space, when the magnetic transmitter is operated, the magnetic transmitter generates a magnetic field around it. On the other hand, a magnetic receiver is attached to the imaging device 1010, and the magnetic receiver detects a change in the magnetic field according to its position and orientation in the magnetic field. The detected result is sent to a sensor control device that controls the operation of the magnetic transmitter. The sensor control device calculates position and orientation information of the magnetic receiver in the sensor coordinate system based on the detection result. Here, the sensor coordinate system is a coordinate system in which the position of the magnetic transmitter is defined as the origin, and the three axes orthogonal to each other at the origin are the x axis, the y axis, and the z axis. Then, the obtained position and orientation information of the magnetic receiver in the sensor coordinate system is sent to the image processing apparatus 1100. Note that the processing for converting the position and orientation information in the sensor coordinate system into the position and orientation information in the world coordinate system and other coordinate systems may be performed as appropriate.

  In the present embodiment, any configuration may be used as long as the position and orientation information of the imaging device 1010 can be obtained. As a result, the position / orientation deriving unit 1120 stores the obtained position / orientation information in the data storage unit 1170.

  The time delay measurement unit 1130 measures the time required for the position / orientation deriving unit 1120 to calculate the position / orientation information (position / orientation deriving time delay). Then, time information indicating the measured time is stored in the data storage unit 1170. Such measurement is performed every time position and orientation information is calculated.

  The virtual space image generation unit 1190 (second unit) constructs a virtual space based on the virtual space data stored in the data storage unit 1170. The virtual space data includes data relating to each virtual object constituting the virtual space and data relating to a light source that irradiates the virtual space. Then, the virtual space image generation unit 1190 sets a viewpoint having the position and orientation indicated by the position and orientation information stored in the data storage unit 1170 in the virtual space. And the image (virtual space image) of the virtual space seen from the viewpoint concerned is generated. Since a technique for generating an image of a virtual space that can be seen from a viewpoint having a predetermined position and orientation is a well-known technique, a detailed description thereof will be omitted.

  The processing speed measurement unit 1140 measures the time (virtual space image drawing processing time) required for the virtual space image generation unit 1190 to generate the virtual space image. Then, time information indicating the measured time is stored in the data storage unit 1170. Such measurement is performed every time a virtual space image is generated.

  First, the image composition unit 1200 draws a real space image stored in the data storage unit 1170 on a memory managed by the image composition unit 1200 (third means). Then, the virtual space image generated by the virtual space image generation unit 1190 is superimposed and drawn on the drawn real space image, thereby generating a composite image of the real space image and the virtual space image on the memory. The generated composite image is output to the display device 1020 of the HMD 1000.

  The processing speed measurement unit 1150 measures the time (real space image drawing processing time) required for the image composition unit 1200 to draw a real space image. Then, time information indicating the measured time is stored in the data storage unit 1170.

  The time delay measurement unit 1160 measures the time from when the image composition unit 1200 outputs the composite image at the nth (n ≧ 1) frame to when the composite image at the (n + 1) th frame is output. . That is, the time required for all the processes for generating a composite image for one frame is measured. Then, time information indicating the measured time is stored in the data storage unit 1170. This measurement is performed every time the image composition unit 1200 outputs a composite image.

  The image composition method designating unit 1180 refers to the time information stored in the data storage unit 1170, and the operation order of the position / orientation deriving unit 1120, the image input unit 1110, the virtual space image generation unit 1190, and the image composition unit 1200. To decide. Then, the operation of each unit is controlled so that each unit operates in the determined order.

  As described above, the data storage unit 1170 is for storing various types of information, and includes a RAM, a hard disk drive device, and the like. In addition to the information described as being stored in the data storage unit 1170 in the above description, the data storage unit 1170 also stores information described as known information in the present embodiment.

  FIG. 3 is a flowchart of processing performed by the image processing apparatus 1100 to generate a composite image of the real space image and the virtual space image and output the composite image to the HMD 1000.

  First, in step S3010, the image composition method designating unit 1180 sets an operation order as an initial value. Such an operation order may be, for example, an operation order instructed by the user of the image processing apparatus 1100 or a predetermined order.

  Next, in step S3020, each unit constituting the image processing apparatus 1100 operates to generate a composite image of the real space image and the virtual space image. Note that the operation order of the position and orientation deriving unit 1120, the image input unit 1110, the virtual space image generation unit 1190, and the image composition unit 1200 that operate in the generation process is the operation order set in step S3010.

  4A to 4E are flowcharts corresponding to various operation orders for performing the processing in step S3020. In the following description, the operation order for executing the process according to the flowchart of FIG. 4A is referred to as “operation order A”, and the operation order for executing the process according to the flowchart of FIG. 4B is referred to as “operation order B”. To do. Also, the operation order for executing the processing according to the flowchart of FIG. 4C is referred to as “operation order C”, and the operation order for executing the processing according to the flowchart of FIG. 4D is referred to as “operation order D”. In addition, the operation order for executing the processing according to the flowchart of FIG. 4E is referred to as “operation order E”. In the following description, it is assumed that the operation order A is set in step S3010. However, even if other operation orders are set as initial values, the following description is substantially the same.

  Therefore, in step S3020, processing according to the flowchart of FIG. 4A is executed. Since the processing according to the flowchart of FIG. 4A derives the position and orientation information immediately after the capture of the real space image, the consistency between the real space image and the virtual space image can be increased.

  First, in step S4010, the image input unit 1110 acquires (captures) a real space image (imaging result) sent from the imaging device 1010. Note that the time delay measurement unit 1160 starts timing from the timing at which the synthesized image of one frame before is output from the image synthesis unit 1200.

  Next, in step S4011, the position / orientation deriving unit 1120 uses the real space image acquired in step S4010 and the arrangement position of each reference index stored in the data storage unit 1170 to determine the position / orientation of the imaging device 1010. Ask for information. At this time, the time delay measuring unit 1130 measures the time required for the position / orientation deriving unit 1120 to obtain the position / orientation information.

  Next, in step S4012, the image composition unit 1200 renders the real space image acquired in step S4010 on the memory managed by the image composition unit 1200. At this time, the processing speed measurement unit 1150 measures the time required for the image composition unit 1200 to draw the real space image.

  In step S4013, the virtual space image generation unit 1190 generates a virtual space image that can be seen from the position and orientation indicated by the position and orientation information calculated in step S4011. At this time, the processing speed measurement unit 1140 measures the time required for the virtual space image generation unit 1190 to generate the virtual space image. Then, the image composition unit 1200 superimposes and draws the virtual space image generated by the virtual space image generation unit 1190 on the real space image drawn on the memory in step S4012, so that the real space is displayed on the memory. A composite image of the image and the virtual space image is generated. The generated composite image is output to the display device 1020 of the HMD 1000. Then, the time delay measuring unit 1160 measures the time from the timing when the synthesized image of the previous frame is output to the timing when the synthesized image in the current frame is output from the image synthesizing unit 1200.

  Then, the process returns to step S3030.

  In step S3030, time information indicating each time measured in steps S4010 to S4013 is stored in data storage unit 1170.

  Next, when an instruction to end the process is input or when a condition for ending the process is satisfied, the process ends. On the other hand, if the end instruction of this process has not been input and the condition for ending this process is not satisfied, the process proceeds to step S3050 via step S3040.

  In step S3050, the image composition method designating unit 1180 refers to each time information collected in the data storage unit 1170 in step S3030. Then, based on the time information referred to, the operation order of the position / orientation deriving unit 1120, the image input unit 1110, the virtual space image generating unit 1190, and the image synthesizing unit 1200 for generating a synthesized image of the next frame. (Operation order of first to third means) is determined and set.

  There are various methods for determining the operation order, and an example will be described below.

  For example, it is assumed that the time tv measured by the processing speed measurement unit 1140 is equal to or greater than the threshold, and the ratio of the time tv to the time measured by the time delay measurement unit 1160 is equal to or greater than the threshold. In this case, the image composition method designating unit 1180 determines to execute the process according to the flowchart of FIG. 4C as the process for generating the composite image of the next frame (the process in step S3020). That is, an “operation order C” that is an operation order for executing the processing according to the flowchart of FIG. 4C is set.

  Then, the process returns to step S3020, and a process for generating a composite image of the next frame is executed. Here, the process in step S3020 that is performed when a composite image of the next frame is generated, that is, the process according to the flowchart of FIG. 4C will be described.

  In step S4030, the position / orientation deriving unit 1120 uses the real space image stored in the data storage unit 1170 as the real space image of the previous frame and the arrangement position of each reference index stored in the data storage unit 1170. The position and orientation information of the imaging apparatus 1010 is obtained. At this time, the time delay measuring unit 1130 measures the time required for the position / orientation deriving unit 1120 to obtain the position / orientation information. Furthermore, the time delay measurement unit 1160 starts timing from the timing when the composite image of one frame before is output from the image synthesis unit 1200.

  In step S4031, the virtual space image generation unit 1190 generates a virtual space image that can be seen from the position and orientation indicated by the position and orientation information calculated in step S4030. At this time, the processing speed measurement unit 1140 measures the time required for the virtual space image generation unit 1190 to generate the virtual space image.

  Next, in step S4032, the image input unit 1110 acquires (captures) the real space image sent from the imaging device 1010.

  Next, in step S4033, the image composition unit 1200 first draws the physical space image acquired in step S4032 on the memory managed by the image composition unit 1200. At this time, the processing speed measurement unit 1150 measures the time required for the image composition unit 1200 to draw the real space image. Next, the image composition unit 1200 superimposes and draws the virtual space image generated by the virtual space image generation unit 1190 in step S4031 on the real space image drawn on the memory. A composite image with the virtual space image is generated. The generated composite image is output to the display device 1020 of the HMD 1000. Then, the time delay measuring unit 1160 measures the time from the timing when the synthesized image of the previous frame is output to the timing when the synthesized image is output from the image synthesizing unit 1200.

  Then, the process returns to step S3030.

  By using the operation sequence C, the capture and the drawing process are performed immediately before the timing of displaying the composite image, so that it is possible to reduce the time delay of the real space image.

  The threshold value may be set in advance or may be input by an instruction input by the user of the image processing apparatus 1100. The same applies to the thresholds appearing later.

  In addition, as the processing for determining the operation order in step S3050, for example, the time tr measured by the processing speed measurement unit 1150 is equal to or greater than a threshold, and the ratio of the time tr to the time measured by the time delay measurement unit 1160 is equal to or greater than the threshold. Suppose there is. In this case, the image composition method designating unit 1180 determines to execute the process according to the flowchart of FIG. 4B as the process of generating the composite image of the next frame (the process in step S3020). That is, an “operation order B” that is an operation order for executing the processing according to the flowchart of FIG. 4B is set.

  First, in step S4020, the image input unit 1110 acquires (captures) a real space image sent from the imaging device 1010. Note that the time delay measurement unit 1160 starts timing from the timing at which the synthesized image of one frame before is output from the image synthesis unit 1200.

  Next, in step S4021, the image composition unit 1200 renders the real space image acquired in step S4020 on the memory managed by the image composition unit 1200. At this time, the processing speed measurement unit 1150 measures the time required for the image composition unit 1200 to draw the real space image.

  Next, in step S4022, the position / orientation deriving unit 1120 uses the real space image acquired in step S4020 and the arrangement position of each reference index stored in the data storage unit 1170 to determine the position / orientation of the imaging device 1010. Ask for information. At this time, the time delay measuring unit 1130 measures the time required for the position / orientation deriving unit 1120 to obtain the position / orientation information.

  Next, in step S4023, the virtual space image generation unit 1190 generates a virtual space image that can be seen from the position and orientation indicated by the position and orientation information calculated in step S4022. At this time, the processing speed measurement unit 1140 measures the time required for the virtual space image generation unit 1190 to generate the virtual space image. Next, the image composition unit 1200 superimposes and draws the virtual space image generated by the virtual space image generation unit 1190 on the real space image drawn on the memory in step S4021, so that the real space is displayed on the memory. A composite image of the image and the virtual space image is generated. The generated composite image is output to the display device 1020 of the HMD 1000. Then, the time delay measuring unit 1160 measures the time from the timing when the synthesized image of the previous frame is output to the timing when the synthesized image is output from the image synthesizing unit 1200.

  Then, the process returns to step S3030.

  Further, as a process of determining the operation order in step S3050, for example, it is assumed that the time tp measured by the time delay measurement unit 1130 is equal to or greater than a threshold value. In this case, the image composition method specifying unit 1180 determines to execute the process according to the flowchart of FIG. 4A as the process of generating the composite image of the next frame (the process in step S3020). That is, an “operation order A” that is an operation order for executing the processing according to the flowchart of FIG. 4A is set. Since the processing according to the flowchart of FIG. 4A is as described later, description thereof is omitted here.

[Second Embodiment]
In the first embodiment, the process of determining one operation order is performed on the image processing apparatus 1100 side using various time information. However, the user of the image processing apparatus 1100 performs the determination. Anyway.

  In this case, each time information is displayed on the display screen of the image processing apparatus 1100 and presented to the user. Then, the user looks at the display screen to determine the operation order, and inputs the determined result as an instruction.

[Third Embodiment]
In the first and second embodiments, the case where a video see-through HMD is connected to the image processing apparatus 1100 has been described. In the image processing apparatus according to the present embodiment, an HMD having both an optical see-through method and a video see-through method is connected. The image processing apparatus according to this embodiment will be described below.

  FIG. 5 is a block diagram illustrating a functional configuration example of the system according to the present embodiment. The same parts as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted. As shown in FIG. 5, the system according to the present embodiment includes an image processing apparatus 1100 and an HMD 5000 as an example of a head-mounted display device. The image processing apparatus 1100 and the HMD 5000 communicate with each other. Are connected as possible. Therefore, the connection between the image processing apparatus 1100 and the HMD 5000 may be either wired or wireless.

  The HMD5000 will be described.

  Reference numeral 5010 denotes a liquid crystal shutter. By opening and closing the liquid crystal shutter 5010, it is possible to switch between direct presentation / non-presentation of the real world to the wearer. In other words, when the optical see-through method is used, the wearer is set to see the real world directly with the liquid crystal shutter 5010 opened, and when the video see-through method is used, the liquid crystal shutter is closed to directly realize the reality to the wearer. Set so that the world is not visible.

  An example of the method for determining the operation order when an HMD having both the optical see-through method and the video see-through method is connected to the image processing apparatus 1100 will be described below.

  As the process for determining the operation order in step S3050, for example, the time tr measured by the processing speed measuring unit 1150 is equal to or greater than the threshold, and the ratio of the time tr occupying the time measured by the time delay measuring unit 1160 is equal to or greater than the threshold. To do. In this case, the image composition method specifying unit 1180 determines to execute the process according to the flowchart of FIG. 4D as the process of generating the composite image of the next frame (the process in step S3020). That is, an “operation order D” that is an operation order for executing the processing according to the flowchart of FIG. 4D is set.

  First, in step S4040, the image composition unit 1200 sends an instruction to switch to the optical see-through method to the HMD 5000. That is, in the subsequent processing, the generation operation of the real space image is made non-operation.

  In step S <b> 4041, the position / orientation deriving unit 1120 obtains viewpoint position / orientation information using the real space image and the arrangement position of each reference index stored in the data storage unit 1170. At this time, the time delay measuring unit 1130 measures the time required for the position / orientation deriving unit 1120 to obtain the position / orientation information.

  In step S4042, the virtual space image generation unit 1190 generates a virtual space image that can be seen from the position and orientation indicated by the position and orientation information obtained in step S4041. The generated virtual space image is output to the HMD 5000. At this time, the processing speed measurement unit 1140 measures the time required for the virtual space image generation unit 1190 to generate the virtual space image.

  Then, the process returns to step S3030.

  In this embodiment, the HMD having both the optical see-through method and the video see-through method connected to the image processing apparatus is described as an HMD having a liquid crystal shutter. However, the HMD may be any HMD that realizes both optical see-through and video see-through methods.

  Whether or not the “operation order D” for switching to the optical see-through method can be set may be set in advance depending on the content displayed in the virtual space or the environmental conditions in which the image processing apparatus is used.

[Fourth Embodiment]
An example of the method for determining the operation order when the HMD 1000 has the chroma key composition function will be described below. That is, in this embodiment, the HMD 1000 will be described as an example of a head-mounted display device with a chroma key composition function.

  As the process for determining the operation order in step S3050, for example, the time tr measured by the processing speed measuring unit 1150 is equal to or greater than the threshold, and the ratio of the time tr occupying the time measured by the time delay measuring unit 1160 is equal to or greater than the threshold. To do. In this case, the image composition method designating unit 1180 determines to execute the process according to the flowchart of FIG. 4E as the process for generating the composite image of the next frame (the process in step S3020). That is, an “operation order E” that is an operation order for executing the processing according to the flowchart of FIG. 4E is set.

  First, in step S4050, the image input unit 1110 acquires (captures) a real space image sent from the imaging device 1010.

  Next, in step S4051, the position / orientation deriving unit 1120 uses the real space image acquired in step S4055 and the arrangement position of each reference index stored in the data storage unit 1170 to determine the position / orientation of the imaging apparatus 1010. Ask for information. At this time, the time delay measuring unit 1130 measures the time required for the position / orientation deriving unit 1120 to obtain the position / orientation information.

  In step S4052, the virtual space image generation unit 1190 generates a virtual space image that can be seen from the position and orientation indicated by the position and orientation information obtained in step S4051. At this time, the processing speed measurement unit 1140 measures the time required for the virtual space image generation unit 1190 to generate the virtual space image.

  Next, in step S4053, the image composition unit 1200 sends the real space image acquired in step S4055, the virtual space image generated in step S4052 and the chroma key composition instruction to the HMD 1000.

  In response to the chroma key composition instruction, the HMD 1000 performs a chroma key composition process of the real space image and the virtual space image to generate a composite image. Since chroma key composition is a well-known technique, a description thereof will be omitted.

  Then, the process returns to step S3030.

  Further, whether or not the “operation order E” for switching the execution of the chroma key composition function can be set may be set in advance depending on the content displayed in the virtual space or the environmental conditions in which the image processing apparatus is used. .

[Fifth Embodiment]
In the above embodiment, the description has been given assuming that all the units included in the image processing apparatus 1100 illustrated in FIG. 1 are configured by hardware. However, some of them may be configured by software. In this case, by causing a computer that implements the remaining part as hardware to execute this software, the computer performs the operation of the image processing apparatus 1100 described in the above embodiment.

  FIG. 2 is a block diagram illustrating an exemplary hardware configuration of a computer applicable to the image processing apparatus 1100.

  The CPU 2001 uses the programs and data stored in the RAM 2002 and ROM 2003 to control the entire computer, and executes the above-described processes described as being performed by the image processing apparatus 1100 in the above embodiments.

  The RAM 2002 has an area for temporarily storing programs and data loaded from the external storage device 2007 and the storage medium drive 2008. Further, the RAM 2002 has an area for temporarily storing data (in the case of FIG. 1, a real space image) received from the outside (in the case of FIG. 1, the imaging device 1010) via the I / F (interface) 2009. . Furthermore, the RAM 2002 also has a work area used when the CPU 2001 executes each process. That is, the RAM 2002 can provide various areas as appropriate. For example, the RAM 2002 also functions as the data storage unit 1170 illustrated in FIG.

  The ROM 2003 stores computer setting data, a boot program, and the like.

  A keyboard 2004 and a mouse 2005 are examples of an operation input device, and various instructions can be input to the CPU 2001 by a user of a computer.

  The display unit 2006 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 2001 as an image or text. For example, a message to be displayed for measuring the position and orientation of the imaging apparatus 1010 can be displayed on the display unit 2006.

  The external storage device 2007 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 2007 stores an OS (operating system) and programs and data for causing the CPU 2001 to execute the above-described processes described as being performed by the image processing apparatus 1100. The program corresponds to each of the time delay measurement units 1130 and 1160, the position / orientation derivation unit 1120, the image synthesis method designation unit 1180, the processing speed measurement units 1140 and 1150, the virtual space image generation unit 1190, and the image synthesis unit 1200. The program is included. The data includes virtual space data and data described as known information in the above description. Programs and data stored in the external storage device 2007 are loaded into the RAM 2002 as appropriate under the control of the CPU 2001. The CPU 2001 executes processes using the loaded program and data, thereby executing each process described above as what the image processing apparatus 1100 performs. The external storage device 2007 may be used as the data storage unit 1170 shown in FIG.

  The storage medium drive 2008 reads a program or data recorded on a storage medium such as a CD-ROM or DVD-ROM, or writes a program or data to the storage medium. Note that some or all of the programs and data described as being stored in the external storage device 2007 may be recorded in this storage medium. Programs and data read from the storage medium by the storage medium drive 2008 are output to the external storage device 2007 and the RAM 2002.

  The I / F 2009 is an analog video port for connecting the imaging device 1010 or a digital input / output port such as IEEE1394, or an Ethernet (registered trademark) port for outputting a composite image to the display device 1020 of the HMD1000. Composed. Data received via the I / F 2009 is input to the RAM 2002 and the external storage device 2007. Note that some of the functions of the image input unit 1110 illustrated in FIG. 1 are realized by the I / F 2009. Further, when a sensor system is used to acquire position and orientation information, a sensor system is connected to the I / F 2009.

  2010 is a bus connecting the above-described units.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structural example of the system which concerns on the 1st Embodiment of this invention. And FIG. 11 is a block diagram illustrating a hardware configuration example of a computer applicable to the image processing apparatus 1100. 12 is a flowchart of processing performed by the image processing apparatus 1100 to generate a composite image of a real space image and a virtual space image and output the composite image to the HMD 1000. It is a flowchart for performing the process in step S3020. It is a flowchart for performing the process in step S3020. It is a flowchart for performing the process in step S3020. It is a flowchart for performing the process in step S3020. It is a flowchart for performing the process in step S3020. It is a block diagram which shows the function structural example of the system which concerns on the 4th Embodiment of this invention.

Claims (11)

A first means for calculating position and orientation information of an imaging device included in the head-mounted display device;
A second means for generating a virtual space image based on the position and orientation information;
Third means for generating a real space image based on a result of imaging by the imaging device;
An image processing apparatus comprising: output means for outputting a composite image of the virtual space image and the real space image to the head-mounted display device,
Time information indicating time required to calculate the position and orientation information; time information indicating time required to generate the virtual space image; time information indicating time required to generate the real space image; Time information indicating the time required for processing for generating an image, and determining means for determining the operation order of the first to third means based on the acquired time information;
An image processing apparatus comprising: means for operating the first to third means in an operation order determined by the determining means.   In the case where the time tv required for generating the virtual space image is equal to or greater than a threshold, and the ratio of the time tv to the time required for all the processes for generating the composite image is equal to or greater than the threshold. The image processing according to claim 1, wherein the operation order of the first to third means is determined as the first means, the second means, and the third means. apparatus.   In the case where the time tr required to generate the real space image is equal to or greater than a threshold, and the ratio of the time tr to the time required for all the processes for generating the composite image is equal to or greater than the threshold. The image processing according to claim 1, wherein the operation order of the first to third means is determined as the third means, the first means, and the second means. apparatus.   When the time tp required for the calculation is equal to or greater than a threshold value, the determining unit determines the operation order of the first to third units as the first unit, the third unit, and the second unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined as means.   When a head-mounted display device having both an optical see-through method and a video see-through method is connected to the image processing device, the determining means includes the head-mounted display device method and the first to first methods. The image processing apparatus according to claim 1, wherein an operation order of the three units is determined.   When a head-mounted display device having both an optical see-through method and a video see-through method is connected to the image processing device, the determining means has a time tr required to generate the real space image equal to or greater than a threshold value. And when the ratio of the time tr occupying the time required for all the processes for generating the composite image is equal to or greater than a threshold, the head-mounted display device is switched to the optical see-through method, and the third 2. The image according to claim 1, wherein the first unit and the second unit are determined as the first unit and the second unit. Processing equipment.   When a head-mounted display device with a chroma key composition function is connected to the image processing device, the determination means determines the operation of the chroma key composition function and the operation order of the first to third means. The image processing apparatus according to claim 1.   When a head-mounted display device with a chroma key composition function is connected to the image processing device, the output means generates a composite image in which a time tr required to generate the real space image is equal to or greater than a threshold value. When the ratio of the time tr occupying the time required for all the processes for the processing is equal to or greater than a threshold value, the output means generates the virtual space image generated by the second means and the third means generates The image processing apparatus according to claim 1, wherein the real space image is output to the head-mounted display device with a chroma key composition function. A first means for calculating position and orientation information of an imaging device included in the head-mounted display device;
A second means for generating a virtual space image based on the position and orientation information;
Third means for generating a real space image based on a result of imaging by the imaging device;
An image processing method performed by an image processing apparatus comprising: output means for outputting a composite image of the virtual space image and the real space image to the head-mounted display device,
Time information indicating time required to calculate the position and orientation information; time information indicating time required to generate the virtual space image; time information indicating time required to generate the real space image; A determination step of acquiring time information indicating a time required for processing for generating an image, and determining an operation order of the first to third means based on the acquired time information;
And a step of operating the first to third means in the operation order determined in the determination step.   A program for causing a computer to execute the image processing method according to claim 9.   A computer-readable storage medium storing the program according to claim 10.

Images