JP2010257081A - Image procession method and image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To show an image of a viewpoint of an operator, when physical constraints are applied to a computer mannequin, and to show a gap between the position of the physical constraint applied to the computer mannequin and the viewpoint position of the operator. <P>SOLUTION: A position/attitude sensor part 204 detects the position and attitude of the operator. An avatar attitude calculating part 205 decides on the position and attitude of an avatar by using detected position/attitude information and physical constraint information set for a virtual space or a complex real space. A virtual image generating part 208 generates an image of the virtual space, based on the viewpoint of the operator. An offset calculation part 206 calculates difference data between the position and attitude of the operator and those of the avatar. Then, a mask model is displayed as a model, displaying the difference data to a composite image combining the virtual space and the real space to show the gap to the operator. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing method, an image processing apparatus, a program, and a storage medium, and more particularly, to a technique suitable for use in performing a simulation using a computer mannequin.

  Today, when designing products, living environments, product manufacturing sites, etc., it is expected to design a better environment (human-centered design) for human beings with an emphasis on safety, comfort and efficiency. ing.

  On the other hand, in recent years, it has become possible to digitize human characteristics and incorporate them into computers as mannequins. Such a mannequin taken into the computer is generally called a computer mannequin.

  An ergonomic evaluation can be performed by reproducing human characteristics in a computer using such a computer mannequin. Specifically, in a computer, a computer mannequin can be incorporated into a three-dimensional virtual space in which products, living environments, product manufacturing sites, etc. are virtually designed, and the three-dimensional virtual space can be ergonomically simulated. . And several systems which support design of a product, a living environment, a product manufacturing site, etc. using this simulation result are proposed.

  Computer mannequins can perform several evaluations, one of which is visual field evaluation. When visual field evaluation is performed, it is possible to confirm how much area can be observed from the viewpoint of a computer mannequin. For example, the visual field region is confirmed by observing an image of a three-dimensional virtual space viewed from the viewpoint of the computer mannequin.

  Further, as a related technique, there is a technique capable of operating the operation of a computer mannequin by inputting an actual human motion to a computer using an existing motion capture technology. By using this technique, the operator of the computer mannequin can check the visual field region while operating the viewpoint of the computer mannequin by changing his own viewpoint.

  For example, in a mixed reality system (hereinafter referred to as MR system) using HMD (Head Mount Display), a mixed reality in which a real space image and a non-real space (virtual space) image such as CG are synthesized. You can experience the space. For example, Non-Patent Document 1 discloses the mixed reality space technology.

H. Tamura, H. Yamamoto and A. Katayama: "Mixed reality: Future dreams seen at the border between real and virtual worlds," Computer Graphics and Applications, vol.21, no.6, pp.64-70, 2001. A. Takagi, S. Yamazaki, Y. Saito, and N. Taniguchi: "Development of a stereo video see-through HMD for AR systems", ISAR2000, pp.68-77, 2000.

  However, when visual field evaluation of a computer mannequin is performed using the above-described motion capture technology, the computer mannequin may not be able to change the viewpoint (position / posture of the head) due to physical restrictions. As a result, there may be a difference between the viewpoint of the computer mannequin operator and the viewpoint of the computer mannequin. For example, when the head of the computer mannequin is hitting the ceiling in a three-dimensional virtual space, the viewpoint position cannot be raised any further. In such a case, in the existing system, the input of the viewpoint operation by the operator of the computer mannequin is ignored, and the viewpoint image by the computer mannequin is continuously presented to the operator.

  Therefore, in such a viewpoint image presentation method, the visual field cannot be actually evaluated from the viewpoint of the operator of the computer mannequin. For example, when the head of the computer mannequin is hitting the ceiling, the visual field cannot be evaluated from the viewpoint when the height of the ceiling is increased by another 10 cm. If such an evaluation is not possible, it is difficult to know how many centimeters the ceiling height can be raised to achieve the intended field of view.

  In an existing system for such a problem, the viewpoint of the computer mannequin can be positioned above the ceiling by releasing the physical restrictions imposed on the computer mannequin. However, if the physical restriction is removed in this way, the visual field evaluation cannot be performed with the physical restriction in the three-dimensional virtual space. For example, when there are physical restrictions other than the height of the ceiling, it may be impossible to perform appropriate visual field evaluation.

  Further, when the physical restriction is released, the distance from the ceiling position to the viewpoint position, which is the physical restriction, is not known, and thus the height to be changed cannot be intuitively fed back. In addition, when the position and orientation of a computer mannequin is controlled by physical restrictions, the image presented to the operator is stopped with the viewpoint image of the computer mannequin, so it is intuitively restricted. It is difficult to judge the moment.

  Furthermore, another problem arises when the above-described viewpoint image presentation method is applied to the MR system. In a situation where the computer mannequin cannot change the viewpoint position / posture due to physical restrictions, if an image is presented to an operator who is wearing the HMD and experiencing the mixed reality space, the real image is not updated. In this way, if the computer mannequin is stopped as it is, the actual image will not be updated, so the operator will not know the surrounding situation, safety will be impaired, and the operator will feel uncomfortable. The problem arises.

  In view of the above-described problems, the present invention presents an image of an operator's viewpoint when a physical restriction is imposed on the computer mannequin, and presents the physical restriction position imposed on the computer mannequin and the operator's viewpoint. The purpose is to be able to present to the operator a deviation from the position.

  The image processing method of the present invention is an image processing method for presenting a computer mannequin to an operator in a virtual space or a mixed reality space, the detection step detecting the position and posture of the operator, and the detection in the detection step. A determination step of determining the position and orientation of the computer mannequin using the position and orientation information obtained and the physical constraint information set in the virtual space or mixed reality space, and based on the viewpoint of the operator An image generation step of generating an image of the virtual space or mixed reality space, and difference information between the position and posture of the operator and the position and posture of the computer mannequin determined in the determination step, A presentation step of presenting the image generated in the image generation step on the basis of the position and orientation of the person.

  According to the present invention, when a physical restriction is imposed on the computer mannequin, the operator is presented with an image of the operator viewpoint, and the physical restriction position imposed on the computer mannequin and the operator viewpoint are displayed. The deviation from the position can be presented to the operator.

It is a figure which shows an example of the mixed reality space produced | generated by mounting | wearing HMD. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to a first embodiment of the present invention. It is a flowchart which shows an example of the main process sequence. It is a flowchart which shows the detail of a process of step S303 of FIG. It is a flowchart which shows the detail of a process of step S403 of FIG. It is a flowchart which shows the detail of a process of step S304 of FIG. It is a figure which shows the example of the mask model with which HMD is mounted | worn and shown to the experiencing person experiencing the mixed reality space. It is a figure which shows the example which expressed the difference of the position and attitude | position between an avatar and an experience person by shifting a captured image. It is a figure which shows an example of mixed reality space. It is a flowchart which shows an example of the main process sequence. It is a figure which shows the example in which the experience person carries out subjective operation of the avatar from the remote position.

(First embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description of the embodiment, a mixed reality space information presentation method in which the present invention is applied to a mixed reality system (hereinafter referred to as MR system) using an HMD (Head Mount Display) will be described.

  In this embodiment, a case where a video see-through HMD is used will be described. The video see-through type HMD is an HMD of a system in which a non-real space image such as CG is synthesized with a real space image captured by a video camera or the like incorporated in the HMD and presented to the user. The video see-through HMD is disclosed in Non-Patent Document 2, for example.

  In the following description, in the embodiment, the display device that presents the mixed reality space to the user is a video see-through HMD, but the display device that presents the mixed reality space is not limited thereto. For example, an optical see-through HMD, a plasma display, a handheld display, or the like may be used as a display device, and any display device that can present a mixed reality space between users may be used.

  First, in the present embodiment, a video presentation method capable of presenting an experience person with constraints in a virtual space imposed on an avatar in a mixed reality space will be described. Here, the avatar is a virtual human model that behaves as a substitute for the operator (experiencer) in the mixed reality space, and is, for example, a computer mannequin. A computer mannequin is a human simulation model that simulates the movement and shape of a person. Details of restrictions in the virtual space imposed on the avatar will be described later. In the following description, “model data” represents a three-dimensional object in a virtual space.

  FIG. 1 is a diagram illustrating an example of a mixed reality space generated by wearing an HMD in the present embodiment. FIG. 1A is a diagram illustrating an example of an image obtained by objectively observing the mixed reality space generated by the system of the present embodiment, and FIG. 1B is generated by mounting the HMD. It is a figure which shows an example of the image shown to the operator who is experiencing the mixed reality space. The outline of the present embodiment will be described with reference to FIG.

  In FIG. 1A, reference numeral 101 denotes an experience person who is experiencing a mixed reality space by wearing an HMD. Reference numeral 102 denotes an avatar that is superimposed on the experience person 101. The avatar 102 is operated in conjunction with the movement of the experience person 101. Reference numeral 103 denotes virtual model data. In the example shown in FIG. 1A, since the head of the avatar 102 interferes with the virtual model data 103, the posture similar to that of the experience person 101 cannot be taken. That is, it shows that the experience person 101 is experiencing the mixed reality space with a posture that exceeds the physical constraints imposed on the avatar 102.

  On the other hand, in FIG. 1B, 104 is an image of the mixed reality space presented to the experience person 101, and is an image obtained by observing the mixed reality space shown in FIG. Reference numeral 105 denotes a mask model, which is an area displayed when the experience person 101 is observing from a viewpoint that exceeds the physical constraints imposed on the avatar 102. As described above, in the present embodiment, when the avatar is operated beyond the physical constraints in the virtual space imposed on the avatar, the fact is presented to the experience person.

FIG. 2 is a block diagram illustrating a configuration example of the image processing apparatus 200 according to the present embodiment.
In FIG. 2, reference numeral 201 denotes an imaging unit, for example, a video camera. The imaging unit 201 captures an image of the real space and outputs the captured video as an image signal to the captured image capturing unit 202.

  Reference numeral 202 denotes a captured image capturing unit that converts an image signal output from the imaging unit 201 into a format suitable for output to the image composition unit 209 and outputs the converted image signal to the image composition unit 209. An image composition unit 209 synthesizes the captured image output from the captured image capturing unit 202 and the virtual space image output from the virtual image generation unit 208 and outputs the combined image to the display unit 210.

  A position / posture sensor unit 204 functions as, for example, an optical reflection type real-time three-dimensional motion analysis system. Here, the optical reflection type real-time three-dimensional motion analysis system refers to the position / orientation of an object composed of a group of retroreflective markers by photographing a group of retroreflective markers in a real space from a plurality of cameras. It is to calculate. Then, it is output as the three-dimensional position / posture information of the object. The position / posture sensor unit 204 functions as a detection unit, acquires information about the position / posture of the imaging unit 201 and the body of the experiencer, and outputs the information to the virtual image generation unit 208, the avatar posture calculation unit 205, and the offset calculation unit 206. To do. In the present embodiment, a plurality of retroreflective markers are attached to the imaging unit 201 and acquired as position / posture data of the head of the experiencer.

  Reference numeral 205 denotes an avatar posture calculation unit that functions as a determination unit. The position / posture of the avatar is determined from the position / posture data output from the position / posture sensor unit 204 and physical constraint information in the virtual space imposed on the avatar. Calculate the data. The calculated position / posture data is output to the offset calculation unit 206 and the virtual image generation unit 208. When calculating the avatar posture, physical constraint information stored in the external storage device 207 is used. Details of the physical constraint information will be described later.

  An offset calculation unit 206 functions as a calculation unit, and calculates posture difference data between the head position of the experience person and the head position of the avatar calculated by the position / posture sensor unit 204. Then, the calculated attitude difference data is output to the avatar attitude calculation unit 205 and the virtual image generation unit 208.

  A virtual image generation unit 208 receives the position / posture data of the experience person from the position / posture sensor unit 204 and inputs the position / posture data of the avatar from the avatar posture calculation unit 205. Further, posture difference data between the head position of the experience person and the head position of the avatar is input from the offset calculation unit 206. Then, a virtual space image is generated using these data, and is output to the image composition unit 209. Note that when generating a virtual space image, model data stored in the external storage device 207 is used. The model data also includes computer mannequin data. In this embodiment, the avatar is generated using computer mannequin data. Note that the virtual image generation unit 208 and the image synthesis unit 209 function as an image synthesis unit, a model generation unit, and a synthesized image generation unit.

  Reference numeral 210 denotes a display unit that displays a composite image output from the image composition unit 209. In the present embodiment, the imaging unit 201 and the display unit 210 are incorporated in the HMD, and are installed so that the optical system of the display unit 210 and the imaging system of the imaging unit 201 coincide. A control unit 211 controls each unit of the image processing apparatus 200.

  Next, the overall flow of processing in the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of a main processing procedure performed in the image processing apparatus 200 according to the present embodiment. 3 is stored in the external storage device 207, loaded into the memory of the image processing device 200 as necessary, and executed under the control of the control unit 211 of the image processing device 200. As a result, the image processing apparatus 200 performs processing according to the flowchart of FIG.

  When the process is started in FIG. 3, in step S <b> 301, the avatar posture calculation unit 205 and the virtual image generation unit 208 read data necessary for the process from the external storage device 207. The data read in step S301 includes the structure and attributes of the virtual space such as model data shape data, arrangement data representing position / posture in the world coordinate system, and user head position / posture data in the world coordinate system. . Also, physical constraint information is included. Details of the physical constraint information will be described later.

  In step S <b> 302, the avatar posture calculation unit 205 and the offset calculation unit 206 read the position / posture data measured by the position / posture sensor unit 204. Here, the position / posture data is the position / posture of the head of the experiencer. Further, it is assumed that the read data is represented as a position / posture in the world coordinate system. Based on the acquired position / posture data, the virtual space data read in step S301 is updated. That is, the position / orientation of the model data is updated so that the virtual space corresponds to the acquired position / orientation data.

  Next, in step S303, the avatar posture calculation unit 205 determines the position / posture of the avatar based on the position / posture data of the experience person's head acquired from the position / posture sensor unit 204 and physical restrictions imposed on the avatar. Determine posture. That is, the position / posture of the avatar is determined by the position / posture data of the head of the experiencer within the imposed physical constraints. Details of step S303 will be described later.

  Next, in step S304, the virtual image generation unit 208 generates a virtual image viewed from the viewpoint of the experience person. Here, the virtual image is an image including an expression of the difference between the experience person and the avatar, which arises from physical restrictions imposed on the avatar. Details of step S304 will be described later.

  Next, in step S305, the virtual image generation unit 208 sends the virtual image generated in step S304 to the image composition unit 209. Then, the image synthesis unit 209 synthesizes the virtual image sent from the virtual image generation unit 208 and the real space image sent from the captured image capturing unit 202 and sends the synthesized image to the display unit 210. Note that the synthesized image is an image of a mixed reality space in which a virtual space is superimposed on the real space. In this embodiment, the image sent to the display unit 210 is an image of the mixed reality space. However, the displayed image is not limited to this, and only the virtual space image may be generated and sent to the display unit 210. Good.

  Next, in step S306, the display unit 210 displays the composite image sent from the image composition unit 209 in step S305. Finally, in step S307, the control unit 211 determines whether or not there has been a process end command from the operator. As a result of this determination, if there is an end command, the processing is ended as it is. On the other hand, if the result of determination in step S307 is that there is no termination command, processing returns to step S302 and steps S302 to S307 are repeated.

  Next, details of the process of determining the position / posture of the avatar based on the position / posture data of the head of the experienced person acquired from the position / posture sensor unit 204 and physical restrictions imposed on the avatar in step S303 Will be described.

FIG. 4 is a flowchart showing details of the process in step S303 of FIG.
When the processing is started in FIG. 4, in step S <b> 401, the avatar posture calculation unit 205 updates the position / posture of the avatar using the position / posture data acquired from the position / posture sensor unit 204. As described above, the position / posture data is data necessary for determining the position / posture of the avatar, and is data of the position / posture of the head of the experiencer. The position / posture is updated based on physical restrictions imposed on the avatar.

  Here, the physical restrictions imposed on the avatar include physical interference with model data, attributes of interfering model data, ergonomic characteristics, and the like. The physical interference with the model data is, for example, a restriction when the head of the avatar cannot physically move due to interference with the model data wall.

  Further, the attribute of the interfering model data is, for example, a constraint that determines whether the avatar can lean on or can be touched by the difference in the attribute of whether the model data is iron or sponge. This includes the restriction that the part cannot be touched due to the safety and security attributes of the part.

  Furthermore, the ergonomic characteristics are constraints such as kinematic characteristics and inverse kinematic characteristics of the computer mannequin. For example, there are restrictions on how much you can turn if you turn your neck in a leaning posture, and the torso moves in conjunction with turning your neck.

  In the present embodiment, existing physical simulation technology is used for physical constraints such as physical interference with model data, attributes of interfering model data, and ergonomic characteristics. In this way, physical simulation is performed using various physical constraint information recorded in the external storage device 207, and position / posture data of the avatar in the whole body is calculated. In the present embodiment, three examples of physical restrictions are given, but the physical characteristics applied to the present embodiment are not limited to this. In other words, any physical constraints that exist in the real world may be applied.

  Next, in step S402, the avatar posture calculation unit 205 determines whether the position / posture data acquired from the position / posture sensor unit 204 is position / posture data within the physical constraints imposed on the avatar. judge. As a result of this determination, if the position / posture data is out of physical constraints, the process proceeds to step S403. On the other hand, if the result of determination in step S402 is within physical constraints, the process proceeds to step S404.

  In this step S402, it is determined whether or not the avatar can physically take the posture determined by the input position / posture data of the head of the experienced user. If the input position / posture data of the user's head is outside the physical constraints imposed on the avatar, the head position of the avatar will be separated from the head position of the user. . This is because the head position of the avatar maintains its posture with priority given to physical constraints.

  Next, in step S403, the offset calculation unit 206 calculates position / posture offset data between the experience person and the avatar. Here, the difference between the position / posture data of the head of the experienced person acquired from the position / posture sensor unit 204 and the position / posture data of the avatar's head calculated in step S401 is calculated as offset data.

  Here, the offset data of the head between the experience person and the avatar is calculated in order to realize an expression when the action by the experience person falls outside the physical restrictions imposed on the avatar. The expression when the physical constraint is not satisfied is an expression that the avatar performs as much as possible within the physical constraint when the physical constraint is not satisfied. For example, when the experiencer moves such that the head of the avatar penetrates perpendicularly to the wall, the head of the avatar stops at the wall due to physical restrictions. However, when the experiencer moves at the same time as moving along the wall, the avatar's head can move along the wall physically and move along the wall.

  In order to realize such a movement, in this embodiment, a restriction is set for every six degrees of freedom for the position / posture data by the experiencer. Then, input data for a component (direction) with physical restrictions is acquired as an offset, and the avatar posture is realized by inversely converting the input data of the experience person by the offset. In other words, there is a physical restriction on the movement of the x component, and if the avatar's posture cannot be changed, the movement of the x component is acquired as an offset, and if there is no restriction on the movement of the y component, only the offset of the x component Only the y component is moved at the inversely transformed position. Details of the process in step S403 will be described later.

  Finally, in step S404, the control unit 211 of the image processing apparatus 200 determines whether or not to end the process of determining the avatar posture. As a result of this determination, if the process ends, the process ends. If not, the process returns to step S401, and the processes of steps S401 to S404 are repeated.

  Next, details of a process in which the offset calculation unit 206 calculates offset data of the position and orientation of the head between the experience person and the avatar in step S403 will be described.

FIG. 5 is a flowchart showing details of the process in step S403 of FIG.
First, in step S501, the offset calculation unit 206 calculates the difference between the position / posture data of the experiencer's head acquired from the position / posture sensor unit 204 and the position / posture data of the avatar's head calculated in step S401. Are temporarily calculated as offset data.

  There are distance and angle elements in the offset data. The distance and angle are calculated using vector components for each position / posture of 6 degrees of freedom in the world coordinate system, as in the following equation (1).

  Here, UserPos_x is the x coordinate position of the head of the experiencer, AvatorPos_x is the x coordinate position of the head of the avatar, and Distance_x is the distance offset of the x component in the world coordinates. UserPos_y is the y coordinate position of the head of the experiencer, AvatorPos_y is the y coordinate position of the head of the avatar, and Distance_y is a distance offset of the y component in the world coordinates. Furthermore, UserPos_z is the z coordinate position of the head of the experiencer, AvatorPos_z is the z coordinate position of the head of the avatar, and Distance_z is the distance offset of the z component in world coordinates.

  On the other hand, Degree_x, Degree_y, and Degree_z are obtained by angle offset instead of distance offset. That is, UserOri_x, UserOri_y, and UserOri_z are angles from the origin with respect to the x-axis, y-axis, and z-axis in the world coordinate system of the head of the experiencer, respectively. AvatorOri_x, AvatorOri_y, and AvatorOri_z are angles from the origin to the x-axis, y-axis, and z-axis in the world coordinate system of the avatar's head.

  Next, in step S502, the offset calculation unit 206 analyzes the position / posture data acquired from the position / posture sensor unit 204, and whether or not the relative position / posture relationship between the experience person and the avatar has been shortened. Determine whether. As a result of the determination, if the relative position / posture relationship has not been shortened, the process proceeds to step S504. On the other hand, if the result of determination in step S502 is that the relative position / posture relationship has been shortened, processing proceeds to step S503.

  Here, the shortening of the relative position / posture relationship means an operation for reducing the difference in the position / posture value of the head between the experience person and the avatar. To confirm whether the relative position / posture relationship has been shortened, use offset data to represent when the movement by the user is outside the physical constraints imposed on the avatar. This is because one problem occurs.

  The problem relates to the offset between the position of the experiencer and the position of the avatar that occurs when the motion by the experiencer changes from outside the physical constraints to within the constraints. For example, similarly to the above-described example, when the experience person moves such that the head of the avatar penetrates perpendicularly to the wall, the head of the avatar stops at the wall due to physical restrictions. In this state, the head of the avatar is stopped at the wall, while the head of the experiencer is recessed into the wall.

  Therefore, if the head of the experiencer moves from this state in a direction that does not sink into the wall (outside the wall model), simply updating the offset will cause the avatar's head to move closer to the outside of the wall when the experiencer's head approaches the outside of the wall The part is away from the wall. As a result, when the experiencer's head is separated from the wall, the position of the experiencer's head is separated from the position of the avatar's head. That is, if the movement within the restriction is performed after the movement outside the physical restriction, the offset remains. This is because the avatar updates the input data while keeping the offset data because there is no physical restriction in the position / posture conversion from the physical restriction to the restriction.

  Therefore, in the present embodiment, it is determined when the movement changes from outside the physical constraint to within the constraint. Specifically, when the experiencer's head is located outside the physical constraints, it is determined whether or not the experiencer's head position / posture is approaching that of the avatar's head . If the position / posture of the user's head is physically constrained to approach the position / posture of the avatar's head, the movement by the user moves from outside the physical constraints to within the constraints. It can be determined that the time has changed.

  Therefore, the determination of the presence or absence of the relative positional relationship shortening operation in step S502 is based on the position / posture data of the avatar's head at the time of processing one cycle before. Then, the position / posture data of the head of the experience person at the time of processing one cycle before is compared with the position / posture data of the head of the experience person in the current processing. That is, the offset calculation unit 206 compares the offset data calculated in step S501 during the process one cycle before with the temporary offset data. Here, the temporary offset data is obtained by subtracting the position / posture data of the avatar's head at the time of processing one cycle before from the position / posture data of the head of the experiencer in the current process. The offset data and the temporary offset data are calculated by the following equation (2).

  Here, UserPos_x ′ is the x coordinate position of the head of the experience person at the time of processing one cycle before, and AvatorPos_x ′ is the x coordinate position of the head of the avatar at the time of processing one cycle before. Distance_x ′ is a distance offset of the x component in the world coordinates at the time of processing one cycle before. On the other hand, UserPos_x is the x coordinate position of the head of the experiencer at the time of the current processing, and Distance_x * is the temporary distance offset of the x component in world coordinates.

  The distance_x ′ and the distance_x * are compared, and if the distance_x * is smaller, it is determined that the relative position / posture relationship has been shortened in the x component. The offset data and the temporary offset data are calculated for each position / posture with six degrees of freedom and are compared with each other. If one of the 6 degrees of freedom position / posture is shortened, it is determined that there is a shortening. When there is no position / posture data of the head of the experience person calculated by the process one cycle before, it is determined that there is no shortening.

  Next, in step S503, the offset calculation unit 206 corrects the offset data. At this time, the temporary offset data calculated in step S502 is corrected as offset data only for the 6-degree-of-freedom component that has been determined to be shortened in step S502. On the other hand, the offset data calculated in step S501 is used as it is for the component with 6 degrees of freedom determined that the offset data is not shortened in step S502. For example, the correction of the offset data is performed by the following equation (3).

  Here, similarly to the equation shown in Equation 2, Distance_x ′ is a distance offset of the x component in the world coordinates at the time of processing one cycle before, and Distance_x * is a temporary distance offset of the x component in the world coordinates. Distance_x is the distance offset of the x component in the world coordinates at the time of the current processing, and Distance_x_adj is the distance offset of the x component in the modified world coordinates.

  For the 6-degree-of-freedom component that has been shortened in this way, the avatar is not subjected to position / posture conversion by incorrect offset data by correcting the offset data to temporary offset data. Here, the temporary offset data is offset data that does not change the position / posture of the avatar from the position / posture of the previous cycle. That is, the position / posture of the avatar is not changed for the 6-degree-of-freedom component on which the shortening operation has been performed. As a result, when the position / posture of the experiencer's head is approaching the position / posture of the avatar's head that is physically restricted, the position / posture of the avatar's head is held.

  Finally, in step S504, the control unit 211 of the image processing apparatus 200 determines whether to end the process. As a result of this determination, if the process ends, the process ends. If not, the process returns to step S501 to repeat steps S501 to S504.

  Next, details of a process in which the virtual image generation unit 208 generates a virtual image viewed from the viewpoint of the experience person in step S304 in FIG. 3 will be described.

FIG. 6 is a flowchart showing details of the process in step S304 of FIG.
When the processing is started in FIG. 6, in step S <b> 601, the virtual image generation unit 208 generates a virtual space based on the position and orientation of the model data recorded in the external storage device 207. At this time, a virtual space is generated based on the avatar position / posture data calculated by the avatar posture calculation unit 205 in step S304. Then, a virtual image viewed from the viewpoint of the experience person is generated using the position / attitude data of the head of the experience person acquired from the position / orientation sensor unit 204.

  Next, in step S602, the virtual image generation unit 208 generates an image that combines the difference expressions of the avatar and the position and orientation of the experience person using the offset data calculated in step S403. In this embodiment, the difference between the position / posture of the avatar and the experience person is expressed by a mask model.

FIG. 7 is a diagram illustrating an example of a mask model in which the image composition unit 209 functions as a presentation unit and presents an HMD to an operator who is experiencing a mixed reality space in the present embodiment.
In FIG. 7, reference numeral 701 denotes an image of the mixed reality space presented to the experience person. Reference numeral 702 denotes a world coordinate axis, and the virtual space is generated based on the world coordinate axis 702. Reference numeral 703 denotes a mask model, which is shown as a difference information model in translucent red so that an image of the mixed reality space can be observed.

A method for generating the mask model 703 will be described below.
First, a position for generating a mask model is set according to the distance and angle components that are elements of the offset data. If the offset data of the y component exists and the value of the offset data is positive, the mask model is positioned at the top of the image (in the + y direction of the virtual image) as shown in FIG. On the other hand, when the value of the offset data is negative, the mask model is positioned below the image (−y direction of the virtual image). Similarly, when there is x component offset data, the mask model is positioned on the left and right of the image in accordance with the sign of the value.

  On the other hand, when a z-component offset exists, a slightly different representation method is used. If z-component offset data exists, the mask model representation method is changed according to the sign of the value. When the offset data value is positive, the mask model is positioned so as to cover the entire image. On the other hand, when the value of the offset data is negative, the mask model is positioned on all the upper, lower, left and right parts of the image.

  Next, the size of the mask model is set according to the distance and the angle that are elements of the offset data. If the offset data of the y component exists and its size is | Distance_y |, as shown in FIG. 7, the width of the mask model 703 in the y-axis direction (pixels in the y-axis direction covering the image) Number) is set to | Distance_y |.

  When the size of the offset data is | Degree_y |, the width of the mask model in the y-axis direction is set using the relationship between the HMD vertical field angle and the vertical resolution of the image. Specifically, the number of pixels per 1 ° angle of view is obtained from the image resolution / HMD angle of view, and the value obtained by multiplying that value by | Degree_y | is the width in the y-axis direction of the mask model (in the y-axis direction covering the image). Set as the number of pixels). Similarly, when there is x component offset data, the size of the mask model is set according to the size of the value.

  On the other hand, if there is z-component offset data and the value is negative, the size of the mask model is set according to the value. If z component offset data exists and the value is positive, the entire image is positioned to be covered with the mask model, and the size of the mask model cannot be adjusted. Therefore, when z component offset data exists, the magnitude of the offset data is expressed by the color density of the mask model.

  In the present embodiment, the size of the mask model is set by the absolute value of the offset data distance or angle × image resolution / HMD angle of view, but the present invention is not limited to this. A value obtained by multiplying the offset data by an arbitrary constant or the like may be set as the size of the mask model, and any setting method that depends on the size of the offset data may be used. Note that the operator may select from a plurality of types of mask models.

  In the present embodiment, when the offset data of the z component is positive, the method for expressing the magnitude of the offset data is expressed by the color density of the mask model positioned so as to cover the entire image. Is not limited to this. For example, any method may be used as long as it can express the size of the offset data, such as displaying the value of the offset data together with the mask model.

  Furthermore, in this embodiment, the difference between the position and orientation of the avatar and the experience person is expressed by a mask model, but the present invention is not limited to this. For example, the difference may be expressed by shifting the captured image at the moment when the offset occurs between the avatar and the experience person according to the size of the offset data. Any method can be used as long as it can express the difference in position / posture between the avatar and the experiencer.

FIG. 8 is a diagram illustrating an example in which the position / posture difference between the avatar and the experiencer is expressed by shifting the captured image in the present embodiment. The image shown in FIG. 8 is an image of the same mixed reality space as that shown in FIG. 7 from the viewpoint of the user.
In FIG. 8, reference numeral 801 denotes an image of the mixed reality space presented to the experience person. Reference numeral 802 denotes a captured image, which is a captured image captured by the imaging unit 201 at the moment when an offset occurs between the avatar and the experience person. This captured image 802 is not updated while the position / posture data of the experience person exceeds the physical restrictions imposed on the avatar. Furthermore, the captured image 802 is displayed while being shifted in the mixed reality space image 801 in accordance with the size of the offset data.

  Reference numeral 803 denotes model data, and the position / posture of the model data 803 is updated while the position / posture data of the experience person exceeds the physical restrictions imposed on the avatar. Reference numeral 804 denotes a blue background, which is generated as a result of the captured image being shifted by the position / posture difference between the avatar and the experience person. In other words, if the experiencer's position / posture data exceeds the physical constraints imposed on the avatar, the update of the real image is stopped, and only the virtual image is updated, so Expresses the difference between position and orientation. In this way, it is also possible to express the position / posture difference between the avatar and the experience person.

  Finally, in step S603, the control unit 211 of the image processing apparatus 200 determines whether the processing is finished. As a result of this determination, if it is the end, the process is ended. If it is not ended, the process returns to step S601 and steps S601 to S603 are repeated.

  In this embodiment, the data input to determine the whole body posture of the avatar is the position / posture data of the head of the experiencer, but the method for determining the whole body posture of the avatar in the present invention is not limited to this. . In order to determine the whole body posture of the avatar, not only the head position / posture data of the experience person but also the position / posture data of the hand of the experience person, and also the position / posture data of the whole body are inputted to input the whole body of the avatar. The posture may be determined. Further, the whole body posture of the avatar may be determined by inputting numerical values with an external input device such as a keyboard. Thus, any method may be used as long as it can determine the whole body posture of the avatar.

  In the above description, in the differential expression presented to the experiencer, the position / posture difference between the head of the avatar and the experiencer's head is expressed. Not as long. For example, the difference in position / posture between the avatar's hand and the experiencer's hand may be presented to the experiencer as a differential expression. As described above, any position / posture between the whole body of the experience person and the whole body of the avatar may be used as long as it expresses a difference of at least one part. In addition, when expressing the difference in the position / posture of the hand, for example, the difference may be expressed by displaying a line between the hands of the experiencer and the hands of the avatar.

  As described above, according to the present embodiment, the constraints in the virtual space imposed on the avatar can be presented to the experience person.

(Second Embodiment)
In the first embodiment, an example has been described in which the experiencer receives the presentation of constraints in the virtual space imposed on the avatar while constantly manipulating the whole body posture of the avatar according to the change of the posture of himself / herself. In this embodiment, the experience person can receive the presentation of the restrictions in the virtual space imposed on the avatar without operating the whole body posture of the avatar. Specifically, by switching the avatar's whole body posture between the subjective operation and the objective operation, it is possible to experience the restrictions in the virtual space imposed on the avatar by switching between the subjectivity and the objective.

  In the present embodiment, a video presentation method capable of presenting to a user an experience that is switched between subjective and objective restrictions in a virtual space imposed on an avatar in a mixed reality space will be described. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the first embodiment, and a description thereof will be omitted.

  FIG. 9 is a diagram illustrating an example of the mixed reality space generated in the present embodiment. FIG. 9A is a diagram illustrating an example of an image obtained by objectively observing the mixed reality space generated by the system of the present embodiment, and FIG. 9B is generated by mounting the HMD. It is a figure which shows an example of the image shown to the experiencing person experiencing the mixed reality space. The outline of the present embodiment will be described with reference to FIG.

  In FIG. 9A, reference numeral 901 denotes an experience person who experiences a mixed reality space by wearing an HMD. Reference numeral 902 denotes an avatar, and the whole body posture is operated by the experience person 901. Reference numeral 903 denotes virtual model data. A three-dimensional point device 904 can acquire a three-dimensional position by the position / posture sensor unit 204. Note that the three-dimensional point device 904 further has an input function such as a button. The experience person 901 can operate the whole body posture by grasping and moving a part of the body of the avatar 902 using the three-dimensional point device 904.

  On the other hand, in FIG. 9B, reference numeral 905 denotes an avatar that is objectively observed by the experience person 901. A three-dimensional point device 906 is used to hook the head of the avatar 905 and operate the whole body posture of the avatar 905.

  In this embodiment, as in the first embodiment, the subjective operation of manipulating the whole body posture of the avatar according to the position and posture of the head of the experiencer, and the experiencer uses some 3D point device as described above. It is possible to switch between objective operation for manipulating the whole body posture of the avatar. As described above, in the present embodiment, the mixed reality space can be operated and experienced.

  Here, the subjective operation and the objective operation are clearly defined. The subjective operation is an operation for controlling the position of the avatar's head according to the position of the head of the experiencer as shown in the first embodiment. That is, the experiencer is operating the avatar by a part physically corresponding to the avatar being transferred to the experiencer. On the other hand, objective operation is an operation to select and control the avatar part by control independent of the position and posture of the operator's viewpoint while objectively observing the avatar while the experiencer and the avatar are separated from each other It is.

  Next, the overall processing flow in the present embodiment will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of a main processing procedure performed in the image processing apparatus 200 according to the present embodiment. Steps S1001 to S1003 are the same as steps S301 to S303 in FIG. 3, and steps S1008 to S1010 are the same as steps S305 to S307 in FIG. Of the processes of the present embodiment shown in FIG. 10, the only processes different from the first embodiment shown in FIG. 3 are steps S1004 to S1007. Therefore, in the present embodiment, only the processing of steps S1004 to S1007 will be described. In the present embodiment, the avatar is determined to have all postures through subjective operations by the experience person shown in the first embodiment.

  In step S1004, the avatar posture calculation unit 205 functions as a switching unit, and switches the operation unit using the offset data calculated by the offset calculation unit 206. Specifically, when the size of the offset data calculated by the offset calculation unit 206 is within the threshold value, the subjective operation is continuously performed on the avatar. On the other hand, when the size of the offset data is outside the threshold, the operation means for the avatar is switched from the subjective operation to the objective operation. When the avatar operation means is switched from the subjective operation to the objective operation, the avatar is separated from the experience person's operation.

  Next, in step S1005, the avatar posture calculation unit 205 determines whether or not the avatar whole body posture operation means is a subjective operation. As a result of the determination, if the operating means is a subjective operation, the process proceeds to step S1006, and if it is an objective operation, the process proceeds to step S1007.

  In step S1006, similarly to step S304 of FIG. 3 described in the first embodiment, the virtual image generation unit 208 generates a virtual image viewed from the viewpoint of the experience person. On the other hand, also in step S1007, the virtual image generation unit 208 generates a virtual image, but the virtual image does not include a representation of the difference between the experience person and the avatar resulting from physical restrictions imposed on the avatar. It is an image. Experiencer can objectively observe the physical constraints imposed on the avatar, and in the state where the avatar and the experiencer are separated, the difference between the experiencer and the avatar is meaningless, so the expression of the difference is It can be said that it is not necessary.

  In this embodiment, the switching of the avatar operation means is performed using the offset data threshold value, but the present invention is not limited to this. For example, the avatar operation means may be switched using a switching button on the console screen, and any method may be used as long as the avatar operation means can be switched.

  In this embodiment, when the avatar operation means is a subjective operation, the avatar is superimposed on the experience person, but the avatar may be subjectively operated while objectively observing the avatar. . In this case, the experience person can operate the avatar remotely and subjectively operate it from a position away from the avatar.

FIG. 11 is a diagram illustrating an example in which an experienced person subjectively operates an avatar from a remote position in the present embodiment, and an example of an objectively observed image in the mixed reality space generated by the system of the present embodiment. It is.
In FIG. 11, reference numeral 1101 denotes an experience person who is experiencing a mixed reality space by wearing an HMD. Reference numeral 1102 denotes an avatar, and the whole body posture is operated by the experience person 1101 at a remote position. Reference numeral 1103 denotes model data, which gives physical restrictions to the avatar 1102. Reference numeral 1104 denotes an actual desk, which is used by the experience person 1101 to simulate the posture of the avatar 1102 as much as possible. As in the example shown in FIG. 11, the avatar operation means and the viewpoint for observing the avatar can be set to be subjective or objective separately.

  As described above, according to the image processing method of the present embodiment, the constraints in the virtual space imposed on the avatar can be presented to the experience person while switching between subjective and objective.

(Other embodiments according to the present invention)
Each means constituting the image processing apparatus and each step of the image processing method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable storage medium storing the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program for realizing the functions of the above-described embodiments (in the embodiment, programs corresponding to the flowcharts shown in FIGS. 3 to 6 and 10) is directly or remotely supplied to the system or apparatus. This includes cases where This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the storage medium for supplying the program include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. It can also be supplied by downloading the computer program itself of the present invention or a compressed file including an automatic installation function from a homepage to a storage medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, the present invention includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

  As another method, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  As another method, a program read from a storage medium is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

  204 position / posture sensor unit, 205 avatar posture calculation unit, 208 virtual image generation unit, 209 image composition unit

Claims (10)

An image processing method for presenting a computer mannequin to an operator in a virtual space or a mixed reality space,
A detection step of detecting the position and orientation of the operator;
A determination step of determining the position and orientation of the computer mannequin using the position and orientation information detected in the detection step and the physical constraint information set in the virtual space or mixed reality space;
An image generating step for generating an image of the virtual space or mixed reality space based on the viewpoint of the operator;
The difference information between the position and posture of the operator and the position and posture of the computer mannequin determined in the determination step is calculated on the image generated in the image generation step based on the position and posture of the operator. The image processing method characterized by including the presentation process shown by this. The presenting step includes
A calculation step of calculating the difference information using the position and orientation information of the operator and the position and orientation information of the computer mannequin;
A model generation step of generating a difference information model using the vector component of the difference information calculated in the calculation step;
The image processing method according to claim 1, further comprising a combined image generation step of generating an image obtained by combining the difference information model generated in the model generation step and the virtual space or the mixed reality space.   3. The image processing according to claim 2, wherein in the calculating step, the difference information is calculated using position and orientation information of the viewpoint of the operator and position and orientation information of the viewpoint of the computer mannequin. Method.   The physical constraint is a constraint due to physical interference with the model data of the virtual space, an attribute of the model data of the virtual space, or an ergonomic characteristic. The image processing method according to claim 1.   4. The model generation step is capable of generating a plurality of types of difference information models, and selecting and generating one of the plurality of types of difference information models. An image processing method described in 1.   The method further comprises a switching step of switching control of the computer mannequin between control based on the position and orientation of the operator's viewpoint and control independent of the position and orientation of the operator's viewpoint. The image processing method according to any one of 1 to 5. An image processing apparatus for presenting a computer mannequin to an operator in a virtual space or a mixed reality space,
Detecting means for detecting the position and posture of the operator;
Determining means for determining the position and orientation of the computer mannequin using the position and orientation information detected by the detecting means and the physical constraint information set in the virtual space or mixed reality space;
Image generating means for generating an image of the virtual space or mixed reality space based on the viewpoint of the operator;
The difference information between the position and posture of the operator and the position and posture of the computer mannequin determined by the determination unit is used to calculate the difference information on the image generated by the image generation unit based on the position and posture of the operator. An image processing apparatus comprising: a presenting means for presenting at the above.   The apparatus further comprises switching means for switching the control of the computer mannequin between control based on the position and posture of the operator's viewpoint and control independent of the position and posture of the operator's viewpoint. 8. The image processing apparatus according to 7. A program for causing a computer to execute each step of an image processing method for presenting a computer mannequin to an operator in a virtual space or a mixed reality space,
A detection step of detecting the position and orientation of the operator;
A determination step of determining the position and orientation of the computer mannequin using the position and orientation information detected in the detection step and the physical constraint information set in the virtual space or mixed reality space;
An image generating step for generating an image of the virtual space or mixed reality space based on the viewpoint of the operator;
The difference information between the position and posture of the operator and the position and posture of the computer mannequin determined in the determination step is calculated on the image generated in the image generation step based on the position and posture of the operator. A program for causing a computer to execute the presenting step presented in the above.   A computer-readable storage medium storing the program according to claim 9.

Images