JP2019040356A - Image processing system, image processing method and computer program - Google Patents

Abstract

To provide a technique for efficiently performing image processing of a human body model expressing a posture of a human body of a user in a composite real space.SOLUTION: An image processing system determines, according to a position or a visual line direction of a user, whether a condition for generating, from human body information obtained by measuring a posture of a human body of the user, a human body model expressing the posture of the human body of the user is satisfied or not. If it is determined that the condition is satisfied, the image processing system generates a human body model, generates an image including the human body model, and outputs to a display device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique related to presentation of a mixed reality space.

  In recent years, mixed reality (hereinafter referred to as MR) technology has become widespread. By using the mixed reality technology, a mixed reality image in which a CG model is arranged on a real image is provided to a user wearing a head mounted display (HMD), and the mixed reality world (mixed reality) is combined with reality and virtual. Space) can be experienced (for example, Patent Document 1). In generating a mixed reality image, a method of specifying the position of the HMD and the position of the CG model using a sensor or a two-dimensional marker is used.

  Patent Document 2 discloses a technique of combining an image including an experienced person photographed by an objective camera and a CG image projected when the experienced person is experiencing and displaying the synthesized image on a display.

  In addition, with the development of technology, motion capture technology can be used, and a mechanism that combines mixed reality technology and motion capture technology has been attempted.

JP 2003-308514 A Japanese Patent Laying-Open No. 2015-170232

  However, in an experiential video system using an HMD such as mixed reality, a process of reflecting a marker coordinate value obtained by motion capture on a human body model (CG) is generated instead of simply displaying the coordinate value. Specifically, it is necessary to repeat the process of matching the human body model to the posture of the person based on the coordinate value of the marker for one second for the number of playback frames (for example, 60 frames / second or 120 frames / second), and the processing load is high. . Particularly in the mixed reality system, since there is a synthesis process with a real image, the processing load is further increased.

  The present invention provides a technique for efficiently performing image processing of a human body model representing the posture of a user's human body in a mixed reality space.

  In order to achieve the object of the present invention, for example, the image processing apparatus of the present invention has the following configuration. That is, based on the position or line-of-sight direction of the user, it is determined whether or not a condition for generating a human body model representing the posture of the user's human body is satisfied from human body information obtained by measuring the posture of the user's human body. A determination unit; and an output unit that generates the human body model from the human body information, generates an image including the human body model, and outputs the generated image to a display device when the determination unit determines that the condition is satisfied. It is characterized by.

  According to the present invention, it is possible to efficiently perform image processing of a human body model representing the posture of the user's human body in the mixed reality space.

The figure which shows the outline | summary of a system. FIG. 3 is a block diagram showing an example of the hardware configuration of each of the HMD 101 and the PC 100. The block diagram which shows the function structural example of PC100. The block diagram which shows the module structural example of PC100. The flowchart of the process which HMD101 and PC100 perform. The flowchart of the process in step S522. The figure which shows the structural example of the information hold | maintained at the external memory. The flowchart of the process in step S602. The flowchart of the process in step S603. The flowchart of the process in step S604. The flowchart of the process in step S811. The figure which shows the example of a display of the objective image display 160. FIG. The figure which shows an example of the image of mixed reality space. The figure which shows the example of a display of preservation | save information.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In the system according to the present embodiment, a mixed reality space that is a composite space of a real space and a virtual space is used in a head-mounted display device such as an HMD or a display device in which a head-mounted display device is provided separately. It is a mixed reality presentation system. First, an outline of a system according to the present embodiment will be described with reference to FIG.

  The HMD 101 is attached to the user's head, and an image of the mixed reality space output from the PC 100 as the image processing apparatus is displayed on the display screen of the HMD 101. The user experiences the mixed reality space by observing the mixed reality space image displayed on the display screen of the HMD 101.

  The optical marker 103 that can be detected (measured) by the optical sensor 104 is attached to the HMD 101, and the measurement result of the optical marker 103 by the optical sensor 104 is sent to the PC 100. In FIG. 1, in order to simplify the illustration, the number of the optical sensors 104 is set to 1, but a plurality of optical sensors 104 are actually arranged. The PC 100 obtains the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104. Since a technique for obtaining the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104 is well known, a detailed description thereof will be omitted. For example, when the camera provided in the HMD 101 is used as the user's viewpoint (user viewpoint), the PC 100 uses the “relative position and orientation relationship between the user viewpoint and the optical marker 103” obtained in advance. Then, the position and orientation of the optical marker 103 are converted into the position and orientation of the user viewpoint.

  The method for acquiring the position and orientation of the user viewpoint is not limited to a specific method. For example, the position and orientation measurement sensor used to obtain the position and orientation of the user viewpoint is not limited to the optical sensor 104, but the position and orientation of the user viewpoint using other types of position and orientation measurement sensors such as a magnetic sensor and an ultrasonic sensor. You may make it acquire. Alternatively, the two-dimensional marker 102 arranged in the real space may be imaged by a camera provided in the HMD 101, and the position and orientation of the user viewpoint may be obtained using a captured image obtained by the imaging. Further, the position and orientation of the user viewpoint may be acquired by using the position and orientation measurement sensor and the two-dimensional marker together.

  Then, the PC 100 generates a virtual object such as the CG model 130 and arranges it in the virtual space, and generates an image of the virtual object viewed from the user viewpoint. The PC 100 generates a composite image obtained by synthesizing the generated virtual object image and the real space image captured by the camera of the HMD 101 as an image of the mixed reality space, and the generated mixed reality space image. Is sent to the HMD 101. Thereby, the user can observe the image of the mixed reality space according to the position and orientation of his / her viewpoint in front of the eyes.

  An objective camera 106 that captures an image of a user experiencing the mixed reality space is arranged in the real space, and an image captured by the objective camera 106 is transmitted to the PC 100. In FIG. 1, the number of objective cameras 106 is set to 1 in order to simplify the illustration, but a plurality of objective cameras 106 may be arranged. The PC 100 uses the position and orientation (known) of the objective camera 106 to generate an image of a virtual object such as the CG model 130 that can be seen from the viewpoint having the position and orientation, and the generated image is captured by the objective camera 106. The image is combined with the real space image and sent to the objective image display 160. That is, the objective image display 160 displays an image of the mixed reality space viewed from a viewpoint (objective viewpoint) different from the viewpoint (subjective viewpoint) of the user who experiences the mixed reality space.

  In addition, a plurality of motion markers 105 used in the motion capture technique are attached to the user's human body. These motion markers 105 are measured by the optical sensor 104, and the measurement results are sent to the PC 100. The motion marker 105 can be measured by a dedicated motion sensor capable of measuring the motion marker, and the system applicable to the motion capture system for measuring the motion marker 105 is not limited to a specific system. The PC 100 obtains the three-dimensional position of each motion marker 105 based on the measurement result of the motion marker 105 by the optical sensor 104. Since a technique for obtaining the three-dimensional position of the motion marker 105 based on the measurement result of the motion marker 105 by the optical sensor 104 is well known, a detailed description thereof will be omitted. The PC 100 generates a human body model representing the current posture of the user based on the three-dimensional position of the motion marker 105 when a condition described later is satisfied, and can be viewed from the viewpoint having the position and posture of the objective camera 106. An image of the human body model is generated, and the generated image is combined with an image captured by the objective camera 106 and output to the objective image display 160. That is, when a condition described later is satisfied, the objective image display 160 includes a virtual object and a human body model image generated according to the position and orientation of the objective camera 106 and an image captured by the objective camera 106. When the synthesized composite image is displayed as an image of the mixed reality space and a condition described later is not satisfied, the objective image display 160 is generated according to the position and orientation of the objective camera 106. A combined image obtained by combining the image of the virtual object and the image captured by the objective camera 106 is displayed as an image of the mixed reality space. The human body model may be, for example, a three-dimensional model obtained by connecting the three-dimensional positions of the motion marker 105 with a line segment model, a three-dimensional model having the three-dimensional position of the motion marker 105 as a joint position, or the motion marker 105. It is also possible to use a three-dimensional model whose contour is the three-dimensional position.

  Next, a hardware configuration example of each of the HMD 101 and the PC 100 will be described with reference to the block diagram of FIG. The configuration illustrated in FIG. 2 is an example of a hardware configuration applicable to the HMD 101 and the PC 100, and is not limited to the configuration illustrated in FIG.

  First, the HMD 101 will be described. The right-eye / left-eye video camera 221 includes a right-eye video camera for capturing a real-space image provided to the user's right eye, a left-eye video camera for capturing a real-space image provided to the user's left eye, Images taken by the respective video cameras (real space images) are sent to the PC 100.

  The right-eye / left-eye display 222 includes a right-eye display for providing video (including images and characters) to the user's right eye, and a left-eye display for providing video (including images and characters) to the user's left eye. Have. The right-eye video sent from the PC 100 is displayed on the right-eye display, and the left-eye video is displayed on the left-eye display.

  Next, the PC 100 will be described. The general-purpose bus 212 functions as an interface for receiving a real space image transmitted from the right-eye / left-eye video camera 221 and includes, for example, an external input terminal (for example, an IEEE 1394 terminal).

  The CPU 201 executes various processes using computer programs and data stored in the ROM 202 and RAM 203. As a result, the CPU 201 controls the operation of the entire PC 100 and executes or controls each process described later as what the PC 100 performs.

  The ROM 202 stores rewrite-free computer programs and data, for example, BIOS (Basic Input / Output System) which is a control program of the CPU 201.

  The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 and the external memory 211. The RAM 203 has an area for storing an image of the real space received from the right-eye / left-eye video camera 221 via the general-purpose bus 212. The RAM 203 has an area for storing measurement results received from the optical sensor 104 via the communication I / F controller 208. The RAM 203 has an area for storing an image of the real space received from the objective camera 106 via the communication I / F controller 208. The RAM 203 has a work area used when the CPU 201 executes various processes. Thus, the RAM 203 can provide various areas as appropriate.

  The video controller 206 is for performing display control of the right eye / left eye display 222, the display 210, and the objective image display 160. For the right-eye / left-eye display 222, for example, an external output terminal (for example, Digital Visual Interface) is used to output a video signal (image or character display signal). The display 210 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 201 using an image, text, or the like.

  The input controller 205 is for performing operation control of the input device 209. The input device 209 functions as a user interface such as a mouse and a keyboard, and can input various instructions to the CPU 201 by a user operation. For example, when the input device 209 is a touch panel, the user can input various instructions by pressing (touching with a finger or the like) an icon, a cursor, or a button displayed on the touch panel. The touch panel may be a touch panel that can detect a position touched by a plurality of fingers, such as a multi-touch screen.

  The memory controller 207 is for controlling reading and writing of information to and from the external memory 211. The external memory 211 is a memory device such as a hard disk drive device, a flexible disk, a card type memory connected to the PCMCIA card slot via an adapter. The external memory 211 stores an OS (Operating System) and computer programs and data for causing the CPU 201 to execute or control each process described below as performed by the PC 100.

  The computer program stored in the external memory 211 includes a computer program for causing the CPU 201 to execute processing according to each flowchart described below. The data stored in the external memory 211 includes information handled as known information in the following embodiments including the present embodiment, and data (drawing data) required for drawing various virtual objects including the CG model 130. )It is included. Computer programs and data stored in the external memory 211 are appropriately loaded into the RAM 203 under the control of the CPU 201 and are processed by the CPU 201. The external memory 211 may be a memory device such as a USB memory or an SD card.

  The communication I / F controller 208 functions as an interface for receiving measurement results and real space images from the optical sensor 104 and the objective camera 106. The communication I / F controller 208 functions as an interface for performing data communication with an external device via a network such as a LAN or the Internet. For example, the communication I / F controller 208 can perform Internet communication using TCP / IP. The communication I / F controller 208 also controls communication with the optical sensor 104 through Gigabit Ethernet (registered trademark) or the like. In FIG. 1, data communication between the optical sensor 104, the objective camera 106, the objective image display 160, and the PC 100 is wired, but may be wireless. For example, the PC 100 can perform data communication with the optical sensor 104, the objective image display 160, and the objective camera 106 by various communication forms such as a LAN cable, a USB cable, and wireless communication.

  The general-purpose bus 212, CPU 201, ROM 202, RAM 203, video controller 206, input controller 205, memory controller 207, and communication I / F controller 208 are all connected to the system bus 204.

  In the present embodiment, the HMD 101 and the PC 100 are separate devices, but the HMD 101 and the PC 100 may be integrated. Further, the PC 100 may be handled as a server device.

  Next, a module configuration example of the PC 100 will be described with reference to the block diagram of FIG. The operating system 401 controls input / output to / from the HMD 101, and transfers an image of the real space obtained from the right-eye / left-eye video camera 221 via the general-purpose bus 212 to the mixed reality platform 403. Further, the operating system 401 outputs the mixed reality space image drawn by the graphic engine 402 to the right-eye / left-eye display 222 via the video controller 206.

  The graphic engine 402 generates a virtual object image using the virtual object drawing data stored in the external memory 211 and superimposes the generated virtual object image on the image captured by the right-eye / left-eye video camera 221. Thus, an image of the mixed reality space is generated. An engine used for generating (drawing) an image of a virtual object may be a widely used graphic engine such as OpenGL or DirectX, or a graphic engine developed independently. In the present embodiment, OpenGL is used as the graphic library.

  The mixed reality platform (also referred to as MR platform) 403 obtains the position and orientation of the user's viewpoint as described above, and aligns the real space and the virtual space. These techniques can be realized by using existing techniques disclosed in, for example, Japanese Unexamined Patent Application Publication Nos. 2002-32784, 2006-072903, and 2007-166427.

  A mixed reality application (also referred to as an MR application or a viewer application) 404 is required to draw a virtual object such as a viewpoint position and orientation, information on the color and shape of a virtual object, and a position and orientation of a virtual object from the mixed reality platform 403. The information is received, and a drawing command for a virtual object image is issued to the graphic engine 402. At this time, using the OpenGL API, a command in which the identification information and the position and orientation information of the virtual object to be drawn are set is issued.

  Further, when the mixed reality platform 403 receives a captured image from the objective camera 106, the mixed reality platform 403 delivers the captured image to the mixed reality application 404. The motion capture system 405 obtains the three-dimensional position of the motion marker 105 based on the measurement result of the motion marker 105 by the optical sensor 104. The mixed reality application 404 includes drawing information of a human body model (when a later-described condition is satisfied) corresponding to the three-dimensional position of the motion marker 105 acquired from the motion capture system 405 and the human body model, a captured image, and a virtual image An object drawing command is output to the graphic engine 402. The graphic engine 402 generates a mixed reality space image in which a human body model and a virtual object are superimposed on a captured image, and outputs the mixed reality space image to the objective image display 160.

  Next, a process performed by the HMD 101 and the PC 100 to generate one (one frame) mixed reality space image and display it on the HMD 101 or the objective image display 160 will be described with reference to the flowchart of FIG.

  In step S <b> 503, the right-eye / left-eye video camera 221 outputs the captured real space image (the image captured by the right-eye camera and the image captured by the left-eye camera) to the PC 100.

  In step S504, the CPU 201 receives the real-space image output from the right-eye / left-eye video camera 221 via the general-purpose bus 212. In step S505, the CPU 201 receives the received real-space image in the RAM 203 or the external memory 211. To store.

  In step S506, the CPU 201 acquires the measurement result of the optical marker 103 by the optical sensor 104 via the communication I / F controller 208, and based on the acquired measurement result, the position and orientation of the user viewpoint as described above. (HMD 101 position and orientation) is obtained. The position and orientation of the user viewpoint obtained in step S506 are the position and orientation of the right viewpoint (right-eye video camera) and the position and orientation of the left viewpoint (left-eye video camera). As with the above, the right viewpoint is obtained by converting the position and orientation of the optical marker 103 using the “relative position and orientation relationship between the right viewpoint and the optical marker 103” obtained in advance. Can do. Similarly, for the left viewpoint, the position and orientation of the optical marker 103 are converted using the “relative position and orientation relationship between the left viewpoint and the optical marker 103” obtained in the same manner as described above. Can be obtained.

  In step S507, the CPU 201 stores the viewpoint position and orientation obtained in step S506 in the external memory 211 in association with the identification information (ID) of the HMD 101. For example, as shown in FIG. 7A, the ID of the HMD 101 is registered in the column 1111 of the HMD information 1110 managed in association with the ID, position, and orientation for each HMD, and the location of the HMD 101 is registered in the column 1112. In the column 1113, the attitude of the HMD 101 is registered. The HMD information 1110 is stored in the external memory 211.

  In step S508, the CPU 201 reads the virtual object drawing data stored in the external memory 211 into the RAM 203, generates a virtual object based on the read drawing data, and places the generated virtual object in the virtual space. To do. Then, the CPU 201 generates an image of the virtual object viewed from the viewpoint based on the position and orientation of the viewpoint obtained in step S506. Then, the CPU 201 generates a mixed reality space image (MR image) by superimposing the virtual object image on the real space image received in step S504. Strictly speaking, the CPU 201 superimposes the image of the virtual object generated based on the position and orientation of the right viewpoint on the image captured by the right-eye camera (image in the real space), so that the image of the mixed reality space for the right eye is superimposed. And the virtual object image generated based on the position and orientation of the left viewpoint is superimposed on the image captured by the left-eye camera (real space image). Generate.

  In step S509, the CPU 201 sends the mixed reality space image (the mixed reality space image for the right eye and the mixed reality space image for the left eye) generated in step S508 to the HMD 101 and the display 210 via the video controller 206. Output.

  In step S510, the right eye / left eye display 222 receives the mixed reality space image for the right eye and the mixed reality space image for the left eye transmitted from the PC 100 in step S509. In step S511, the right-eye / left-eye display 222 displays the right-eye mixed reality space image on the right-eye display and the left-eye mixed reality space image on the left-eye display.

  An example of the mixed reality space image displayed on the right-eye / left-eye display 222 is shown in FIG. FIG. 13A shows an example of an image of the mixed reality space based on the user viewpoint for observing the CG model 130. FIG. 13B shows an example of an image of the mixed reality space based on the user viewpoint located below the CG model 130, and the user's arm 1301 is shown in addition to the CG model 130.

  Further, the PC 100 also executes the processes of steps S520 to S524 in parallel with the processes of steps S504 to S509.

  In step S520, the CPU 201 acquires the measurement result of the motion marker 105 by the optical sensor 104 via the communication I / F controller 208, and uses the three-dimensional position of the motion marker 105 as human body information based on the acquired measurement result. Ask.

  In step S521, the CPU 201 stores the human body information obtained in step S520 in the external memory 211.

  In step S522, the CPU 201 executes processing according to the flowchart of FIG.

  In step S601, the CPU 201 determines whether or not to generate the above human body model based on the human body information (whether or not to generate the human body model and superimpose it on the image captured by the objective camera 106). Among the plurality of conditions (modes) for determining, the mode currently set in the PC 100 is determined.

  As a result of this determination, if the mode currently set in the PC 100 is the “proximity / designated position mode”, the process proceeds to step S602. If the mode currently set in the PC 100 is “interference mode”, the process proceeds to step S603. If the mode currently set in the PC 100 is the “line-of-sight mode”, the process proceeds to step S604.

  Details of the processing in step S602 will be described with reference to FIG. 8 showing a flowchart of the processing.

  In step S801, the CPU 201 obtains the position of the viewpoint obtained in step S506 (which may be a right viewpoint or a left viewpoint).

  Here, the “proximity / designated position mode” includes “designated position proximity mode” and “CG model proximity mode”. In step S <b> 802, the CPU 201 determines whether the mode currently set for the PC 100 is the “specified position proximity mode” in the “proximity / specified position mode” or the “CG model proximity mode”. As a result of this determination, if the mode currently set in the PC 100 is “specified position proximity mode” in the “proximity / specified position mode”, the process proceeds to step S804, and the mode currently set in the PC 100 is “ If it is “CG model proximity mode” in “proximity / designated position mode”, the process proceeds to step S803.

  In step S803, the CPU 201 obtains a distance between the viewpoint position obtained in step S506 and the arrangement position of a specific virtual object among the virtual objects arranged in the virtual space.

  Here, as shown in FIG. 7C, the external memory 211 stores CG model information 1120 for managing the respective virtual objects arranged in the virtual space. An ID (model ID) unique to the virtual object is registered in the column 1121 in the CG model information 1120, and a file name (model name) of the drawing data of the virtual object is registered in the column 1122. In 1123, a path (file path) of a file in which drawing data of the virtual object is stored is registered, in the column 1124, the arrangement position of the virtual object is registered, and in the column 1125, the virtual object is stored. Is registered. However, in step S508 described above, the CPU 201 refers to the CG model information 1120 stored in the external memory 211, and is registered in each of the column 1122 and the column 1123 corresponding to the virtual object ID to be rendered. The virtual object drawing data is read based on the model name and the file path, the virtual object is generated based on the read drawing data, and the generated virtual object is displayed in columns 1124 and 1125 corresponding to the ID. Are arranged in the virtual space with the positions and postures registered in each of them.

  Further, the value (ON or OFF) of the human body control flag for the virtual object is registered in the column 1126 of the CG model information 1120. In step S803, the CPU 201 arranges the position of the viewpoint obtained in step S506 and the virtual object whose human body control flag value is “ON” among the virtual objects registered in the CG model information 1120 (column 1124). Find the distance between and. In the case of FIG. 7C, the distance between the position of the virtual object corresponding to the model ID = M111 and the virtual object corresponding to the model ID = M333 and the position of the viewpoint is obtained.

  On the other hand, in step S804, the CPU 201 obtains the distance between the position of the viewpoint obtained in step S506 and the specified position (designated position designated in advance) in the real space (or virtual space).

  In step S <b> 805, the CPU 201 acquires a specified distance as a set value that is set in advance for the current mode. For example, as shown in FIG. 7B, a mode setting 1130 indicating a mode currently set in the PC 100 is registered in the external memory 211, and a column 1131 in the mode setting 1130 is currently set in the PC 100. The mode is registered, and a setting value corresponding to the mode currently set in the PC 100 is registered in the column 1132. In the mode setting 1130 shown in FIG. 7B, the “specified position proximity mode” is set as the mode set in the PC 100, and “50 cm” is set as the setting value corresponding to the “specified position proximity mode”. Has been. In this case, however, in step S805, “50 cm” is acquired as the set distance from the mode setting 1130 as the specified distance.

  In step S806, the CPU 201 compares the distance d obtained in step S803 or step S804 with the specified distance θ acquired in step S805. When a plurality of distances d are obtained in step S803, one of them (for example, the minimum distance) is compared in magnitude with the specified distance θ acquired in step S805. As a result of the size comparison, if d <θ, the process proceeds to step S807. If d ≧ θ, the process proceeds to step S809.

  In step S807, the CPU 201 reads out the human body information stored in the external memory 211 in the above step S521. In step S808, the CPU 201 generates the above human body model based on the human body information. For example, when the human body model shown in FIG. 7E is used as the human body model representing the basic posture of the user's human body, the position of the part indicated by the black circle is changed to the three-dimensional position of the motion marker 105 corresponding to the part. By moving it accordingly, it is transformed into a human body model representing the posture of the current user.

  On the other hand, in step S809, the CPU 201 sets the value of the end determination flag (initial value is OFF) to ON.

  When the process proceeds from step S808 to step S810, in step S810, the CPU 201 generates images of a virtual object and a human body model that can be seen from the viewpoint having the position and orientation of the objective camera 106, and the generated image is displayed on the objective camera 106. The mixed reality space image based on the objective viewpoint is generated by synthesizing it with the image of the real space imaged in the above, and the mixed reality space image based on the generated objective viewpoint is transmitted to the objective image display 160.

  On the other hand, when the process proceeds from step S809 to step S810, in step S810, the CPU 201 generates an image of a virtual object that can be seen from the viewpoint having the position and orientation of the objective camera 106, and the generated image is displayed by the objective camera 106. A composite reality space image based on the objective viewpoint is generated by combining with the captured real space image, and the mixed reality space image based on the generated objective viewpoint is transmitted to the objective image display 160.

  In step S <b> 811, the CPU 201 performs processing for saving the mixed reality space image and human body information in the external memory 211. Details of the processing in step S811 will be described with reference to FIG. 11 showing a flowchart of the processing.

  In step S1101, the CPU 201 determines whether or not the end determination flag is ON. As a result of this determination, if the end determination flag = ON, the process proceeds to step S1102, and if the end determination flag = OFF, the process proceeds to step S1103.

  The end determination flag = ON is a condition that the human body information is not stored in association with each other. For example, when the distance between the user and the target model is outside the specified distance, the human body information and the mixed reality space image are associated with each other. Do not save. This is to reduce processing for recording postures (human body information) unnecessary for verification when the posture of the user is verified later.

  In step S1103, the CPU 201 determines whether to store the mixed reality space image as a moving image or a still image. “Saving the mixed reality space image as a moving image” means storing the continuously generated mixed reality space image of each frame in one file. “Saving the mixed reality space image as a still image” means storing one frame of the mixed reality space image in one file. It is assumed that whether to store the mixed reality space image as a moving image or a still image is set in advance. If the mixed reality space image is stored as a moving image, the process proceeds to step S1104. If the mixed reality space image is stored as a still image, the process proceeds to step S1106.

  In step S1104, the CPU 201 stores the mixed reality space image in the external memory 211 as a moving image.

  In step S <b> 1105, the CPU 201 stores human body information in the external memory 211. Note that the mixed reality space image stored in step S1104 and the human body information stored in step S1105 are stored in association with each other. Instead of human body information, a human body model deformed based on the human body information may be stored, or an image of a mixed reality space in which a virtual object and a human body model are superimposed on an image captured by the objective camera 106 May be saved.

  In step S1106, the CPU 201 has continuously been in the state of d <θ in step S806 above for a predetermined time (for example, 3 seconds) or longer, and in step S902 below, the interference state with the determination target is continuously longer than the predetermined time. It is determined whether or not any of the following conditions has been satisfied in the following step S1003 in which the state having the line-of-sight vector and the intersection has continuously passed for a predetermined time or more. If any condition is satisfied as a result of the determination, the process proceeds to step S1007. If neither condition is satisfied, the process according to the flowchart of FIG. 11 is completed.

  In step S1107, the CPU 201 stores the mixed reality space image in the external memory 211 as a still image.

  In step S <b> 1108, the CPU 201 stores human body information in the external memory 211. Note that the mixed reality space image stored in step S1107 and the human body information stored in step S1108 are stored in association with each other. Instead of human body information, a human body model deformed based on the human body information may be stored, or an image of a mixed reality space in which a virtual object and a human body model are superimposed on an image captured by the objective camera 106 May be saved.

  On the other hand, in step S1102, if the mixed reality space image is stored as a moving image, the CPU 201 stores the file in which the mixed reality space image storage processing is performed in step S1104 in the external memory 211.

  Returning to FIG. 8, in step S812, the CPU 201 determines whether or not the end determination flag is ON. As a result of this determination, if the end determination flag = ON, the process proceeds to step S523. On the other hand, if the end determination flag = OFF, the process proceeds to step S801.

  Next, details of the processing in step S603 will be described with reference to FIG. 9 showing a flowchart of the processing.

  In step S901, the CPU 201 identifies a specific virtual object among the virtual objects arranged in the virtual space as an interference determination target (determination target model). For example, the CPU 201 specifies a virtual object whose human body control flag value is “ON” in the CG model information 1120 as the determination target model.

  In step S902, the CPU 201 determines whether there is interference between the user and the determination target model. Various methods can be applied to this determination. For example, the optical marker 103 is also attached to the user's hand, and the three-dimensional position of the optical marker 103 attached to the user's hand is obtained by the optical sensor 104. If the distance between the three-dimensional position of the optical marker 103 attached to the user's hand and the three-dimensional position of the determination target model is within a specified distance, the user's hand and the determination target model It can be determined that there was interference between the two. Further, the three-dimensional position of the hand is calculated from the captured image captured by the right-eye / left-eye video camera 221 of the HMD 101, and the distance between the calculated three-dimensional position and the three-dimensional position of the determination target model is defined. Within the distance, it can be determined that there is interference between the user's hand and the determination target model. Note that interference between another object and the determination target model may be determined instead of the user's hand. For example, the interference between the pointing tool held by the user with the hand and the determination target model may be determined. In this case, the optical marker 103 is attached to the indicator, and the three-dimensional position of the optical marker 103 attached to the indicator is obtained by the optical sensor 104. If the distance between the three-dimensional position of the optical marker 103 attached to the pointing tool and the three-dimensional position of the determination target model is within a specified distance, interference occurs between the pointing tool and the determination target model. It can be determined that there was. Further, it may be determined whether or not there is interference between the pointing tool as a virtual object that changes the position and orientation of the user's hand and the determination target model.

  If it is determined that there is interference as a result of the determination, the process proceeds to step S903. If it is determined that there is no interference, the process proceeds to step S905.

  In step S903, the CPU 201 reads out the human body information stored in the external memory 211 in the above step S521. In step S904, the CPU 201 generates the above human body model based on the human body information in the same manner as in the above step S808.

  On the other hand, in step S905, the CPU 201 sets the value of the end determination flag (initial value is OFF) to ON.

  When the process proceeds from step S904 to step S906, in step S906, the CPU 201 generates a virtual object image and a human body model image that can be seen from the viewpoint having the position and orientation of the objective camera 106, and the generated image is displayed on the objective camera 106. The mixed reality space image based on the objective viewpoint is generated by synthesizing it with the image of the real space imaged in the above, and the mixed reality space image based on the generated objective viewpoint is transmitted to the objective image display 160.

  On the other hand, when the processing proceeds from step S905 to step S906, in step S906, the CPU 201 generates an image of a virtual object that can be seen from the viewpoint having the position and orientation of the objective camera 106, and the generated image is displayed by the objective camera 106. A composite reality space image based on the objective viewpoint is generated by combining with the captured real space image, and the mixed reality space image based on the generated objective viewpoint is transmitted to the objective image display 160.

  When there is no interference between the user's hand and the CG model 130, the display screen of the objective image display 160 includes a user 1200 wearing the HMD 101 on the head, the CG model 130, as shown in FIG. , Is displayed. While interference occurs between the user's hand and the CG model 130, the objective image display 160 includes a user 1200 with the HMD 101 mounted on the head, the CG model 130, as shown in FIG. In addition, the human body model 1201 that matches the current posture of the user 1200 is displayed by deforming the human body model representing the basic posture of the user 1200 according to the human body information.

  In step S907, the CPU 201 performs processing according to the flowchart of FIG.

  In step S908, the CPU 201 determines whether or not the end determination flag is ON. As a result of this determination, if the end determination flag = ON, the process proceeds to step S523. On the other hand, if the end determination flag = OFF, the process proceeds to step S902.

  Next, details of the process in step S604 will be described with reference to FIG. 10 showing a flowchart of the process.

  In step S1001, the CPU 201 specifies a specific virtual object among the virtual objects arranged in the virtual space as a determination target (determination target model). For example, the CPU 201 specifies a virtual object whose human body control flag value is “ON” in the CG model information 1120 as the determination target model.

  In step S <b> 1002, the CPU 201 obtains a vector representing the orientation of the viewpoint (which can be either the right viewpoint or the left viewpoint) as a gaze direction vector. When the HMD 101 includes a line-of-sight sensor, a vector representing the line-of-sight direction specified by line-of-sight detection (eye tracking) by the line-of-sight sensor may be acquired as the line-of-sight direction vector.

  In step S <b> 1003, the CPU 201 determines whether or not an intersection exists between the straight line having the line-of-sight direction vector as the direction vector and the determination target model. As a result of this determination, if there is an intersection, the process proceeds to step S1004. If there is no intersection, the process proceeds to step S1006.

  In step S1004, the CPU 201 reads out the human body information stored in the external memory 211 in the above step S521. In step S1005, the CPU 201 generates the above human body model based on the human body information in the same manner as in the above step S808.

  On the other hand, in step S1006, the CPU 201 sets the value of the end determination flag (initial value is OFF) to ON.

  When the process proceeds from step S1005 to step S1007, in step S1007, the CPU 201 generates a virtual object image and a human body model image that can be seen from the viewpoint having the position and orientation of the objective camera 106, and the generated image is displayed on the objective camera 106. The mixed reality space image based on the objective viewpoint is generated by synthesizing it with the image of the real space imaged in the above, and the mixed reality space image based on the generated objective viewpoint is transmitted to the objective image display 160.

  On the other hand, when the process proceeds from step S1006 to step S1007, in step S1007, the CPU 201 generates an image of a virtual object that can be seen from the viewpoint having the position and orientation of the objective camera 106, and the generated image is displayed by the objective camera 106. A composite reality space image based on the objective viewpoint is generated by combining with the captured real space image, and the mixed reality space image based on the generated objective viewpoint is transmitted to the objective image display 160.

  In step S1008, the CPU 201 performs processing according to the flowchart of FIG.

  In step S1009, the CPU 201 determines whether or not the end determination flag is ON. As a result of this determination, if the end determination flag = ON, the process proceeds to step S523. On the other hand, if the end determination flag = OFF, the process proceeds to step S1002.

  Returning to FIG. 5, in step S523, the CPU 201 then mixes a mixed reality space image (right-eye mixed reality space image, left-eye mixed reality space image) based on the user viewpoint generated in step S508, and The mixed reality space image based on the objective viewpoint generated in any one of step S810, step S906, and step S1007 is stored in the external memory 211 as storage information in association with each other. Further, when a human body model is superimposed on an image of the mixed reality space based on the objective viewpoint, the human body information used when generating the human body model (the human body information acquired in any one of steps S807, S903, and S1004) ) Are also stored in the external memory 211 as storage information in association with each other. A configuration example of the stored information is shown in FIG.

  A unique ID (registration ID) for the mixed reality space image based on the user viewpoint for one frame is registered in the column 1141 in the saved information 1140 shown in FIG. 7D, and one frame is stored in the column 1142. The date and time when the mixed reality space image based on the user viewpoint is stored is registered, the human body information is registered in the column 1143, and the mixed reality space image file based on the user viewpoint is registered in the column 1144. The name (including the file path) and the file name (including the file path) of the mixed reality space image based on the objective viewpoint are registered. Instead of the human body information, a human body model deformed based on the human body information may be registered in the field 1143.

  Further, it is desirable to store each record of the storage information 1140 together with the current mode setting 1130 and the model ID (column 1121) of the virtual object to be stored. This is because it is possible to specify at what timing the model has been saved at the time of later verification.

  Note that the storage in step S523 is to determine the storage information stored by the storage processing in FIG. 11, and various information is stored in the external memory 211. In this embodiment, the process stores the information stored in FIG. 11 in step S523, but the process in step S523 may be performed in FIG. That is, step S523 can be omitted, and FIG. 11 and step S523 are processes without inconsistencies.

  In step S524, the CPU 201 displays the stored information stored in the external memory 211 on a display device such as the display 210. The operator of the PC 100 can specify any line included in the displayed stored information by operating the input device 209 or the like, and the CPU 201 can specify the human body information corresponding to the line specified by the operator, the user viewpoint, and so on. Are read from the external memory 211 and displayed on the display 210. In addition, the display form of these information is not restricted to a specific display form, It is not restricted to displaying all these information. Further, in addition to / instead of these pieces of information, other information may be displayed.

  A display example of the stored information is shown in FIG. The display example of FIG. 14A shows an example in which the human body model 1402 generated based on the human body information is superimposed on the upper right of the mixed reality space image based on the user viewpoint positioned below the CG model 130 and displayed. ing. The mixed reality space image and the human body model are not limited to being displayed in a superimposed manner, and may be displayed in separate display areas. The display example of FIG. 14B shows an example in which a mixed reality space image 1450 based on an objective viewpoint is superimposed and displayed on the upper right of the mixed reality space image based on the user viewpoint positioned below the CG model 130. ing. Note that the present invention is not limited to superimposing and displaying the mixed reality space image based on the objective viewpoint on the mixed reality space image based on the user viewpoint, and may be displayed in separate display areas.

  When verifying the posture of the user, an image in which the posture of the user is objectively viewed (an image of a mixed reality space based on an objective viewpoint) and an image that the user is viewing at that time (an image of a mixed reality space based on the user viewpoint) ) Is desirable.

  Generating a human body model based on human body information and generating a human body model image viewed from the viewpoint for each frame places a large processing burden on the PC 100. According to the present embodiment, when the above condition is satisfied, the human body model is generated based on the human body information and the human body model image viewed from the viewpoint is generated, so that the processing burden on the PC 100 can be reduced. .

  In addition, according to the present embodiment, it is possible to easily record various information related to the experience of the mixed reality space by the user. Such recording is effective, for example, when verifying the posture of the user (whether it is an unreasonable posture) using human body information.

  Finally, a functional configuration example of the PC 100 will be described with reference to the block diagram of FIG. Each functional unit in FIG. 3 may be implemented by hardware or software (computer program). In the former case, for example, by installing such hardware in the HMD 101, the HMD 101 can also execute the processes described above as being performed by the PC 100. In the latter case, such a computer program is stored in the external memory 211, and the CPU 201 reads out this computer program to the RAM 203 and executes it appropriately, thereby realizing the function of the corresponding functional unit.

  The acquisition unit 301 acquires various data input to the PC 100. The user viewpoint image generation unit 302 generates an image of the mixed reality space based on the user viewpoint based on the data acquired by the acquisition unit 301 and transmits the generated image to the HMD 101. The determination unit 303 determines whether to generate the above human body model based on the human body information. The objective viewpoint image generation unit 304 generates an image of the mixed reality space based on the objective viewpoint and sends it to the objective image display 160. The storage unit 305 performs processing for storing various types of information in the external memory 211.

  That is, the image processing apparatus according to the present embodiment determines whether to generate a human body model representing the posture of the user's human body from the human body information obtained by measuring the posture of the user's human body. When it is determined based on the line-of-sight direction and it is determined that a human body model is generated from the human body information, a human body model is generated from the human body information, and an image including the human body model is generated and output to the display device.

[Second Embodiment]
In the first embodiment, the HMD 101 and the PC 100 are separate devices. However, the HMD 101 and the PC 100 are integrated, and the HMD 101 is described as being performed by the PC 100 in addition to the above-described processes performed by the own device. The above processes may also be performed.

  In the first embodiment, the HMD 101 is described as being a video see-through head-mounted display device. However, an optical see-through head-mounted display device may be used as the HMD 101.

  In addition to or instead of the HMD 101 mounted on the user's head, an image of the mixed reality space based on the user viewpoint is output to another display device, for example, a mobile terminal device such as a tablet terminal device or a smartphone. Anyway. In this case, the mobile terminal device can acquire the position and orientation of the mobile terminal device as a user viewpoint by using a sensor or application provided in the mobile terminal device. The image of the mixed reality space based on the user viewpoint may be generated by the PC 100 or may be generated on the mobile terminal device side.

  In the first embodiment, the mixed reality presentation system (MR system) that presents the user with a mixed reality space obtained by synthesizing the real space and the virtual space has been described. However, only the virtual space is presented to the user. It may be applied to a so-called VR system. In that case, processing related to imaging of the real space and synthesis with the image of the real space is not necessary.

  Instead of the objective camera 106, the right-eye / left-eye video camera 221 provided in the HMD 101 mounted on the head of another user may be used.

  Further, in the first embodiment, for simplicity of explanation, it is assumed that one HMD 101 is connected to one PC 100. However, two or more HMDs 101 are connected to one PC 100. May be.

[Third Embodiment]
In the first embodiment, the state of d <θ has continuously passed for a predetermined time (for example, 3 seconds) in step S806, and the interference state with the determination target has continuously passed for a predetermined time in step S902. The image of the mixed reality space is stored as a still image and the human body information is stored when any of the conditions in which the state having the line-of-sight vector and the intersection in S1003 has continuously passed for a predetermined time is satisfied . However, the conditions for storing the mixed reality space image as a still image and the human body information are not limited to these conditions. For example, if any of the above conditions is satisfied, d <θ in Step S806, interference with the determination target in Step S902, or an intersection with the line-of-sight vector in Step S1003, is satisfied. The physical space image may be stored as a still image and the human body information may be stored. Further, as in the first embodiment, in addition to or instead of the human body information, other information such as an image of a mixed reality space based on a human body model or an objective viewpoint may be stored.

  In the first embodiment, when saving an image of the mixed reality space as a moving image, no conditions for saving are provided. However, as in the case of saving as a still image, various conditions are provided. When the provided condition is satisfied, the mixed reality space image may be stored as a moving image and the human body information may be stored. Also, when two or more conditions are satisfied, an image of the mixed reality space (which may be a moving image or a still image) and human body information may be stored. In the first embodiment, various information storage steps are provided, but some storage steps may be omitted depending on the situation.

[Other Embodiments]
Although the embodiment of the present invention has been described above, the present invention can take an embodiment as a system, apparatus, method, program, recording medium, or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The program according to the present invention is a program that allows a computer to execute the processing method of the flowchart, and the storage medium according to the present invention stores a program that allows the computer to execute the processing method.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by executing reading.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk, solid state drive, or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a communication line using a communication program, the system or apparatus can enjoy the effects of the present invention.

  In addition, a part or all of each embodiment described above may be used in combination as appropriate, or a part or all of each embodiment described above may be selectively used.

Claims (9)

Determination means for determining whether or not a condition for generating a human body model representing the posture of the user's human body is satisfied from human body information obtained by measuring the posture of the user's human body based on the position or line-of-sight direction of the user When,
Output means for generating the human body model from the human body information, generating an image including the human body model, and outputting the generated image to a display device when the determination unit determines that the condition is satisfied. Image processing device.   The determination means determines that the human body model is generated from the human body information when a condition that the position of the user is within a predetermined distance from a predetermined position or a position of a specific virtual object is satisfied. The image processing apparatus according to claim 1.   The said determination means determines that the said human body model is produced | generated from the said human body information, when the conditions that the said user is interfering with a specific virtual object are satisfy | filled. Image processing device.   The said determination means determines that the said human body model is produced | generated from the said human body information, when the conditions that the specific virtual object exists in the said user's gaze direction are satisfy | filled. Image processing device.   The image processing apparatus according to claim 2, wherein the specific virtual object is a virtual object designated in advance among virtual objects arranged in a virtual space.   When the determination unit determines that the human body model is to be generated, the output unit generates the human body model from the human body information, and determines the human body model according to the position and orientation of the imaging device arranged in the real space. 6. An image is generated, and the generated image of the human body model is synthesized with an image of a real space imaged by the imaging device and output to the display device. An image processing apparatus according to 1.   The said human body information is the three-dimensional position of this marker obtained by measuring the marker attached to the said user with a motion capture system, The any one of Claim 1 thru | or 6 characterized by the above-mentioned. Image processing device. An image processing method performed by an image processing apparatus,
The determination means of the image processing apparatus generates a human body model that represents the posture of the user's human body from human body information obtained by measuring the posture of the user's human body based on the user's position or line-of-sight direction. A judgment process for judging whether or not to satisfy,
When the output unit of the image processing apparatus determines that the condition is satisfied in the determination step, the output unit generates the human body model from the human body information, generates an image including the human body model, and outputs the generated image to the display device An image processing method comprising the steps of:   The computer program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7.

Images