JP6741939B2 - Information processing apparatus, information processing system, control method thereof, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing system, a control method thereof, and a program.

In recent years, mixed reality (hereinafter referred to as MR) technology has become widespread. Using the mixed reality technology, a user who wears a head mounted display (HMD) is provided with a mixed reality image in which a CG model (virtual object) is arranged in the real image, and a mixed reality world combining reality and virtual is provided. Can be experienced. In generating the mixed reality image, a method of identifying the position of the HMD and the position of the CG model using a sensor or a two-dimensional marker is used.

Japanese Patent Application Laid-Open No. 2004-242242 describes a technique of specifying the position and orientation of the HMD by using a two-dimensional marker appearing in the image captured by the HMD and providing a mixed reality image. Further, Patent Document 2 describes a technique of identifying the position and orientation of the HMD using a magnetic sensor and providing a mixed reality image.

Further, Patent Document 3 describes a technique of, when virtual objects interfere with each other, deforming the shape of the virtual objects that interfere with each other according to the interference and notifying the user of the fact of the interference.

JP, 2003-308514, A JP, 2003-240532, A JP, 2007-179483, A

For example, in the technique of the cited document 3, when both of the interfering objects are displayed so that they can be seen by the user, interference/collision between the objects can be easily confirmed. , When multiple virtual objects are arranged in the virtual object, when the interfering part is on the back side of the virtual object as seen from the user, which virtual object is in contact with which virtual object Is hard to understand.

Therefore, it is an object of the present invention to provide a mechanism that allows a user to easily confirm interference between objects in mixed reality.

The present invention is an information processing apparatus that stores a plurality of positions and orientations of virtual objects, which is connectable to a display device that includes a display unit that displays virtual objects, and that identifies the position and orientation of the display device. Means, generating means for generating an image of the virtual object based on the position and orientation of the display device specified by the display device position and orientation specifying means, and the position and orientation of the virtual object, and the generating means for generating the image of the virtual object. And a transmitting unit configured to transmit an image to be displayed on the display device; and an interference object specifying unit that specifies a virtual object that interferes with another virtual object based on the position and orientation of the virtual object. Means, other than the interference object based on the position and orientation of the display device specified by the display device position and orientation specifying unit, and the position and orientation of the interference object specified by the interference object specifying unit An image of an object is generated so that the interference object can be confirmed on the display device, and the transmission means is an image of another virtual object except the interference object, the interference object being generated by the generation means. Is transmitted to the display device so that an image generated so that can be confirmed on the display device is transmitted.

According to the present invention, it is possible to provide a mechanism that allows a user to easily confirm interference between objects in mixed reality.

It is a figure which shows an example of an information processing system block diagram in embodiment of this invention. It is a figure which shows an example of the hardware constitutions of various apparatuses in embodiment of this invention. It is a figure showing an example of functional composition of various devices in an embodiment of the present invention. FIG. 6 is a processing diagram showing an example of a module configuration of various devices according to the embodiment of the present invention. 6 is a flowchart illustrating an example of mixed reality image generation and display processing according to the first embodiment of the present invention. It is a figure which shows an example of a data structure in the 1st Embodiment of this invention. 6 is a flowchart showing details of mixed reality image generation processing in the first embodiment of the present invention. 9 is a flowchart showing details of mixed reality image generation processing according to the second embodiment of the present invention. 9 is a flowchart showing details of mixed reality image generation processing in the third embodiment of the present invention. It is a figure which shows an example of a data structure in embodiment of this invention. It is a figure which shows an example of a 3D model in embodiment of this invention. It is a figure which shows an example of a display of the mixed reality image in the 1st Embodiment of this invention. It is a figure which shows an example of a display of the mixed reality image in the 2nd Embodiment of this invention. It is a figure which shows an example of a display of the mixed reality image in the 3rd Embodiment of this invention. It is a figure which shows an example of a display of the mixed reality image in embodiment of this invention.

An example of the configuration of the information processing system according to the embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 1, various devices of the information processing system according to the present invention are communicatively connected via a network 150 (each device is in a connectable state). For example, the PC 100 is communicably connected to the HMD 101 (a generic term for HMD 101A to HMD 101C).

The PC 100 stores a three-dimensional model (CG model/virtual object) to be superimposed on the real image captured by the HMD 101.

Further, the PC 100 acquires a real image from the HMD 101 (HMD 101A to 101C in FIG. 1) managed by the PC 100 and stores the real image in the storage unit. In addition, the PC 100 identifies and stores the position and orientation of the HMD 101. The position/orientation specifying method of the HMD 101 can be specified by using the two-dimensional marker in the real image captured by the HMD 101, which is described in Patent Document 1.

Further, the sensor (for example, the optical sensor 104 (infrared sensor) in FIG. 1) described in Patent Document 2 detects the position and orientation of the optical marker 103 (infrared marker) installed on the HMD 101 as the position and orientation of the HMD 101. It can be specified by the PC 100 acquiring it. For example, three optical markers 103 are installed on the HMD 101 as shown in FIG. The positions of these three optical markers 103 are specified by the optical sensor 104, and the PC 100 stores these three optical markers in which position of the HMD 101 (in which position when viewed from the position of the HMD 101) stored in advance in the PC 100 itself. The position and orientation of the HMD is specified using information indicating whether the 103 is installed and the positions of the three optical markers 103 specified by the optical sensor 104.

The PC 100 uses the position and orientation of the HMD 101 and the information of the three-dimensional model and the position and orientation of the three-dimensional model stored in the storage unit to generate a mixed reality image in which the three-dimensional model is superimposed on the real image. Then, the mixed reality image is transmitted to the HMD 101 so as to be displayed on the display of the HMD 101. The HMD 101 displays the received mixed reality image on the display. The above is the description of FIG. 1.

Next, with reference to FIG. 2, an example of a hardware configuration of various devices in the embodiment of the present invention will be described.

The CPU 201 centrally controls each device and controller connected to the system bus 204.

Further, the ROM 202 stores various programs necessary for realizing the functions executed by various devices such as a BIOS (Basic Input/Output System) which is a control program of the CPU 201, an operating system (OS), and the like.

The RAM 203 functions as a main memory, a work area, etc. of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing the processing into the RAM 203 and executing the program.

Various programs and the like used by the PC 100 of the present invention to execute various processes described later are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as necessary. Further, definition files and various information tables used by the program according to the present invention are stored in the external memory 211.

The input controller (input C) 205 controls input from a pointing device (input device 209) such as a keyboard and a mouse.

The video controller (VC) 206 controls display on a display device such as the right eye/left eye display 222 provided in the HMD 101. To the right-eye/left-eye display 222, for example, an external output terminal (for example, Digital Visual Interface) is used for output. The right/left eye display 222 includes a display for the right eye and a display for the left eye. Further, the input controller (input C) 205 controls display on the display device of the display 210 (CRT display or the like) included in the PC 100. In addition, in FIG. 2, the display is not limited to the CRT display, and may be another display such as a liquid crystal display.

The memory controller (MC) 207 is an adapter for a hard disk (HD) or a flexible disk (FD) or a PCMCIA card slot that stores a boot program, browser software, various applications, font data, user files, edit files, various data, and the like. It controls access to the external memory 211 such as a card type memory connected via.

A communication I/F controller (communication I/FC) 208 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, Internet communication or the like using TCP/IP is possible. The communication I/F controller 208 also controls communication with the optical sensor 104 via Gigabit Ethernet (registered trademark) or the like.

The general-purpose bus 212 is used to capture an image from the right-eye/left-eye video camera 221 of the HMD 101. Input from the right-eye/left-eye video camera 221 is performed using an external input terminal (for example, an IEEE 1394 terminal). The right/left-eye video camera 221 includes a right-eye video camera and a left-eye video camera.

The CPU 201 enables display on the display by executing the outline font rasterization process in the display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction with a mouse cursor or the like (not shown) on the display. The above is the description of FIG.

Next, with reference to FIG. 3, an example of a functional configuration of various devices in the embodiment of the present invention will be described.

The real image transmission unit 301 is a transmission unit that transmits a real image captured by using the camera device included in the HMD 101 to the PC 100.

The real image reception/storage unit 311 is a storage unit that receives and stores a real image from the HMD 101. An example of a real image stored in the real image receiving/storing unit 311 is shown at 630 in FIG. The HMD position/orientation identifying unit 312 identifies and stores the position/orientation of the HMD 101.

The 3D model storage unit 313 is a storage unit that stores a 3D model that is a virtual object (virtual object) in association with the position and orientation of the 3D model. An example of 3D model information stored in the 3D model storage unit 313 is indicated by 620 in FIG.

The interference model identification unit 314 is a 3D model (also referred to as an interference model/interference object) that interferes (contacts/collides) with another virtual model based on the shapes and the positions and orientations of the plurality of 3D models stored in the 3D model storage unit 313. ) Is a specific part for specifying. The interference point identification unit 315 identifies an interference point that is a point on the outermost surface of the 3D model that is undergoing interference, and that is in interference/contact with the outermost surface of another 3D model.

The transparentization target model identification unit 316 identifies a 3D model to be transparentized, for example, in order to make the model around the model in the interference transparent so that the model in the interference can be seen well by the user wearing the HMD 101. The transparentized image generation control unit 317 is a generation unit that generates a transparentized image of the 3D model specified by the transparentization target model specifying unit 316. For example, a transparent image (drawing data 640 in FIG. 6) that is a 3D model image viewed from the position and orientation of the HMD 101, which is used for superimposition with a real image when generating mixed reality, is generated. Further, a mixed reality image is generated by superimposing the transparent image on the real image acquired from the HMD 101. The transparency at the time of making it transparent can be arbitrarily set on a setting screen (not shown).

The transparent image transmission unit 318 is a transmission unit that transmits a transparent image (eg, a mixed reality image in which a 3D model around a model in interference is transparent) to the HMD 101.

The transparent image receiving unit 302 receives the transparent image, and the transparent image display unit 303 displays the received transparent image. The above is the description of FIG.

In the present embodiment, the PC 100 includes the processing units (functions) 311 to 318, but the HMD 101 itself may include these components, for example.

Next, with reference to FIG. 4, an example of a module configuration of various devices in the embodiment of the present invention will be described.

The PC 100 includes an operating system 401 (OS), a graphic engine 402, a mixed reality platform 403 (also called MR platform), and a mixed reality application 404 (also called MR application or viewer application), and is controlled by the CPU 201. ..

The operating system 401 controls the input/output of the HMD 101 and transfers the real image obtained from the camera 221 via the input interface to the mixed reality platform 403. The mixed reality image drawn by the graphic engine 402 is output to the display 222 via the output interface.

The graphic engine 402 generates an image to be drawn from the three-dimensional model stored in the external memory 211, superimposes it on the real image, and synthesizes it. The engine used for drawing may be, for example, a widely used graphic engine such as OpenGL or DirectX, or an independently developed graphic engine. In this embodiment, OpenGL is used as the graphic library.

The mixed reality platform 403 identifies the position and orientation of the HMD 101 by acquiring the position of the optical marker 103 from the optical sensor 104, and aligns the real space and the virtual space. It should be noted that the position/orientation and position alignment techniques can be realized by using known techniques, such as JP-A-2002-32784, JP-A-2006-072903, and JP-A-2007-166427.

It is also possible to provide the HMD 101 with a position sensor without using the two-dimensional marker, and specify the position and orientation of the HMD 101 based on the position measured by triangulation using this position sensor. Is.

The mixed reality application 404 receives the position and orientation of the HMD 101 from the mixed reality platform 403, information on the shape of the three-dimensional model, and information on the position and orientation, and issues a drawing command of the three-dimensional model to the graphic engine 402. At this time, a command in which the identification information of the three-dimensional model to be drawn and the position and orientation information are set is issued using the OpenGL API. The above is the description of FIG.

<First Embodiment>
Next, with reference to FIG. 5, a mixed reality image generation and display process according to the first embodiment of the present invention will be described.

The CPU 201 of the PC 100 receives a start operation of the mixed reality application 404. The processing after step S504 in FIG. 5 is started and executed by the mixed reality application 404, the mixed reality platform 403, the graphic engine 402, and the OS 401 when the mixed reality application 404 is activated. In addition, when the mixed reality application 404 is terminated and the mixed reality application 404 is terminated, the operation is terminated.

After being activated in step S501, the HMD 101 starts capturing a real image using the function of the camera 221 (step S502). Then, the physical image acquired by the imaging process is transmitted to the PC 100 (step S503).

The CPU 201 of the PC 100 receives the real image from the HMD 101 (step S504/corresponds to the real image acquisition means) and stores the received real image in the external memory 211 (step S505). For example, as shown in the real image table 630 of FIG. 6, the HMD ID 631 that is the identification information of the HMD 101 that is the transmission source of the real image and the real image 632 are stored in association with each other.

The CPU 201 of the PC 100 acquires the position and orientation of the HMD 101 (step S506) and stores it in the external memory 211 (step S507). For example, as shown in the HMD information 610 of FIG. 6, it is represented by an HMD ID 611 that is identification information of the HMD 101, a position 612 (X, Y, Z coordinates) of the HMD, and a posture 613 (X, Y, Z coordinate values). The direction in which the HMD 101 is turned on) is stored.

The CPU 201 of the PC 100 acquires the 3D model information from the external memory 211, superimposes it on the real image received from the HMD 101 to generate a mixed reality image (step S508), and transmits it to the HMD 101 (step S509). Details of step S508 will be described later in the description of FIG. 7.

The 3D model information is, for example, the information shown in the model information 620 in FIG. The model information 620 is information stored in the external memory 211 of the PC 100 in advance. The model ID 621 is identification information of the 3D model. The model name 622 is the file name of the 3D model. File path 623 indicates the location where the file is stored. A position 624 and a posture 625 indicate the position and posture of the 3D model.

When the camera having the same angle of view as the HMD 101 captures the 3D model at the position 624/attitude 625 from the position 612/attitude 613 of the HMD, the CPU 201 of the PC 100 generates an image of the 3D model as the drawing data 640. .. Then, the drawing data is combined with the real image to generate a mixed reality image (MR image) shown in the MR image table 650 of FIG. Also, the MR image table 650 is stored in association with the HMD (HMD ID 651) that is the transmission source of the real image used to generate the mixed reality image. After that, the mixed reality image 652 is transmitted to the HMD 101 indicated by the HMD ID 651.

The HMD 101 receives the mixed reality image from the PC 100 (step S510) and displays it on the display screen (step S511). The above is the description of FIG.

Next, with reference to FIG. 7, details of the mixed reality image generation processing according to the embodiment of the present invention will be described.

In the first embodiment, by making another 3D model around the 3D model (interference model) which is in interference with the other 3D model transparent, the interference model can be easily seen by the user.

The CPU 201 of the PC 100 reads the model information 620 (FIG. 6) stored in the external memory of the PC 100 onto the memory, and superimposes it on the 3D model included in the angle of view of the HMD 101, that is, the real image acquired from the HMD 101. Then, the model information of the 3D model to be displayed is specified and held as a list (interference information 1010) in the memory (step S701). For example, the table of the interference information 1010 in FIG. 10 is generated, and the ID of the specified model is stored in the model ID 1011. The values of the interference destination model 1012 and the interference point 1013 are deleted.

The CPU 201 of the PC 100 refers to the value of the interference display mode 1050 of FIG. 10 stored in the external memory of the own device, and determines whether the interference display mode is set (step S702). The interference display mode is a mode in which the display of other 3D models excluding the interference model is changed (for example, transparent) so that the interference model can be easily seen (corresponding to mode setting means). When the value of the interference display mode 1050 is ON, it is determined that the interference display mode is set. When the value of the interference display mode 1050 is OFF, it is determined that the interference display mode is not set.

When the interference display mode is set (YES in step S702), the CPU 201 of the PC 100 executes the processes of steps S703 to S705 for all 3D models in the list. If the interference display mode is set (NO in step S702), the process proceeds to step S709 described later. When the process proceeds from step S702 to step S709, in step S709, the CPU 201 of the PC 100 renders 3D model drawing data with 0% transparency to display all 3D models within the angle of view of the HMD 101 with 0% transparency. Generate and store in memory. Then, in step S710, the drawing data and the real image are combined to generate a mixed reality image. For example, an example of a mixed reality image in which the reality image 1103 and the model 1101 are combined is shown at 1200 in FIG. In reality, as shown at 1100 in FIG. 11, it is assumed that another model 1102 is located in the model 1101 displayed at 1200.

The CPU 201 of the PC 100 acquires one unprocessed model information from the list generated and stored in step S702 (step S703) and determines whether the 3D model indicated by the model information interferes with another model (step S703). S704/corresponding to the interference object specifying means). The interference between the 3D models is specified and determined using the 3D shape of the model that is specified by reading the model with the model name 622 stored in the file path 623 and the model position 624 and the attitude 625 (position/orientation).

As a case where the 3D models interfere with each other, for example, a case where the position 624 of the 3D models is changed by a user operation is assumed. Further, as shown in 1110 and 1120 of FIG. 11, the shape of the model 1111 is updated/changed to 1121 (because the pipe-shaped three-dimensional shape 1122 is added to the model 1111 in FIG. 11). It is conceivable that the model 1112, which had not been in contact with or interfered with the model 1112 before the change, would come to interfere.

If it is determined that the model ID is interfering with another model, the model ID of the interfering other model is acquired, and is acquired in step S703 to determine that the model ID is interfering with another model 3D. It is stored in association with the model ID of the model. For example, the model ID of the other model is inserted and stored in the interference destination model 1012 of the interference information 1010. In addition, a point on the outermost surface of each interference model, where the interference models are in contact with each other, is specified as an interference point (position where 3D models interfere with each other), and is stored in the interference point 1013. (Step S705).

After performing the processes of steps S703 to S705 for all the 3D models indicated by the model ID 1011 of the interference information 1010, the process proceeds to step S706.

The CPU 201 of the PC 100 changes the color of the interference model (3D model that is in interference with another model) to a color indicating that interference is occurring (a color that can be identified as being in interference). The color indicating the interference is, for example, red. Specifically, based on the angle of view and the position and orientation of the HMD 101 and the position and orientation of the interference model, the image data of the red interference model when the interference model is imaged from the position and orientation of the HMD 101 is drawn in FIG. The data 640 is generated and stored in the memory (step S706). The color change of the interference model and the generation of the drawing data are executed for all the interference models, and the process proceeds to step S707.

The CPU 201 of the PC 100 specifies and stores a 3D model other than the interference model, which is located within a predetermined range from the interference point of each interference model (step S707). For example, as shown in the in-range model 1030 of FIG. 10, a list of model IDs is generated and stored in the memory. The predetermined range here is specified from the radius stored in advance in an arbitrary range 1020 (FIG. 10) on the external memory of the PC 100.

According to FIG. 10, a range (area) having a radius of 20 cm from the interference point is specified as the predetermined range. The arbitrary range 1020 can be arbitrarily set/changed by receiving a user operation via a setting screen (not shown).

The CPU 201 of the PC 100 acquires a 3D model located within a predetermined range from the interference point from the external memory and changes the color of the 3D model to a transparent color. That is, the 3D model around the interference model is made transparent (step S708). For example, as indicated by 1213 in 1210 of FIG. 12, the 3D model within the distance 1220 (arbitrary range) from the interference point is made transparent. That is, by making the 3D model around the interference point 1221 transparent, it is possible to generate a mixed reality image in which the models 1111 and 1112, which are interference models, are easily visible to the user, and display the mixed reality image on the display screen. Become.

The degree of transparency can be arbitrarily set and changed. For example, as indicated by transparency 1040 in FIG. 10, information indicating how transparent the 3D model is made to be transparent is stored in the external memory in step S708. When a value is inserted in the transparency 1040, the model located within the predetermined range from the interference point is changed to the transparency having the value indicated by the transparency 1040 in step S708.

Specifically, based on the angle of view and position/orientation of the HMD 101 and the position/orientation of the 3D model of the in-range model 1030, a transparent 3D model when a 3D model within the predetermined range is imaged from the position/orientation of the HMD 101. Image data is generated and stored in the memory as drawing data 640 shown in FIG. With respect to all 3D models (all models stored in the in-range model 1030) other than the interference model, which are located within a predetermined range from the interference point, the transparency and the generation of the drawing data are executed, and the processing is performed. The process moves to step S709.

The CPU 201 of the PC 100 specifies a normal 3D model that is a 3D model to be displayed on the screen of the HMD 101 and is not an interference model or a 3D model stored in the in-range model 1030. Specifically, in the interference information 1010, a 3D model in which the model ID is not inserted in the interference destination model, and a 3D model with a model ID not included in the in-range model 1030 is specified as a normal model. To do. The CPU 201 of the PC 100, based on the view angle/position/posture of the HMD 101 and the identified position/posture of the normal model, of the normal model when the normal model is imaged at the view angle/position/posture of the HMD 101. An image (drawing data 640) is generated with transparency=0% and stored in the memory (step S709).

The CPU 201 of the PC 100 superimposes the drawing data of the interference model, the drawing data of the in-range model, and the drawing data of the normal model on the real image stored in the memory obtained from the HMD 101 to form an MR image (mixed reality image). It is generated (step S710). The above is the description of FIG. 7.

As described above, according to the first embodiment of the present invention, it is possible to provide a mechanism that allows the user to easily confirm the interference between objects in mixed reality.

For example, by making another 3D model around the 3D model (interference model) that is in interference with another 3D model transparent, an image that makes the interference model easy to see is generated, and the user can confirm it on the display screen. You can

<Second Embodiment>
Next, with reference to FIG. 8, details of the mixed reality image generation processing according to the second embodiment of the present invention will be described.

In the second embodiment, the interference model is made visible to the user by making the 3D model (interference model) in interference with the other 3D model and the other 3D model between the HMD transparent.

For example, by making another 3D model between the interference point and the HMD transparent, the interference model can be easily seen by the user.

The process of FIG. 8 is executed by the CPU 201 of the PC 100 immediately after the process of step S706 of FIG. 7 in the first embodiment. Note that the description of the processing common to the first embodiment will be omitted.

The CPU 201 of the PC 100, in order to make the interference model easily visible, a mode in which another 3D model within an arbitrary range (within a predetermined distance) from the interference point is made transparent, and between the interference point and the HMD 101. Which of the modes for making the 3D model transparent is set is determined (step S801). It is assumed that the mode is preset to the transparent mode 1070 of FIG. 10 stored in the external memory of the PC 100.

When the mode for making another 3D model within an arbitrary range from the interference point transparent is set, the process proceeds to step S707 in FIG. 7. When the mode for making another 3D model located between the interference point and the HMD 101 transparent is set, the process proceeds to step S802.

The CPU 201 of the PC 100 executes the processes of steps S803 to S805 for all the interference points.

The CPU 201 of the PC 100 acquires one unprocessed interference point 1013 from the interference information 1010 (step S803), and is located between the position of the HMD 101 and the position of the interference point indicated by the acquired interference point 1013, other than the interference model. The 3D model of is specified as a model to be made transparent and stored in the memory (step S804). For example, as shown by 1311 in FIG. 13, a straight line from the position of the HMD 101 to the interference point is generated. Then, when it is arranged at the position 624 and the posture 625, it is a 3D model that has an intersection 1312 with the straight line (crosses the straight line), and all 3D models other than the interference model are specified, and the model ID of the specified 3D model is specified. Are stored in the memory in the format shown in the HMD-interference point model 1060 of FIG. In FIG. 13, the hood 3D model 1301 is specified as the 3D model between the HMD 101 and the interference point.

The CPU 201 of the PC 100 makes all other 3D models (all models in the HMD-interference point model 1060) between the acquired interference point and the HMD 101, except the interference model, transparent. Specifically, an image (drawing data) in which the other 3D model is made transparent with a value of transparency 1040 is generated and stored in the memory. The drawing data is an image of the 3D model when the 3D model of the position 624/posture 625 is photographed from the position/posture of the HMD 101 at the angle of view of the HMD 101. Then, in step S710, the drawing data is combined with the real image to generate a mixed reality image. For example, as shown at 1310 in FIG. 13, the hood 3D model 1301 that has not been transparentized at 1300 is specified as a model between the HMD 101 and the interference point and is made transparent to generate a mixed reality image. In 1310, the transparent model 1301 is indicated by a dotted line.

The CPU 201 of the PC 100 executes the processes of steps S803 to S805 for all the interference points 1013 and all other 3D models between the interference points and the HMD 101, and then shifts the process to step S709 of FIG. 7. To do. The above is the description of FIG. 8.

As described above, according to the second embodiment of the present invention, it is possible to provide a mechanism that allows the user to easily confirm the interference between objects in mixed reality.

For example, by making the 3D model (interference model) that is in interference with another 3D model and the other 3D model between the HMDs transparent, it is possible to make the interference model easily visible to the user.

For example, by making another 3D model between the interference point and the HMD transparent, it is possible to make the interference model visible to the user.

In the above description of the second embodiment, the 3D model other than the interference model between the interference point and the HMD is made transparent. However, for example, between the interference model and the HMD, The 3D model other than the interference model, which is located, may be made transparent. Specifically, the processes of steps S803 to S805 of FIG. 8 are executed for all the interference models. In step S803, one unprocessed interference model is acquired from the interference information 1010. In step S804, a 3D model other than the interference model intersecting with the straight line connecting the position 624 of the interference model acquired in step S803 and the position of the HMD 101 (between the interference model and the HMD) is specified, and the specified 3D model is specified. Is made transparent in step S805. By applying the processes of steps S803 to S805 to all the interference models, it is possible to make the 3D model other than the interference model located between each interference model and the HMD transparent.

<Third Embodiment>
Next, with reference to FIG. 9, details of the mixed reality image generation processing according to the third embodiment of the present invention will be described.

In the third embodiment, when the interfering 3D model (interference model) is inside or behind the other 3D model, the 3D model other than the interfering model is made transparent to the user. Make the interference model easier to see.

The process of FIG. 9 is executed by the CPU 201 of the PC 100 immediately before the process of step S706 of FIG. 7 in the first embodiment. Note that the description of the processing common to the first embodiment and the second embodiment will be omitted.

The CPU 201 of the PC 100 changes the color of the interference model to the color indicating that interference is occurring (step S901). The details of the process of step S901 are the same as the process of step S706 of FIG. 7, and thus detailed description will be omitted. The CPU 201 of the PC 100 acquires one unprocessed model among the interference models (step 902) and shifts the processing to step S903.

The CPU 201 of the PC 100 places the interference model acquired in step S902 inside the 3D model other than the interference model, or on the back surface of the 3D model other than the interference model when viewed from the position and orientation of the HMD 101. It is determined whether the interference model is located and a part or all of it is hidden by the other 3D model (step S903).

For example, the interference model is included in another 3D model which is arranged at the position 624 in the orientation 625 and is larger than the interference model itself (all dimensions of height and width are larger than the interference model) itself. If the interference model is present, it is determined that the interference model is located inside another 3D model.

In addition, for example, when drawing data is created and superimposed on a real image, if there is another 3D model that is drawn so as to overlap the interference model, the interference model is displayed behind the other 3D model. It is determined to be.

The CPU 201 of the PC 100 determines that the interference model acquired in step S902 is not a model located inside another 3D model other than the interference model (it is the outermost model), and is viewed from the position and orientation of the HMD 101. When it is determined that the interference model is located behind the other 3D model other than the interference model and a part or all of the 3D model is not hidden by the other 3D model, the process proceeds to step S906.

The CPU 201 of the PC 100 places the interference model acquired in step S902 inside the 3D model other than the interference model, or on the back surface of the 3D model other than the interference model when viewed from the position and orientation of the HMD 101. When it is determined that the interference model is located and a part or all of it is hidden by the other 3D model, the process proceeds to step S904.

In step S904, the CPU 201 of the PC 100 specifies all other 3D models within an arbitrary range from the interference point of the interference model acquired in step S902 (step S904), and makes all the specified other 3D models transparent. (Step S905). The details of the processing of steps S904 and S905 are the same as the processing of steps S707 and S708 of FIG. 7, respectively, and thus detailed description thereof will be omitted.

The CPU 201 of the PC 100 determines whether or not the processes of steps S903 to S905 have been applied to all the interference models (step S906). If there is an unapplied interference model, the process proceeds to step S902, and One interference model of the processing is acquired, and the processing of step S903 and subsequent steps is applied. If the processes of steps S903 to S905 have been applied to all the interference models, the process proceeds to step S709 of FIG. The above is the description of FIG. 9.

A model 1111 and a model 1112, which are interference models, are included in the model 1101 at 1400 in FIG. 14 (a model around the interference point is shown transparent). Also, 1410 shows that the model 1411 which is an interference model and the model 1101 are models of the outermost surface (a display state is shown only for identifying and displaying the interference model). In 1410 of FIG. 14, only the interference model 1411 and the solid model 1412 of the interference model 1101 that is in contact with the interference model 1411 are displayed.

As described above, according to the third embodiment of the present invention, it is possible to provide a mechanism that allows the user to easily confirm the interference between objects in mixed reality.

For example, when the interfering 3D model (interference model) is inside or behind the other 3D model, by making the 3D model other than the interference model transparent, the user can easily see the interference model. To do.

If the interference model is not hidden by another 3D model, the user may be able to check the interference model without changing the display form of the other 3D model. In the third embodiment, the interference model other than the interference model is hidden only when the interference model is hidden by the other 3D model without changing the unnecessary display form/state of the other 3D model. By changing the display form/state of another 3D model, the interference model can be easily seen by the user.

<Fourth Embodiment>

A fourth embodiment of the present invention will be described. In the fourth embodiment, if the 3D model, which is the design data (design product), has a design change and the 3D model interferes with another 3D model, the interference is performed. The 3D model other than the 3D model (interference model) that has become the above is made transparent so that the interference model can be easily seen by the user.

When the CPU 201 of the PC 100 receives the ending operation of the mixed reality application 404 (user operation for instructing the end of generation and display of the mixed reality image), the model information at the time of receiving the ending operation is stored in the external memory. To do. The model information includes the last update date and time 1082 and the design history 1083 shown in 1080 of FIG. 10 in addition to the information shown in 620 of FIG. In this embodiment, the file indicated by the model name 622 is a CAD file, and the 3D model is CAD data. The last update date and time 1082 and the design history 1083 are information of the last update date and time of the file and the design history stored in each CAD data. That is, the CPU 201 of the PC 100 copies all the CAD data indicated by the model information 620 to a predetermined area of the external memory and stores it when a user operation for instructing the end of generation and display of the mixed reality image is received.

When the CPU 201 of the PC 100 receives the activation operation of the mixed reality application 404 in order to restart the processing of FIG. 5, the model information at the time of the previous end stored in the external memory (620 in FIG. 6 and FIG. 10). 1080) is read and held on the memory. Then, in step S701, the latest model information (620 in FIG. 6 and 1090 in FIG. 10) read onto the memory in order to specify the model to be drawn is acquired, and the model information at the end of the previous time is acquired for each model. It is determined whether the last update date and time 1082 and the last update date and time 1092 of the latest model information are the same.

The CPU 201 of the PC 100 identifies the models determined to have different final update dates and times, and determines whether the design histories of these models are different (changed). For example, according to FIG. 10, the design history of “face pressing 01” which is not present in the model at the time of the previous termination is added to the model at the current activation. Therefore, it is determined that the design history is different (updated) (corresponding to design change determination means).

Even when the number of design histories and the identification information are the same, the parameters of the contents of the design history may be updated. Here, it is only necessary to judge whether the number of design histories and the identification information are different, but referring to the parameters of each design history, when there is a change in the parameters, It may be determined that they are different.

The CPU 201 of the PC 100 stores information on whether or not the design history has been changed (design change 1002 in the design change information 1000 in FIG. 10) on the memory for each model ID 1001.

The CPU 201 of the PC 100 limits the processing of steps S703 to S705 of FIG. 7 to 3D models within the angle of view of the HMD 101 in which the design change 1002 indicates “there is a design change”, which has been changed from the previous time. Then, the 3D model whose design has been changed from the previous time is executed.

The design was changed by limiting the design change to the 3D model and changing the display mode/state of models other than the interference model in order to identify the interference model and make the interference model easier to see. Thus, the user can easily confirm the model that may have interfered with another model. The above is the description of the fourth embodiment.

In the above description of the fourth embodiment, the presence or absence of a design change is determined by comparing the data of the design history of the 3D model. However, for example, the 3D model at the time when the mixed reality application was terminated last time The outer shape may be compared with the outer shape of the 3D model at the time of starting this time, and if the shapes are different, it may be determined that the design has been changed.

As described above, according to the fourth embodiment of the present invention, the user can easily confirm the interference between objects in mixed reality.

For example, when a 3D model whose design has been changed becomes an interference model, by changing the display form/state of other interference models other than the interference model, the interference model can be displayed so that it is easy for the user to see. it can.

As described above, according to the embodiments of the present invention, it is possible for the user to easily confirm the interference between objects in mixed reality.

In the above-described embodiment, the models other than the interference model are made transparent so that the interference model can be displayed easily. However, for example, the models other than the interference model can be easily seen. The interference model may be displayed so that it can be easily confirmed by moving the interference model to another position.

For example, as shown in FIG. 15, a model within an arbitrary range from the interference point, or a model 1301 between the interference point and the HMD 101 is shown at 1500 from the original position of the model 1301 to a predetermined distance from the interference point shown at 1510. Move (change position) to a distant position. Specifically, in step S708 of FIG. 7, a spherical 3D model is arranged around the interference point, and the position of the model 1301 is set in advance in the external memory of the PC 100 in the direction indicated by the normal vector of the spherical 3D model. Change to a position separated by a predetermined distance stored in. Then, drawing data of the position-changed 3D model is generated, and the process proceeds to step S709.

That is, by moving a model other than the interference model, the interference model can be displayed in an easily confirmable manner.

Further, in the above description of the embodiment, the virtual object is a 3D model, but a plane (2D) model having no depth may be treated as a virtual object.

Further, in the description of the above-described embodiment, a mixed reality image in which a 3D model is superimposed on a real image is generated and displayed on the HMD 101. However, in the case of an HMD having a transmissive display, for example, in a real space, Since the state can be seen through the other side of the display, it is not always necessary to display a real image on the display. Therefore, in the case of using an HMD having a transmissive display, for example, the transparency of the real image is set to 100% (invisible state), and an image obtained by superimposing the rendering data of the 3D model (virtual object) on the real image is combined. It may be generated as a real image and transmitted to the HMD 101 having a transmissive display.

Further, in the above description of the first embodiment, it is assumed that another model within an arbitrary range from the interference point is specified and made transparent. However, for example, each interference point is detected at the timing of step S707. As a center position, a spherical 3D model (a spherical 3D model with a radius of 20 cm according to FIG. 10) showing an arbitrary range is generated for each interference point, and drawing data of the 3D model is generated inside these spherical models. The interference model may be set not to be drawn and displayed, and the interference model may be set to the layer in front of the spherical model. By doing so, the image of the 3D model within an arbitrary range centering on the interference point is not drawn/displayed, and the interference model can generate a mixed reality image that can be visually displayed to the user.

Further, the above-described embodiments can be freely combined and executed.

Although the respective embodiments of the present invention have been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a recording medium, or the like. Specifically, it may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of one device.

For example, the HMD 101 including all the configurations of the PC 100 illustrated in FIG. 2 may execute all the processes described as being executed by the PC 100 in the above-described embodiment.

Further, the program according to the present invention is a program that allows a computer to execute the processing method of the flowcharts shown in FIGS. 5 and 7 to 9, and the storage medium according to the present invention has the processing method shown in FIGS. A computer-executable program is stored. The program in the present invention may be a program for each processing method of each device shown in FIG.

As described above, the recording medium recording the program that realizes the functions of the above-described embodiments is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It is needless to say that the object of the present invention can be achieved by executing the reading.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, non-volatile memory card, ROM, EEPROM, silicon. A disk, a solid state drive, etc. can be used.

Further, not only the functions of the above-described embodiments are realized by executing the program read by the computer, but also the OS (operating system) operating on the computer is actually executed based on the instructions of the program. It goes without saying that a case where some or all of the processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Furthermore, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is instructed based on the instruction of the program code. Needless to say, this also includes the case where the CPU or the like included in the function expansion unit performs some or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system including a plurality of devices or an apparatus including a single device. Further, it goes without saying that the present invention can be applied to the case where it is achieved by supplying a program to a system or an apparatus. In this case, by reading the recording medium storing the program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

Further, by downloading and reading a program for achieving the present invention from a server, a database or the like on a network using a communication program, the system or apparatus can enjoy the effects of the present invention.

It should be noted that the present invention also includes all configurations that combine the above-described embodiments and modifications thereof.

100 PC
101 HMD
103 optical marker 104 optical sensor 150 network

Claims (11)

An information processing device connectable to a display device including a display unit for displaying a virtual object,
An acquisition unit for acquiring a design history of the design object displayed as the virtual object,
Based on the design history acquired by the acquisition unit, a determination unit that determines whether or not a design change has been made in the design object,
Wherein as the first virtual object is a virtual object that indicates a designed object that has been determined to be the design change by determining means is visible, the first I virtual object der showing another designed object position of the virtual object and display control means for controlling the display of the second virtual object Ru predetermined positional relationship near,
An information processing apparatus comprising: Wherein the display control unit, an information processing apparatus according to claim 1, characterized in that to control the display of the second virtual object that does not interfere with the previous SL first virtual object. The display control means is within a predetermined distance from the point of interference between the second virtual object that interferes with the first virtual object or the first virtual object and the second virtual object, The information processing apparatus according to claim 1, wherein an image in which the transparency of the virtual object that does not interfere with the virtual object is adjusted is displayed. Wherein the display control unit, said interfering with the previous SL first virtual object second virtual object, or, the interference point position and the first virtual object and the second virtual object in the display device The information processing apparatus according to any one of claims 1 to 3, wherein control is performed to display an image in which the transparency of another virtual object positioned between the display apparatus and the position of the display apparatus is adjusted. The display control means controls the display of the second virtual object when the first virtual object is a virtual object located inside another virtual object. 4. The information processing device according to any one of 4 above. The display control means controls the display of the second virtual object when the first virtual object is a virtual object behind another virtual object when viewed from the position of the display device. The information processing apparatus according to any one of claims 1 to 5, which is characterized. A method for controlling an information processing device connectable to a display device including a display unit for displaying a virtual object, comprising:
An acquisition step of acquiring a design history of the design object displayed as the virtual object;
Based on the design history acquired by the acquisition step, a determination step of determining whether a design change has been made in the design object,
It said determining step such that the first virtual object is a virtual object becomes visible indicating the determined design was to have been the design change by the I virtual object der showing another designed object first position of the virtual object and a display control step of controlling the display of the second virtual object Ru predetermined positional relationship near,
Control method including. A program for controlling an information processing device connectable to a display device including a display unit for displaying a virtual object,
The information processing device,
An acquisition unit for acquiring a design history of the design object displayed as the virtual object,
Based on the design history acquired by the acquisition unit, a determination unit that determines whether or not a design change has been made in the design object,
Wherein as the first virtual object is a virtual object that indicates a designed object that has been determined to be the design change by determining means is visible, the first I virtual object der showing another designed object program for functioning as display control means for controlling the display of the second Ru position a predetermined positional relationship near according to the virtual object in the virtual object. An information processing system including a display device including a display unit for displaying a virtual object, and an information processing device,
An acquisition unit for acquiring a design history of the design object displayed as the virtual object,
Based on the design history acquired by the acquisition unit, a determination unit that determines whether or not a design change has been made in the design object,
Wherein as the first virtual object is a virtual object that indicates a designed object that has been determined to be the design change by determining means is visible, the first I virtual object der showing another designed object position of the virtual object and display control means for controlling the display of the second virtual object Ru predetermined positional relationship near,
An information processing system comprising: A method for controlling an information processing system including a display device having a display unit for displaying a virtual object, and an information processing device,
An acquisition step of acquiring a design history of the design object displayed as the virtual object;
Based on the design history acquired by the acquisition step, a determination step of determining whether a design change has been made in the design object,
It said determining step such that the first virtual object is a virtual object becomes visible indicating the determined design was to have been the design change by the I virtual object der showing another designed object first position of the virtual object and a display control step of controlling the display of the second virtual object Ru predetermined positional relationship near,
Control method including. A program for controlling an information processing system including a display device that displays a virtual object and an information processing device,
The information processing system,
An acquisition unit for acquiring a design history of the design object displayed as the virtual object,
Based on the design history acquired by the acquisition unit, a determination unit that determines whether or not a design change has been made in the design object,
Wherein as the first virtual object is a virtual object that indicates a designed object that has been determined to be the design change by determining means is visible, the first I virtual object der showing another designed object program for functioning as display control means for controlling the display of the second Ru position a predetermined positional relationship near according to the virtual object in the virtual object.

Images