JP2020187778A - Information processing device, information processing system, control method therefor, and program - Google Patents

Abstract

To provide an information processing device for easily confirming result of verification of a work space on a design object that is displayed by using mixed reality (MR), and a system, a control method, and a program.SOLUTION: Confirmation processing includes the steps of: storing a position-posture of a three-dimensional model; storing a first model to be a work object and a second model indicating a tool for performing work on the first model in correlation to each other; detecting a real object; determining whether a position of the detected real object and a position of the first model are within prescribed distance; specifying the second model that corresponds to the first model when it is determined that both positions are within the prescribed distance; determining a spatial position-posture of the second model in accordance with the position-posture of the real object; determining, when it is detected that the real object made a prescribed motion, whether the second model whose position-posture was determined has contacted another three-dimensional model in accordance with the real object having made the prescribed motion or the position-posture of the real object; and when it is determined that the second model has contacted, controlling so as to display information indicating the fact of contact on a display screen.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing device, an information processing system, a control method and a program thereof.

In recent years, mixed reality (hereinafter referred to as MR) technology has become widespread. MR technology can be used to provide a user wearing a head-mounted display (hereinafter referred to as HMD) with a simulated experience of a space in which a real object and a CG model are arranged. In generating MR space (mixed reality space in which real space and virtual space are combined), a marker may be used as a position index used for alignment of virtual space in real space.

Patent Document 1 describes a technique for drawing an image in which the marker is hidden when the marker is detected so that the user is not aware of the existence of the marker. Further, Patent Document 2 describes a technique for detecting the position and orientation of the HMD using an optical sensor.

MR technology is often used to allow HMD wearers to experience and visually recognize product design verification, construction / work site work simulation, and so on.

Japanese Unexamined Patent Publication No. 2000-350860 Japanese Unexamined Patent Publication No. 2006-301924 Japanese Unexamined Patent Publication No. 08-185431

When designing the shape of a product, it may be necessary for a worker to have a space for assembling and working on parts in the design object. For example, in Patent Document 3, in order to examine whether these work spaces are sufficiently secured, a plurality of tool and hand shape data are prepared in advance, and a tool corresponding to a part manually selected on the apparatus is provided. The size of the tool and the mounting direction of the parts are specified, and the image of the tool and its movement are displayed on the screen to confirm whether the work space is sufficient.

On the other hand, in an actual design site, the designer does not always know the working direction of the part. Therefore, even if the product designer or system designer uniformly stores and sets information on the system, such as from which direction the tool is inserted and how much the tool swing width is required for work, the actual situation is If there is a difference between the setting and the setting, the design must be modified after all.

Therefore, in the actual field, it is possible to ask a worker who has the know-how of these works to perform work such as assembling parts to verify whether the work space of the product is sufficient and to have the verification result fed back. It has been.

An object of the present invention is to provide a mechanism that makes it possible to easily confirm the verification result of a work space on a design object displayed by using MR.

The information processing device of the present invention can communicate with a display device that displays a composite real image in which a real image and a three-dimensional model are superimposed based on the position and orientation of the three-dimensional model in space. It is an information processing apparatus to be stored, and among the three-dimensional models, a first model to be worked on and a second model indicating a tool for performing work on the first model are stored in association with each other. A storage means, a real object detecting means for detecting a real object, a determining means for determining whether the position of the real object detected by the real object detecting means, and the position of the first model are within a predetermined distance. When the determination means determines that the position of the real object and the position of the first model are within a predetermined distance, the identification of specifying the second model corresponding to the first model. The means, the determining means for determining the position and orientation in space of the second model specified by the specific means according to the position and orientation of the real object, and the real object detected by the detecting means move in a predetermined manner. When the motion detecting means for detecting this and the motion detecting means detect that the real object has made a predetermined movement, the real object making the predetermined movement or the position / orientation of the real object is described. When it is determined that the second model whose position and orientation have been determined by the determining means has contacted the contact determining means for determining whether or not it has contacted another three-dimensional model, and the contact determining means has contacted the contact. It is characterized by including a display control means for controlling the information indicating the above to be displayed on the display screen of the display device.

According to the present invention, it is possible to provide a mechanism that makes it possible to easily confirm the verification result of the work space on the design object displayed by using MR.

It is a figure which shows an example of the structure of the information processing system in embodiment of this invention. It is a figure which shows an example of the hardware configuration of various devices in embodiment of this invention. It is a figure which shows an example of the functional structure of various devices in embodiment of this invention. It is a flowchart which shows the flow of the model display process of the spanner which matched with a nut in embodiment of this invention. It is a flowchart which shows the flow of the confirmation process of the movable range in embodiment of this invention. It is a flowchart which shows the flow of the distance specifying process in embodiment of this invention. It is a figure which shows an example of various data structures in embodiment of this invention. It is a figure which shows an example of the position of the user and CG when the real space and the virtual space are superposed in the embodiment of this invention. It is a figure which shows an example of the confirmation display of the movable range in embodiment of this invention. It is a figure which shows an example of the confirmation display of the movable range in embodiment of this invention. It is a figure which shows an example of the distance display in embodiment of this invention.

Hereinafter, an example of the embodiment of the present invention will be described with reference to the drawings.

First, an example of the configuration of the information processing system according to the embodiment of the present invention will be described with reference to FIG.

The PC 200 is communicably connected to an optical sensor and an HMD 101 (head-mounted display) via a network 150 (for example, LAN). The PC200 stores information on the shape, size, and position / orientation of a virtual object (CG / 3D model) superimposed on the real image captured by the camera of the HMD101, and the HMD101 obtained from the optical sensor. Using the position and orientation, generate an image of the virtual space (an image of a virtual object arranged in the above-mentioned position and orientation on the virtual space) obtained when the image is taken with a camera having the same angle of view in the same position and orientation as the HMD 101. Then, it is superimposed on the real image to generate a composite real image (MR image) and output it for display on the HMD 101.

The HMD 101 displays the MR image on the display screen. Further, the optical sensor tracks the optical marker installed in the HMD 101, identifies the position and orientation of the HMD 101 in the real space, and transmits the position and orientation to the PC 200.

Further, the PC 200 determines whether or not a real object possessed by a user who has experienced MR wearing the HMD 101 has made a predetermined movement, specifies a range of movement when the movement has a predetermined movement, and has a range of the movement. It is determined whether the virtual object is in contact with the virtual object (whether there is sufficient free space (space) for the work due to the movement), and the user is notified. For example, an MR image as shown in 910 of FIG. 9 is generated and transmitted to the HMD 101. Details of 910 will be described later in the description of FIG. The above is the description of FIG.

Next, an example of the hardware configuration of various devices according to the embodiment of the present invention will be described with reference to FIG.

The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

Further, the ROM 202 stores various programs necessary for realizing the functions executed by the BIOS (Basic Input / Output System), which is the control program of the CPU 201, the operating system (OS), and other various devices.

The RAM 203 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing a process into the RAM 203 and executing the program.

Various programs and the like used by the PC 200 of the present invention to execute various processes described later are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as needed. Further, the definition file and various information tables used by the program according to the present invention are stored in the external memory 211.

The input controller (input C) 205 controls the input from a pointing device (input device 210) such as a keyboard and a mouse.

The video controller (VC) 206 controls the display on the display such as the right-eye / left-eye display 222 included in the HMD 101. For the right-eye / left-eye display 222, for example, an external output terminal (for example, Digital Visual Interface) is used for output. The right-eye / left-eye display 222 is composed of a display for the right eye and a display for the left eye.

The memory controller (MC) 207 is an adapter to a hard disk (HD), a flexible disk (FD), or a PCMCIA card slot for storing boot programs, browser software, various applications, font data, user files, edit files, various data, and the like. Controls access to an external memory 211 such as a card-type memory connected via.

The communication I / F controller (communication I / FC) 208 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, Internet communication using TCP / IP is possible. The communication I / F controller 208 also controls communication with an optical sensor (optical sensor) through Gigabit Ethernet (registered trademark) or the like.

The general-purpose bus 209 is used to capture images from the right-eye / left-eye video camera 221 of the HMD 101. Input is performed from the right-eye / left-eye video camera 221 using an external input terminal (for example, an IEEE1394 terminal). The right-eye / left-eye video camera 221 is composed of a video camera for the right eye and a video camera for the left eye.

The CPU 201 can display the outline font on the display by executing the outline font expansion (rasterization) process on the display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction with a mouse cursor or the like (not shown) on the display. The above is the description of FIG.

Next, with reference to FIG. 3, the functional configurations of various devices according to the embodiment of the present invention will be described.

The image pickup unit 311 is an image pickup unit (camera) that captures a real image. The captured image transmission unit 312 is a transmission unit that transmits the captured real image to the PC 200.

The captured image storage unit 321 is a storage unit that stores a real image (captured image) received from the HMD 101. The HMD position / posture storage unit 322 is a storage unit that stores the position / posture of the HMD 101. For example, the position and orientation of the HMD 101 are acquired from an optical sensor and stored.

The model information storage unit 323 is a storage unit that stores the position and orientation of the three-dimensional model. For example, the model information 720 described later in the description of FIG. 7 is stored. The MR image generation unit 324 uses the position and orientation of the HMD 101 and the position and orientation of the three-dimensional model to superimpose the actual image viewed from the position of the HMD 101 and the image of the three-dimensional model viewed from the position of the HMD 101. It is a generation unit that generates (composite reality image). The MR image transmission unit 325 is a transmission unit (display control unit) that transmits an MR image to the HMD 101.

The MR image receiving unit 313 is a receiving unit that receives an MR image, and the MR image display unit 314 is a display unit that displays the received MR image.

The real object detection unit 326 is a processing unit that detects a real object, identifies the position and orientation of the detected real object, and stores it in the storage unit. The real object is, for example, an object to which a two-dimensional marker is attached (example: 801 in FIG. 8). The CPU 201 of the PC 200 is pasted with the two-dimensional marker based on the identification information, shape, size, angle, and position / orientation information of the two-dimensional marker specified from the image of the two-dimensional marker in the real image. Identify the position and orientation of the actual object.

The shape of the markers does not matter, but for example, they are square and all have the same size. It is assumed that a unique marker number is embedded in each marker. Then, it is assumed that each marker can be identified when the image is taken by the camera provided in the HMD 101, and the marker number is obtained when the marker is decoded. The distance from the HMD 101 to each marker is obtained from the size of each position detection marker in the captured image. (The position where the three distances obtained from the three markers overlap is the position of the HMD 101.) When the viewpoint is in the normal direction of the markers, the marker image looks like a square. Then, when the viewpoint deviates from the normal direction, the square appears distorted according to the degree of the deviation. That is, from this distortion, the orientation of the plane defined by the marker with respect to the axis of the viewpoint can be found, the distance between the viewpoint and the marker can be detected from the size of the marker, and the plane to which the marker is attached can be defined. Further, in the embodiment, the markers are provided with two marks that can be distinguished from each other, the center position of the markers is set as the origin, and two vectors from the origin to the marks on the above plane defined by the markers are set to 2. It is assumed that one axis in the normal direction from the center position of the axis and the marker defines three axes that define local (local) three-dimensional coordinates.

The distance determination unit 327 is a determination unit that determines whether the distance between the real object and the predetermined three-dimensional model is within the predetermined distance. For example, it is determined whether the distance between the position of the nut model and the real object 801 (target 801) held by the user is within a predetermined distance. It is assumed that the information of the predetermined distance (for example, the predetermined distance 740) is stored in advance in the external memory of the PC 200.

When it is determined that the nut model and the actual object are within a predetermined distance, the tool model specifying unit 328 specifies the type of the nut and specifies the three-dimensional model of the tool corresponding to the specified nut type. It is a specific part to do.

The movable range determination unit 329 is specified as a tool corresponding to the nut by the tool model specifying unit 328, and is the movable range of the three-dimensional model (tool model) of the tool arranged and displayed superimposed on the real object is sufficient? It is a judgment unit for judging. Specifically, when the user detects a predetermined movement while keeping the real object positioned within a predetermined distance from the nut model (for example, the operation and motion of tightening the nut stored in the external memory of the PC200 in advance). Another 3D model is a model that shows the range of movement of the tool model centered on the position and direction of the nut (when an operation whose value is determined to be within a predetermined range / operation of tightening the nut is detected). Determine if it indicates contact with.

The notification unit 330 is a notification unit that notifies the user that the movable range of the tool model is not sufficient when it is determined that the movable range of the tool model is not sufficient. For example, as shown in 910 of FIG. 9, it is a display control unit that generates an MR image indicating that the tool model is in contact with another three-dimensional model when the tool model is operated, and transmits the MR image to the HMD 101 for display. The above is the description of FIG.

Next, with reference to FIG. 4, the flow of the model display processing of the spanner according to the nut in the embodiment of the present invention will be described. FIG. 4 is a flowchart showing a flow of identification, arrangement, and display processing of a tool model corresponding to a predetermined model. The CPU 201 of the PC 200 performs the processes of steps S401 to S408 from the start of the HMD 101 and the PC 200 to the reception of the MR image generation / display end operation on the MR image display start / end instruction reception screen (not shown) of the PC 200. Execute repeatedly.

The CPU 201 of the PC 200 acquires the position / orientation information of the HMD 101 from the optical sensor and stores it in the external memory (step S401). Specifically, the HMD information 700 of FIG. 7 stores the HMD ID 701, which is the identification information of the HMD 101 that has acquired the position / orientation, the position 702 of the HMD 101 in the real space, and the attitude 703 (step S402).

The CPU 201 of the PC 200 acquires a real image captured by the camera of the HMD 101 and stores it in the memory. Then, the position and orientation of the detected real object is specified and stored by detecting the two-dimensional marker in the real image (step S403). For example, in the marker information 710 of FIG. 7, the position / orientation of the marker detected in the real image is overwritten / updated to the position 713 and the attitude 714. The marker ID 711 is the identification information of the two-dimensional marker, and the size 712 indicates the size (size) of the two-dimensional marker. It is assumed that the marker ID 711 and the size 712 are stored in the external memory of the PC 200 in advance.

Further, the CPU 201 of the PC 200 updates the detecting flag 715 of the two-dimensional marker (a real object to which the two-dimensional marker is attached) detected in step S403 to 1. The detecting flag 715 is a flag indicating whether or not the marker corresponding to the flag is being detected, is being detected when the detecting flag 715 = 1, and is detected when the detecting flag 715 = 0. Indicates that it is not in the state.

The CPU 201 of the PC 200 determines whether or not there is a predetermined type of 3D model (for example, a nut model) within a predetermined distance from the target 801 (FIG. 8: a real object to which a 2D marker is attached) detected in step S403. (Step S404). FIG. 8 shows an example of the positions of the user and CG when the real space and the virtual space are superimposed.

The position / orientation and type of the three-dimensional model are stored in the model information 720, the arranging model information 730, and the like in FIG.

The model information in the embodiment of the present invention will be described with reference to FIG. 7.

The model information 720 is information on the three-dimensional model stored in advance in the external memory of the PC 200. The model name 721 is a file name of the model which is the identification information of the three-dimensional model. It is assumed that the model file is stored in a predetermined folder on the external memory of the PC200. Type 722 indicates the type of 3D model. For example, nat indicates the model type = nut, spanner01 indicates the model type = spanner (tool), move01 indicates the model type = the movable range of the tool corresponding to the tool, and real indicates the model type = the actual object. It is shown that the model has the same shape and size as the superposed real object. Note that nat01 and nat02 are different types of nuts.

The corresponding model 723 is a type of model corresponding to the type shown in the type 722. For example, if type 722 = nut, the corresponding model 723 stores the type of tool (spanner) corresponding to the nut of the corresponding model 723. If type 722 = spanner, the corresponding model 723 stores the type of nut model (part model / model to be worked with a tool) corresponding to the spanner of the type 722. Further, if the type 722 = movable range, the corresponding model 723 stores the type of the spanner model corresponding to the movable range of the type 722.

The movable range 724 is a type of model indicating the movable range of the tool, which is stored in association with a model (tool / tool model) of type 722 = spanner (for example, 921 in FIG. 9). The marker ID 725 is identification information of the two-dimensional marker.

The placed model information 730 (placed model information 730A, 730B, 730C) stores the information of the model placed in the virtual space stored in the external memory of the PC 200.

The marker ID 731 is stored when the model with the model name 732 is arranged with reference to the position of the two-dimensional marker. For example, in the case of a tool (spanner) model, when the model is placed in the virtual space, the identification information of the marker indicating the reference position of the placement is stored.

The position 733 and the posture 734 indicate the position and posture of the model of the model name 732 in the virtual space.

In step S404, the CPU 201 of the PC 200 determines whether the position 733 of the model of type 722 = nut and the position 733 of the real object (position of the model of type 722 = real01) are within a predetermined distance of 740. Specifically, the CPU 201 of the PC 200 is prepared in advance on the external memory at a position in the virtual space corresponding to the position in the real space of the real object specified from the two-dimensional marker attached to the real object. Place a model (type 722 = real01 model) that has the same size and shape as the actual object, and place the point on the outer surface of the model that is the shortest distance from the position of the nut model. It is determined whether the distance of the nut from the model position is within a predetermined distance of 740.

When it is determined that the nut model and the real object are within a predetermined distance, the nut model closest to the real object is specified (step S405), and the specified nut model type 722 is referred to. The model of the tool corresponding to the type (model of the corresponding model 723) is acquired from the memory (step S406).

Then, the acquired tool model is used as a virtual space based on the position of the real object within a predetermined distance between the nut and the marker ID 731 referred to by the three-dimensional model indicating the real object used for determining the distance between the nut and the nut. The upper position) is determined as the placement position of the model and is placed (step S407). That is, as shown in 730B of FIG. 7, the same position and orientation as the position and orientation of the marker of the marker ID 731 are inserted and stored in the position 733 and the attitude 734.

The CPU 201 of the PC 200 is an image of a three-dimensional model arranged in the virtual space shown in the model information 730 being arranged in the real image acquired from the HMD 101, and is an image in the virtual space seen from the position and orientation of the HMD 101. An MR image on which a virtual image) is superimposed is generated and transmitted to the HMD 101 so as to be displayed on the display screen of the HMD 101 (step S408). For example, as shown in 910 of FIG. 9, an MR image in which the tool model 911 is arranged is generated by superimposing the position on the target 801. Note that 900 in FIG. 9 shows the state of the target 801 and the three-dimensional model 802 before the tool model 911 is arranged. The HMD 101 receives the MR image and displays it on the display screen. The above is the description of FIG.

Next, with reference to FIG. 5, the flow of the movable range confirmation process in the embodiment of the present invention will be described.

The CPU 201 of the PC 200 receives an instruction to press the "simulation start" button on the simulation start screen (not shown) displayed on the display screen of the PC 100 (step S501), and executes the processes after step S502.

The CPU 201 of the PC 200 determines whether the nut model and the tool model are in contact with each other (step S502). The contact between the models is determined by using the position and orientation of each model in the placed model information 730 and the shape and size of each model stored in the model file of the model name 732.

When the CPU 201 of the PC 200 determines that the nut model and the tool model are not in contact with each other, the CPU 201 ends the process of FIG. When it is determined that the nut model and the tool model are in contact with each other, it is determined whether or not the real object has made a predetermined movement (step S503). Judgment as to whether or not the real object has made a predetermined movement is executed by using, for example, a camera installed in the HMD 101 and a known motion detection technique (eg, a technique described in Japanese Patent Application Laid-Open No. 2012-008772). It shall be. It is assumed that the information on the predetermined movement is separately stored in the external memory of the PC 200. The predetermined movement referred to here is a movement for tightening the nut with the position and orientation of the nut model (axial direction of the nut) as the axis.

When the predetermined movement is not performed, the process of FIG. 5 is terminated. When it is determined that the tool is moving in a predetermined manner, a model having a movable range of 724 corresponding to the three-dimensional model of the tool superimposed on the real object in the predetermined motion is acquired (step S504). This is a model such as 921 in FIG. 9 showing the range in which the tool is moved to turn / tighten the nut with the tool. The CPU 201 of the PC 200 arranges the model showing the acquired movable range in the virtual space in the same position and posture as the model of the tool. Specifically, as shown in 730C of FIG. 7, a movable range model (M004.vrml) is added to the arranging model information 730, and the same position / orientation value as the spanner model (M003.vrml) is stored. (Step S505).

The CPU 201 of the PC 200 determines whether the model showing the movable range is in contact with a three-dimensional model other than the nut (step S506). An example of contact with a three-dimensional model other than the nut model is shown in FIG. 930. An example of no contact with a three-dimensional model other than the nut model is shown in FIG. 920.

When the model showing the movable range is not in contact with the 3D model other than the nut, the model showing the movable range is in contact with the 3D model other than the nut, as shown in the “OK” object 922 of FIG. A process of displaying information indicating that there is no such information on the display screen of the HMD 101 is performed (step S509). Specifically, an MR image is generated by arranging a three-dimensional model as shown in 922 of FIG. 9 at a position on a virtual space separated from the model showing the movable range of the tool by a predetermined distance, and transmitted to the HMD 101. The HMD 101 receives and displays this. Then, when it is determined that the model showing the movable range is not in contact with the three-dimensional model other than the nut, the position of the model of the movable range (for example, the apex of the model of the movable range as shown in 923 of FIG. 9 is the position). The position on the virtual space) and the time when it is determined that the contact is not made are stored in the memory (OK position 751 of the determination position information 750 in step S510 / FIG. 7, date and time 752).

On the other hand, when it is determined in step S506 that the model showing the movable range is in contact with the three-dimensional model other than the nut, the model showing the movable range is as shown in the “NG” object 932 of FIG. 9 930. A process of displaying information indicating contact with a three-dimensional model other than the nut on the display screen of the HMD 101 is performed (step S507). Specifically, a three-dimensional model as shown in 932 of FIG. 9 is placed at a position on the virtual space separated from the model showing the movable range of the tool by a predetermined distance, and the position where each model is in contact is shown. A model as shown in 931 is generated, arranged at a position (contact position) where each model is in contact, an MR image is generated, and the model is transmitted to the HMD 101. The HMD 101 receives and displays this.

Then, when it is determined that the model showing the movable range is in contact with a three-dimensional model other than the nut, the position of the model in the movable range and the time determined to be in contact are stored in the memory ( Step S508 / NG position 753 of the determination position information 750 in FIG. 7, date and time 754 / corresponding to the position storage means). Then, the model in which the nut model to which the tool model is in contact is located (the nut model matches / is installed) is specified, and the nut model on the model is specified. The installation position of the nut in contact with the nut is specified (step S511). For example, a point on the surface of the model in contact with the nut, which is closest to the center point of the nut, is specified as the installation position.

The CPU 201 of the PC 200 copies a file of a model in contact with the nut (eg, the model shown in the three-dimensional model 802) to a predetermined folder, and places the copied file at the installation position on the model. A model of a sphere showing the above is generated and arranged, and overwritten and saved (step S512). That is, if it is determined in step S506 that the model showing the movable range is in contact with another three-dimensional model (eg, the model shown in the three-dimensional model 802), a red sphere is generated. If it is determined that the model showing the movable range is not in contact with other 3D models, a blue sphere shall be generated and placed. This makes it possible for each ball to easily confirm whether or not the movable range of the tool with the point as the reference position is sufficient. The above is the description of FIG.

In step S512, a model showing the contact position between the model 921 and the model 802 showing the movable range and the model 921 itself showing the movable range were generated on the copied model file, and the model 802 was contacted in the virtual space. It may be arranged at a position relative to the model 802 at the time when it is determined to be.

Further, in the determination in step S506, the contact determination with the model indicating the movable range of the tool may be performed except for the surface on which the nut is installed.

Further, immediately after the process of step S512, the image displayed on the HMD 101 (for example, the MR image generated in step S507 or S509) may be captured, output to a predetermined folder, and saved.

By outputting the copy file, outputting the captured image, and saving the copy file, other users can check the copy file. Therefore, for example, the information on whether or not the work space (movable range of the tool) considered by the worker wearing the HMD 101 is sufficient can be easily handed over to the designer who is another user. Also, it can be easily confirmed.

According to the process of FIG. 5, an appropriate model corresponding to the model can be selected and displayed by using the position of the real object and the position of the model within a predetermined distance from the real object.

In addition, by acquiring and arranging a model of a movable range corresponding to the model of the tool, it is possible to visually confirm whether or not a sufficient working range using the tool is secured.

Next, with reference to FIG. 6, the flow of the distance specifying process in the embodiment of the present invention will be described.

In the process of FIG. 6, for example, a worker who wears the HMD 101 and verifies whether the work space (movable range of the tool) is sufficiently secured by using MR is a work space in a place where a nut currently exists. Is not enough, so it is a process to display the verification result that verified to which position on the model the nut (bolt to be tightened with the nut) should be moved.

For example, when the spanner is swung left and right at the position 1001 in 1000 of FIG. 10 and then the spanner is moved to the position 1011 in 1010 of FIG. 10, as shown in FIG. 11, from the position of 1001. It makes the distance between the positions of 1011 identifiable. The details will be described below.

The CPU 201 of the PC 200 determines whether the model of the tool and the surface of the model other than the nut are in contact with each other, and if it is determined that they are in contact with each other, determines whether the contact time is equal to or longer than a predetermined time (step S601). ). It is assumed that the information at the predetermined time is stored in advance in the external memory of the PC 200.

If the contact time is less than a predetermined time, the process of FIG. 6 is terminated. If it is determined that the tool has been in contact for a predetermined time or longer, a model of the movable range corresponding to the model of the tool is acquired, and the model of the movable range is virtualized in the same position and posture as the model of the tool. Arrange in space (step S602).

Then, it is determined whether the model of the movable range is in contact with a surface of the model other than the surface with which the tool is in contact (step S603). If the tool is in contact with a surface of the model other than the surface it is in contact with, the process ends. If the tool is not in contact with a surface of the model other than the surface in contact, the process proceeds to step S604, a point on the model in contact with the model of the tool is specified, and the point is stored in the memory. (Step 604). For example, when the model of the tool is a spanner, the position of the center point (example: 1011 in FIG. 10) of the head portion for tightening the nut of the spanner is specified.

Then, it is determined whether the model of the movable range of the tool and the model other than the nut are in contact within the past predetermined time (step S605). Specifically, it is determined whether the time difference between the date and time 754 in FIG. 7 and the current date and time that can be specified by the clock function of the PC 200 is within a predetermined time (not shown) stored in advance in the external memory. is there.

If there is a record that the model of the movable range of the tool and the model other than the nut are in contact within the past predetermined time, the model is stored in the memory in step S604 from the position indicated by the record (NG position 753). The distance to the position is calculated (step S606).

Then, a three-dimensional model showing the distance is generated (step S607). For example, it is a three-dimensional model showing the distance between two points as shown in models 1101 and 1102 shown in 1100 of FIG.

The CPU 201 of the PC 200 arranges the model indicating the distance between the position indicated by the NG position 753 and the position stored in the memory in step S604 so as to connect the two points indicated by the two positions (step). S608), an MR image is generated by superimposing it on the real image acquired from the HMD 101 (corresponding to the relative relationship image generation means indicating the relative position / relative relationship of the two points), and transmitted to the HMD 101 (step S609). The HMD 101 receives the MR image and displays the MR image as shown by 1100 in FIG. 11 on the display screen. The above is the description of FIG.

The model showing the distance described above is not limited to the shapes 1101 and 1102 in FIG. For example, a model having a shape like the model 1111 shown in 1110 of FIG. 11 may be generated.

Further, after step S609, the CPU 201 of the PC 200 copies and saves the file of the model 802 to a predetermined folder, and two points on the model 802 connected by the model of the distance (the position indicated by the NG position 753). , The point of the position stored in the memory in step S604), and a three-dimensional model indicating the distance between the two points may be generated, arranged, and overwritten on the copied model file. .. That is, the two points on the model 802 connected by the model of the distance (the position indicated by the NG position 753 and the point of the position stored in the memory in step S604) and the distance between the two points are shown. The process of associating and storing the three-dimensional model is performed.

Further, immediately after step S609, the image displayed on the HMD 101 (for example, the MR image generated in step S609) may be captured, output to a predetermined folder, and saved.

By outputting the copy file, outputting the captured image, and saving the copy file, other users can check the copy file. Therefore, for example, the information on whether or not the work space (movable range of the tool) considered by the worker wearing the HMD 101 is sufficient can be easily handed over to the designer who is another user. Moreover, it can be easily confirmed.

For example, it is possible to easily inform the designer to which position the nut (bolt to be tightened with the nut) should be moved to secure a sufficient work space.

In step S601, it is determined whether the surface of the model of the tool and the surface of the model other than the nut are in contact with each other for a predetermined time or longer. For example, as in S503 of FIG. 5, the model is in the same position as the model of the tool. It may be determined whether or not the real object is making a predetermined movement, and if it is determined that the real object is making a predetermined movement, the process may be shifted to step S602. By doing so, the user determines that the work space is not sufficient simply by performing an intuitive operation such as tightening a part such as a nut at a position different from the position of the nut that is determined to have insufficient work space. It is possible to know the distance from the position of the nut to the position on the model where the work space is judged to be sufficient. In addition, the information can be output.

As described above, according to the present invention, it is possible to provide a mechanism that makes it possible to easily confirm the verification result of the work space on the design object displayed by using MR.

In the above-described embodiment, it is determined whether or not the real object (target 801) has made a predetermined movement by using the camera installed in the HMD 101 and the information of the predetermined movement stored in the external memory of the PC200. However, for example, the history of the position and orientation of the model indicating the real object arranged at the same position and orientation as the target 801 is stored, and the history of the position and orientation and the predetermined movement separately stored in the external memory of the PC 200 are stored. It is also possible to judge whether or not the real object has made a predetermined movement by comparing the information of.

Further, in the above-described embodiment, the example of the tool is a spanner and the example of a part is a nut, but the model of the tool and the part is not limited to this. For example, a model in which a value indicating a screw or a bolt is inserted as a part type may be stored and used instead of a nut. Further, a model in which a value indicating a wrench or a driver is inserted as a tool type may be stored and used instead of a spanner.

Further, the model (move) showing the movable range is assumed to be stored in the external memory in advance, but for example, a real object determined to have made a predetermined movement in S503 of FIG. 5 or S601 of FIG. The history of the position and orientation of the superimposed tool model (located at the position of the target marker M111) within a predetermined time is stored and stored, and the history is referred to, for example, specified from the history. , A model having a shape (showing a predetermined movement) showing a trajectory that the model of the tool specified from the coordinate values has passed by acquiring a coordinate value indicating the trajectory that the model of the tool has passed in the past 5 seconds. A model) may be generated and arranged in the virtual space in S505 or S602. By using the movement of the actual tool model, it is possible to verify the existence of a more realistic work space.

The present invention can be, for example, an embodiment as a system, an apparatus, a method, a program, a storage medium, or the like, and specifically, may be applied to a system composed of a plurality of devices. It may be applied to a device consisting of one device.

The present invention includes a software program that realizes the functions of the above-described embodiments, which is directly or remotely supplied to the system or device. The present invention also includes cases where the computer of the system or device can also read and execute the supplied program code.

Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself installed on the computer also realizes the present invention. That is, the present invention also includes a computer program itself for realizing the functional processing of the present invention.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, MOs, CD-ROMs, CD-Rs, CD-RWs, and the like. There are also magnetic tapes, non-volatile memory cards, ROMs, DVDs (DVD-ROM, DVD-R) and the like.

In addition, as a program supply method, a browser of a client computer is used to connect to an Internet homepage. Then, the computer program itself of the present invention or a compressed file including the automatic installation function can be supplied from the homepage by downloading it to a recording medium such as a hard disk.

It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from different homepages. That is, the present invention also includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, and distributed to users, and the key information for decrypting the encryption is downloaded from the homepage to the user who clears the predetermined conditions. Let me. Then, by using the downloaded key information, it is also possible to execute an encrypted program and install it on a computer.

Further, the function of the above-described embodiment is realized by the computer executing the read program. In addition, based on the instruction of the program, the OS or the like running on the computer performs a part or all of the actual processing, and the function of the above-described embodiment can be realized by the processing.

Further, the program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer. After that, based on the instruction of the program, the function expansion board, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing, and the function of the above-described embodiment is also realized by the processing.

It should be noted that the above-described embodiments merely show examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these.
That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

101 HMD
150 network 200 PC

Claims (9)

An information processing device that stores the position and orientation of the 3D model that can communicate with a display device that displays a mixed reality image in which a real image and a 3D model are superimposed based on the position and orientation of the 3D model in space. ,
Among the three-dimensional models, a storage means for storing the first model to be worked on and the second model indicating a tool for performing work on the first model in association with each other.
Real object detection means for detecting real objects and
A determining means for determining whether the position of the real object detected by the real object detecting means and the position of the first model are within a predetermined distance.
Specific means for identifying the second model corresponding to the first model when the determination means determines that the position of the real object and the position of the first model are within a predetermined distance. When,
A determination means for determining the position and orientation of the second model in space specified by the specific means according to the position and orientation of the real object.
A motion detecting means for detecting that a real object detected by the detecting means has made a predetermined movement,
When the motion detecting means detects that the real object has made a predetermined movement, the position / orientation is determined by the determining means according to the position / orientation of the real object making the predetermined movement or the real object. A contact determination means for determining whether the second model has contacted another three-dimensional model,
A display control means that controls to display information indicating the contact on the display screen of the display device when it is determined that the contact is made by the contact determination means.
An information processing device characterized by being equipped with. When the contact determination means determines that the real object or the second model whose position and orientation have been determined by the determination means in accordance with the position and orientation of the real object has come into contact with another three-dimensional model. A first capturing means for capturing a mixed reality image to be displayed on the display device,
The information processing apparatus according to claim 1, further comprising. When the movement detecting means detects that the real object has made a predetermined movement, the contact determination means determines the real object or the real object according to the position and orientation of the real object. The first model used to identify the second model by the specific means when it is determined that the second model whose position and orientation are determined by the means is in contact with another three-dimensional model. The first position storage means for storing the position of
When the movement detecting means detects that the real object has made a predetermined movement, the contact determination means determines the real object making the predetermined movement and the position / posture of the real object. A second position storage means for storing a position at a predetermined distance from the real object when it is determined that the second model whose position and orientation have been determined by the means did not come into contact with another three-dimensional model. When,
A relative relationship image generation means for generating the mixed reality image showing the relative relationship between the first position stored in the first position storage means and the second position stored in the second position storage means. When,
With
The display control means is characterized in that a mixed reality image showing the relative relationship between the first position and the second position generated by the relative relationship image generation means is transmitted to be displayed on the display device. The information processing device according to claim 1 or 2. When it is determined by the contact determination means that the second model whose position and orientation have been determined by the determination means does not contact another three-dimensional model according to the position and orientation of the real object and the real object. A second capturing means for capturing the relative image, which is a composite reality image to be displayed on the display device.
The information processing apparatus according to claim 3, further comprising. After it is determined by the contact determination means that the second model whose position and orientation have been determined by the determination means according to the position and orientation of the real object or the real object has come into contact with another three-dimensional model, the above. When it is determined by the contact determination means that the second model whose position and orientation have been determined by the determination means does not contact another three-dimensional model according to the position and orientation of the real object and the real object. A third position storage means for storing the first position and the second position in association with another three-dimensional model determined to be in contact with each other.
The information processing apparatus according to claim 3 or 4, wherein the information processing apparatus is provided with. Claims 3 to 5 characterized in that the relative relationship image generation means generates a mixed reality image including information indicating a distance between the first position and the second position as the relative relationship image. The information processing apparatus according to any one of the above items. A control method for an information processing device that stores the position and orientation of the 3D model, which can communicate with a display device that displays a composite reality image in which a real image and a 3D model are superimposed based on the position and orientation of the 3D model in space. And
Among the three-dimensional models, a storage process of associating and storing a first model to be worked on and a second model indicating a tool for performing work on the first model.
A real object detection process that detects a real object,
A determination step of determining whether the position of the real object detected in the real object detection step and the position of the first model are within a predetermined distance.
A specific step of specifying the second model corresponding to the first model when it is determined in the determination step that the position of the real object and the position of the first model are within a predetermined distance. When,
A determination step of determining the position and orientation of the second model in space specified in the specific step according to the position and orientation of the real object.
A motion detection process for detecting that a real object detected in the detection process has made a predetermined motion,
When it is detected that the real object has made a predetermined movement in the motion detection step, the position / posture is determined in the determination step according to the position / orientation of the real object having the predetermined movement or the real object. A contact determination step of determining whether the second model has contacted another three-dimensional model, and
A display control step of controlling to display information indicating the contact on the display screen of the display device when it is determined that the contact is made in the contact determination step.
A control method for an information processing device, which comprises. Execution is performed by an information processing device that stores the position and orientation of the 3D model, which can communicate with a display device that displays a composite reality image in which the reality image and the 3D model are superimposed based on the position and orientation of the 3D model in space. It ’s a possible program,
The information processing device
Among the three-dimensional models, a storage means for storing the first model to be worked on and the second model indicating a tool for performing work on the first model in association with each other.
Real object detection means for detecting real objects and
A determining means for determining whether the position of the real object detected by the real object detecting means and the position of the first model are within a predetermined distance.
Specific means for identifying the second model corresponding to the first model when the determination means determines that the position of the real object and the position of the first model are within a predetermined distance. When,
A determination means for determining the position and orientation of the second model in space specified by the specific means according to the position and orientation of the real object.
A motion detecting means for detecting that a real object detected by the detecting means has made a predetermined movement,
When the motion detecting means detects that the real object has made a predetermined movement, the position / orientation is determined by the determining means according to the position / orientation of the real object making the predetermined movement or the real object. A contact determination means for determining whether the second model has contacted another three-dimensional model,
A program of an information processing device characterized in that when it is determined by the contact determination means that the contact is made, the information indicating that the contact is made is made to function as a display control means for controlling display on the display screen of the display device. .. An information processing system including a display device that displays a composite real image in which a real image and a three-dimensional model are superimposed based on the position and orientation of the three-dimensional model in space, and an information processing device that stores the position and orientation of the three-dimensional model. And
Among the three-dimensional models, a storage means for storing the first model to be worked on and the second model indicating a tool for performing work on the first model in association with each other.
Real object detection means for detecting real objects and
A determining means for determining whether the position of the real object detected by the real object detecting means and the position of the first model are within a predetermined distance.
Specific means for identifying the second model corresponding to the first model when the determination means determines that the position of the real object and the position of the first model are within a predetermined distance. When,
A determination means for determining the position and orientation of the second model in space specified by the specific means according to the position and orientation of the real object.
A motion detecting means for detecting that a real object detected by the detecting means has made a predetermined movement,
When the motion detecting means detects that the real object has made a predetermined movement, the position / orientation is determined by the determining means according to the position / orientation of the real object making the predetermined movement or the real object. A contact determination means for determining whether the second model has contacted another three-dimensional model,
A display control means that controls to display information indicating the contact on the display screen of the display device when it is determined that the contact is made by the contact determination means.
An information processing system characterized by being equipped with.

Images