JP2018018362A - Virtual space control device, control method thereof, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mechanism that enables component data to be easily viewed on a virtual space, even when a plurality of pieces of the component data having design changed are included in a three-dimensional model.SOLUTION: In a virtual space control device for presenting a virtual space in which a three-dimensional model constituted from a plurality of pieces of component data is placed, the component data having design changed is identified by comparing the plurality of component data constituting the three-dimensional model before design change with the plurality of component data constituting the three-dimensional model after the design change. And, the three-dimensional model is placed in the virtual space, and a display state of the identified component data existing in a predetermined range from a current position of a user in the virtual space is set to a recognizable state in the placed three-dimensional model.SELECTED DRAWING: Figure 10

Description

  The present invention is a virtual space control device capable of making it easy to view the part data on the virtual space even when the design-changed part data is included in the 3D model, a control method thereof, And the program.

  Conventionally, front loading of business is performed in the assembly manufacturing industry. Front loading is an effort to improve quality by examining problems that may occur in the post-process in the initial process. For example, CAD data being created in the design department is displayed with three-dimensional CAD software, and downstream departments such as an assembly department and a quality assurance department simulate part interference checks.

  However, the CAD data acquired from the design department is incomplete. Since the design department and the downstream department cooperate to advance product design, CAD data is exchanged many times between the design department and the downstream department. In other words, it is necessary for the downstream department to acquire CAD data improved by the design department over and over, and to perform simulation and the like each time.

  At that time, it is important to properly communicate the details of the design change from the design department to the downstream department. However, if there are many places where the design has been changed, there is a possibility that the design change part may be forgotten. Therefore, the downstream department must confirm what was changed from the previously acquired CAD data by comparing the CAD data before the design change with the CAD data after the design change.

  Therefore, in Patent Document 1, the CAD data before the design change and the CAD data after the design change are divided into a plurality of areas each having a fixed shape, and the volume is calculated for each area and compared before and after the design change. A mechanism for identifying a location where the shape has been changed is disclosed.

JP-A-11-345256

  Since there is a mechanism for automatically specifying a design change location by the mechanism of Patent Document 1 and the conventional function of 3D CAD software, an operator in the downstream department can easily check the design change location. it can.

  As a method for allowing the user to confirm the location where the design has been changed, there is a method in which the CAD data of the assembly is displayed, and the location where the design has been changed in the CAD data is colored and displayed. However, if there are many design-changed portions, all of them are identified and displayed, and there is a problem that it is difficult for the user to visually recognize.

  An object of the present invention is to provide a mechanism that makes it easy to view the part data in a virtual space even when the design-modified part data is included in a plurality of three-dimensional models. .

  In order to achieve the above object, a virtual space control device of the present invention is a virtual space control device for presenting a virtual space in which a three-dimensional model composed of a plurality of component data is arranged, and is designed before design change. A plurality of part data constituting the three-dimensional model and a plurality of part data constituting the three-dimensional model after the design change, specifying means for specifying the part data that has undergone the design change, and the tertiary The arrangement means for arranging the original model in the virtual space, and the display state of the part data specified by the specifying means existing within a predetermined range from the current position of the user in the virtual space is the tertiary arranged by the arrangement means. Setting means for setting in an identifiable state in the original model.

  According to the present invention, even when a plurality of part data whose design has been changed is included in the three-dimensional model, the part data can be easily viewed in the virtual space.

It is a figure which shows an example of the system configuration | structure of the information processing system in embodiment of this invention. 2 is a diagram illustrating an example of a hardware configuration of an information processing apparatus 101 and an HMD 102. FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing apparatus 101. FIG. It is a flowchart which shows an example of the flow of the whole process. It is a flowchart which shows an example of the detailed process flow of a difference specific process. It is a figure which shows the reference structure of old assembly data and new assembly data, and an example of the data structure of the comparison table. 5 is a diagram illustrating an example of a table configuration of a difference list 700. FIG. It is a figure which shows the user who mounted | worn HMD102, and the three-dimensional model of the new assembly data arrange | positioned in virtual space. It is a flowchart which shows an example of the detailed process flow of a difference data identification process. It is a figure which shows an example at the time of identifying and displaying the design-changed difference data which exists in the predetermined range from the user wearing HMD102. It is a flowchart which shows an example of the detailed process flow of a difference data arrangement | positioning process. It is a figure which shows an example at the time of selecting the difference data by which the design was changed. It is a figure which shows an example at the time of selecting the difference data by which the design was changed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to the present embodiment. The information processing system shown in FIG. 1 is a system for allowing a user to browse a virtual space using mixed reality technology (hereinafter referred to as MR technology). In the information processing system, the HMD 102 is connected to the information processing apparatus 101 so as to be capable of data communication with each other. The connection between the information processing apparatus 101 and the HMD 102 may be a wired connection or a wireless connection. The configuration of various terminals connected on the system of FIG. 1 is an example, and there are various configuration examples depending on the application and purpose. For example, although the present embodiment will be described on the premise of the MR technique, it can be realized by using a technique such as virtual reality or augmented reality.

  The information processing apparatus 101 (virtual space control apparatus) is a general-purpose computer. The information processing apparatus 101 captures (captures) a real space image (hereinafter referred to as a real space image) taken by the HMD 102 and a virtual space image generated by the information processing apparatus 101 (hereinafter referred to as a virtual space image). And are superimposed. The image generated in this way (hereinafter referred to as a mixed reality image) is transmitted to the HMD 102. Since the conventional technique is used for the MR technique, a detailed description is omitted. The information processing apparatus 101 may be a personal computer or a large computer such as a server. Furthermore, a mobile terminal such as a mobile phone or a tablet terminal may be used. The type of computer is not particularly limited.

  The HMD 102 is a head mounted display. The HMD 102 is a device that is worn on the user's head, and includes a right-eye and left-eye video camera, and a right-eye and left-eye display. The HMD 102 transmits a real space image captured by the video camera of the HMD 102 to the information processing apparatus 101. The mixed reality image is received from the information processing apparatus 101 and displayed on the display. Since the HMD 102 is provided with displays for the right eye and the left eye, a stereoscopic effect can be obtained by parallax. The physical space image captured by the HMD 102 and the mixed reality image displayed by the HMD 102 are preferably moving images (video), but may be still images captured at a predetermined interval. Further, a device in which a display and a video camera are installed on the front and back of the hardware, such as a smartphone, may be used as a substitute for the head mounted display.

  In this embodiment, the system including the information processing apparatus 101 and the HMD 102 is described. However, a device (virtual space control apparatus) in which the information processing apparatus 101 and the HMD 102 are integrated may be used. That is, various types of hardware of the information processing apparatus 101 may be mounted on the HMD 102 so that the HMD 102 operates only.

  In addition, an infrared camera 104 is connected to the information processing apparatus 101. The infrared camera 104 is an optical sensor using infrared rays. The infrared camera 104 irradiates the real space with infrared rays and shoots the infrared rays reflected by the objects in the real space, thereby specifying the position and orientation of the object in the real space in the coordinate system defined by the infrared camera 104. Using this infrared camera 104, the position and orientation (direction, inclination, direction of line of sight, etc .; the same applies hereinafter) of the HMD 102 (that is, the user) in the real space are specified. The HMD 102 includes an optical marker 103 that reflects infrared rays, and the infrared camera 104 can identify the position and orientation of the HMD 102 by photographing the infrared rays reflected by the optical marker 103. It is desirable to install a plurality of infrared cameras 104 in the information processing system so that the optical marker 103 of the HMD 102 worn by the user can be photographed or detected regardless of the position or posture of the user. The HMD 102 that can specify the position and orientation will be described as being present in the imaging range of the infrared camera 104.

  In the present embodiment, the infrared camera 104 is used to specify the position and orientation of the HMD 102 in the real space. However, the present invention is not limited to this as long as the position and orientation in the real space can be specified. For example, a magnetic sensor may be used, or an image captured by the HMD 102 may be analyzed to specify the position and orientation.

  In the present embodiment, the user wearing the HMD 102 performs various operations using the target marker 105. The target marker 105 is composed of a two-dimensional marker. The information processing apparatus 101 specifies the position / posture of the target marker 105 in the real space by specifying the inclination of the two-dimensional marker and the distance from the HMD 102 by image recognition. be able to. The type of the two-dimensional marker is not particularly limited, but it is desirable that the type of the two-dimensional marker is easily acquired by image recognition.

  FIG. 2 is a diagram illustrating an example of each hardware configuration of the information processing apparatus 101 and the HMD 102.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  The ROM 202 or the external memory 211 stores a BIOS (Basic Input / Output System) that is a control program of the CPU 201 and an operating system. Furthermore, various programs to be described later necessary for realizing the functions executed by the information processing apparatus 101 are stored. The RAM 203 functions as a main memory, work area, and the like for the CPU 201.

  The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 203 and executing the program.

  An input controller (input C) 205 controls input from a pointing device (input device 210) such as a keyboard and a mouse.

  A video controller (VC) 206 of the information processing apparatus 101 controls display on a display device such as the right-eye / left-eye display 222 or the display 212 provided in the HMD 102. For example, an external output terminal (for example, Digital Visual Interface) is used for output to the right eye / left eye display 222. The right-eye / left-eye display 222 includes a right-eye display and a left-eye display. The display 212 is a liquid crystal display or the like, and displays the same display as the right eye / left eye display 222 or a GUI (Graphical User Interface) for operating the virtual space.

  A memory controller (MC) 207 controls access to an external memory 211 (storage means) that stores a boot program, browser software, various applications, font data, user files, edit files, various data, and the like. The external memory 211 includes, for example, a hard disk (HD), a flexible disk (FD), or a card type memory connected to a PCMCIA card slot via an adapter.

  A communication I / F controller (communication I / FC) 208 is connected to and communicates with an external device via a network, and executes communication control processing in the network. For example, Internet communication using TCP / IP is possible. In particular, the communication I / F controller 208 of the information processing apparatus 101 also controls communication with the infrared camera 104.

  A general-purpose bus 209 of the information processing apparatus 101 is used to capture an image captured by the right-eye / left-eye video camera 221 of the HMD 102 connected to the information processing apparatus 101. Input from the right-eye / left-eye video camera 221 is performed using an external input terminal (for example, an IEEE 1394 terminal). The right-eye / left-eye video camera 221 includes a right-eye video camera and a left-eye video camera.

  Note that the CPU 201 enables display on a display by executing outline font rasterization processing on a display information area in the RAM 203, for example. In addition, the CPU 201 allows a user instruction with a mouse cursor (not shown) on the display.

  Various programs used by the information processing apparatus 101 of the present invention to execute various processes described later are recorded in the external memory 211 and are executed by the CPU 201 by being loaded into the RAM 203 as necessary. is there. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 211.

  FIG. 3 is a functional configuration diagram illustrating a functional configuration of the information processing apparatus 101. Note that the functional configuration of the information processing apparatus 101 in FIG. 3 is merely an example, and there are various configuration examples depending on applications and purposes.

  The information processing apparatus 101 includes, as functional units, a communication control unit 301, a position / attitude specifying unit 302, a real space image acquisition unit 303, a virtual space generation unit 304, a three-dimensional model control unit 305, a virtual space image acquisition unit 306, and a mixed reality. An image generation unit 307 is provided. Furthermore, the information processing apparatus 101 includes a display control unit 308, a data management unit 309, a variable control unit 310, a data comparison unit 311, an operation reception unit 312, and a distance calculation unit 313 as functional units.

  The communication control unit 301 is a functional unit that transmits and receives various types of information between the HMD 102 and the infrared camera 104 that can communicate with the information processing apparatus 101. The communication control unit 301 transmits / receives information to / from these apparatuses through the video controller 206, the communication I / F controller 208, the general-purpose bus 209, and the like described above.

  The position / posture specifying unit 302 is a functional unit for specifying the position and posture in the virtual space based on information indicating the position and posture (orientation) in the real space of the HMD 102 acquired from the infrared camera 104. In the present embodiment, since the optical marker 103 is mounted on the HMD 102, the position and orientation of the optical marker 103 in the real space are first acquired. Since the coordinate system of the real space (the coordinate system of the infrared camera 104) and the coordinate system of the virtual space are calibrated (positioned) in advance, a virtual corresponding to the position and orientation in the real space is based on this. Identify the position and posture in space. It should be noted that the real space image acquisition unit 303 uses the right eye real space image and the left eye real space image acquired from the right eye / left eye video camera 221 to determine the position of the object in the real space by a method such as triangulation. The posture may be specified.

  The position / posture specifying unit 302 analyzes the real space image acquired by the real space image acquisition unit 303, specifies the position and shape of the two-dimensional marker that appears in the real space image, and determines the position of the target marker in the real space. It is also possible to specify the posture. The position / posture specifying unit 302 also specifies the position and posture in the virtual space from the position and posture of the target marker in the real space image.

  The real space image acquisition unit 303 is a functional unit that acquires real space images captured by the right-eye / left-eye video camera 221 of the HMD 102.

  The virtual space generation unit 304 is a functional unit that generates a virtual space based on information stored in the external memory 211 of the information processing apparatus 101. Since the virtual space is a virtual space generated inside the information processing apparatus 101, information regarding the shape and size of the space is stored in the external memory 211, and a virtual space is generated based on this information. A three-dimensional model can be placed in the virtual space. The 3D model is arranged by the 3D model control unit 305.

  The 3D model control unit 305 is a functional unit for arranging the 3D model in the virtual space. When the virtual space is generated, a three-dimensional model is arranged on the virtual space at a predefined position and posture. If necessary, the arranged three-dimensional model is rearranged (moved) at a different position and posture. Also, the 3D model control unit 305 can arrange the 3D model in the virtual space based on the position and orientation of the target marker 105 in the virtual space specified by the position / posture specifying unit 302. That is, when a mixed reality image is generated, a three-dimensional model can be arranged so as to be superimposed on the target marker 105. Furthermore, the 3D model control unit 305 can change the display state (display form) of the 3D model arranged in the virtual space. For example, the color of the three-dimensional model can be changed or the shape can be changed.

  The virtual space image acquisition unit 306 is a functional unit that acquires an image of the virtual space generated by the virtual space generation unit 304. When the virtual space image acquisition unit 306 acquires a virtual space image, a viewpoint is set on the virtual space based on the position and posture of the HMD 102 in the virtual space specified by the position / posture specifying unit 302. And the virtual space image at the time of seeing from the said viewpoint is produced | generated, and this is acquired. A virtual camera may be installed at this viewpoint to image the virtual space. That is to render

  The mixed reality image generation unit 307 is a functional unit that generates a mixed reality image by superimposing the virtual space image acquired by the virtual space image acquisition unit 306 on the real space image acquired by the real space image acquisition unit 303.

  The display control unit 308 is a functional unit that performs display control of various types of information on the right-eye / left-eye display 222 of the HMD 102 connected to the information processing apparatus 101 and the display 212 connected to the information processing apparatus 101. In particular, it has a function of displaying the mixed reality image generated by the mixed reality image generation unit 307 on the right-eye / left-eye display 222 of the HMD 102.

  The data management unit 309 is a functional unit for managing data stored in a storage unit such as the RAM 203 or the external memory 211 of the information processing apparatus 101. Add / update / delete data as necessary.

  The variable control unit 310 is a functional unit for controlling a variable used in a program executed by the information processing apparatus 101. Execute processing such as variable declaration and assignment.

  The data comparison unit 311 is a functional unit for comparing a plurality of three-dimensional models stored in the external memory 211 or the like and specifying the difference. Since the three-dimensional model has information on the number of polygons and the position and orientation of the three-dimensional model, whether or not there is a difference is specified by comparison using this information. Further, as in the prior art, comparison by volume or area or comparison by image processing may be used.

  The operation receiving unit 312 is a functional unit for receiving a selection operation for the 3D model arranged by the 3D model control unit 305. Operations on the three-dimensional model arranged in the virtual space are performed using the target marker 105. Since the position and posture of the target marker 105 in the virtual space are specified by the position / posture specifying unit 302, a cursor object is placed on the upper portion of the target marker 105 based on this position and posture. When the cursor object and the 3D model are in contact with each other for a certain time, it is determined that the 3D model has been selected. The selection operation method for the three-dimensional model is not limited to this, and may be selected by the finger of the user wearing the HMD 102, or an operation screen may be displayed on the display 212 of the information processing apparatus 101 and the selection operation may be performed by the input device 210. It may be accepted.

  The distance calculation unit 313 is a functional unit for calculating the distance between two points in the virtual space. The distance between two points in the coordinate system in the virtual space is calculated using the three-square theorem.

  Next, an example of the overall processing flow in the embodiment of the present invention will be described with reference to the flowchart shown in FIG. Each step of S401 to S411 described below is a process executed by the CPU 201 of the information processing apparatus 101 operating each function unit.

  In step S401, the information processing apparatus 101 executes a process for specifying a difference between the assembly before the design change and the assembly after the design change. FIG. 5 shows a detailed processing flow of the difference identification processing.

  FIG. 5 is a flowchart illustrating an example of a detailed process flow of the difference specifying process. Steps S501 to S509 described below are processes executed by the CPU 201 of the information processing apparatus 101 operating each function unit.

  In step S501, the data management unit 309 of the information processing apparatus 101 stores assembly data before design change (hereinafter, referred to as old assembly data) stored in the external memory 211 and assembly data after design change ( Hereinafter referred to as new assembly data). The user stores these data in the external memory 211 in advance. Then, the data management unit 309 acquires old assembly data and new assembly data specified by the user. In the present embodiment, the assembly data is assembly data of CAD data.

  In FIG. 6A, examples of reference structures of old assembly data and new assembly data (hereinafter referred to as new and old assembly data) are shown in 601 and 602, respectively. Since the old and new assembly data is the top assembly, it is composed of a plurality of parts and subassemblies. For example, the old assembly data A is composed of part data B, part data C, and assembly data D, and the assembly data D that is a subassembly is composed of part data E, F, and G. Hereinafter, parts and subassemblies to which assembly data is referred are referred to as reference destination data.

  In step S502, the data management unit 309 of the information processing apparatus 101 acquires information regarding reference destination data that is a reference destination of the new and old assembly data acquired in step S501, and a comparison table 610 as illustrated in FIG. 6B. Create The comparison table 610 includes an old reference destination data name 611 and a new reference destination data name 612, and indicates names of reference destination data of a comparison source and a comparison destination, respectively. If there is reference data of the new assembly data corresponding to the reference data of the old assembly data, store them side by side in the same row, and in the case of reference data that has only one of them, the assembly where the reference data does not exist Leave the item data item blank. FIG. 6B shows the result of creating the comparison table 610 between the old assembly data A and the new assembly data A shown in FIG.

  In step S503, the data management unit 309 of the information processing apparatus 101 generates a difference list 700 as illustrated in FIG. In the process of step S503, an empty difference list 700 is generated.

  The difference list 700 in FIG. 7 is a data table for indicating what kind of difference exists before and after the design change. The difference list 700 includes a difference data name 701, an addition flag 702, a shape change flag 703, a position / orientation change flag 704, and a deletion flag 705. The table configuration of the data table is not limited to this.

  The difference data name 701 is an item in which the name of the reference destination data that has a difference before and after the design change is stored. The difference referred to in the present embodiment refers to a difference before and after the design change. For example, it refers to addition of new reference destination data to the old assembly data, change of the shape of the reference destination data, change of the position or orientation of the reference destination data, and deletion of the reference destination data from the old assembly data. An add flag 702, a shape change flag 703, a position / orientation change flag 704, and a delete flag 705 are items that are flagged when there is a difference between them.

  Returning to the description of FIG. In step S504, the variable control unit 310 of the information processing apparatus 101 declares a counter variable i and substitutes 0 as an initial value. In step S505, the variable control unit 310 of the information processing apparatus 101 increments the variable i.

  In step S506, the data management unit 309 of the information processing apparatus 101 acquires the reference destination data of the i-th row of the records in the comparison table 610 from the external memory 211 and refers to this. Here, the reference destination data constituting the old assembly data indicated by the old reference destination data name 611 and the reference destination data constituting the new assembly data indicated by the new reference destination data name 612 are acquired. When either the old reference destination data name 611 or the new reference destination data name 612 is blank, only the reference destination data in which the name is input is acquired.

  In step S507, the data comparison unit 311 of the information processing apparatus 101 compares the reference destination data constituting the old assembly data acquired in step S506 with the reference destination data constituting the new assembly data. The comparison here refers to whether or not the reference destination data has been newly added, the shape of the reference destination data has been changed, the position or orientation of the reference destination data has been changed, the reference destination data has been deleted, etc., in step S506. It is specified by comparing the reference data acquired in step 1. In other words, the plurality of part data constituting the three-dimensional model before the design change is compared with the plurality of part data constituting the three-dimensional model after the design change, and the part data that has undergone the design change is specified. (Specific means)

  For example, in the case of reference destination data that does not exist in the old assembly data but exists in the new assembly data (when the reference destination data constituting the old assembly data could not be acquired in step S506), a new one is newly added. Identify the added reference data. If the number of polygons included in the reference data of the old assembly data acquired in step S506 does not match the number of polygons included in the reference data of the new assembly data, it is determined that the reference data has been changed in shape. To do. The determination of coincidence / non-coincidence is not limited to the number of polygons, and may be made by comparison using other elements such as volume and area. In addition, if the information indicating the position or posture indicated by the assembly data that is the parent of the reference data of the old assembly data and the assembly data that is the parent of the reference data of the new assembly data do not match, the position or posture Is specified as the reference data that has been changed. In the case of reference destination data that exists in the old assembly data but does not exist in the new assembly data (when the reference destination data constituting the new assembly data cannot be obtained in step S506), the deleted reference is deleted. It is identified as the previous data. If it does not correspond to any of these, it specifies that there was no design change about the acquired reference destination data.

  As a result of the comparison, it is determined whether there is a design change in the reference destination data in the old and new assembly data referred to in step S506, that is, whether there is a difference. If it is determined that there is a difference (addition, shape change, position or orientation change, deletion has occurred), the process proceeds to step S508. If it is determined that there is no difference, the process proceeds to step S509.

  In step S508, the data management unit 309 of the information processing apparatus 101 adds a new record to the difference list 700 generated in step S503, and adds the name of the reference destination data referenced in step S506 to the difference data name 701. . Then, “1” is stored in the flag corresponding to the difference content. That is, set a flag. “0” is stored in the other flags. In this way, information related to reference destination data whose design has been changed is added to the difference list 700. The result of executing step S508 for all the reference destination data is the difference list 700 shown in FIG. Hereinafter, the reference data that has undergone a design change is referred to as difference data.

  In step S509, the data management unit 309 of the information processing apparatus 101 determines whether or not the i-th row of the comparison table 610 is the last row (last record). If it is determined that the current line is the last line, the difference specifying process is terminated, and the process proceeds to the next step of step S401. If it is determined that it is not the last line, the process returns to step S505.

  In this way, a plurality of reference data constituting the old assembly data is compared with a plurality of reference data constituting the new assembly data, and the reference data whose design has been changed and the contents of the change are specified. To do.

  Returning to FIG. When step S401 is completed, in step S402, the virtual space generation unit 304 of the information processing apparatus 101 acquires the virtual space information stored in the external memory 211. The virtual space information is various pieces of information for generating a virtual space. Specifically, it is information necessary for forming the virtual space, such as the shape and size of the virtual space, the three-dimensional model to be arranged, and the arrangement location thereof. Then, the virtual space generation unit 304 generates a virtual space using the acquired virtual space information. The generated virtual space is a virtual space having the shape and size indicated by the virtual space information, and a three-dimensional model indicated by the virtual space information is arranged on the virtual space. In the present embodiment, the virtual space generation unit 304 places the new assembly data acquired in step S501 in the virtual space (placement means). Since the new assembly data is assembly data as described above, the reference destination data is displayed as a three-dimensional model and arranged at the position and orientation defined in the assembly data. As a result, the three-dimensional model indicating the new assembly data is arranged in the virtual space.

  In the present embodiment, the new assembly data is arranged in the virtual space, and the display control and selection reception for the component data constituting the new assembly data will be described. However, the old assembly data is stored in the virtual space. You may arrange. In this case, display control and selection for the component data constituting the old assembly data are performed.

  FIG. 8 is a diagram illustrating a state in which a three-dimensional model 800 of new assembly data is arranged in front of the user wearing the HMD 102. Since the user wearing the HMD 102 exists in the real space and the three-dimensional model 800 of the new assembly data is arranged in the virtual space, they do not exist in the same space when viewed objectively. When viewed, the state is as shown in FIG. In order to explain the positional relationship between the user and the three-dimensional model 800 in an easy-to-understand manner, the following description will be given with reference to FIGS.

  In step S <b> 403, the real space image acquisition unit 303 of the information processing apparatus 101 acquires real space images from the right-eye / left-eye video camera 221 of the HMD 102 and stores them in the RAM 203.

  In step S <b> 404, the position / posture specifying unit 302 of the information processing apparatus 101 acquires information indicating the position and posture of the HMD 102 in the real space. As described above, information indicating the position and orientation specified by the infrared camera 104 detecting the optical marker 103 included in the HMD 102 is acquired from the infrared camera 104. Then, the position / posture specifying unit 302 specifies the position and posture in the virtual space from the information indicating the acquired position and posture. The specified position and orientation are stored in the RAM 203 as HMD position / posture information. Since this information is the position and orientation of the HMD 102 in the current frame, this information is updated each time the frame is switched.

  In step S <b> 405, the position / posture specifying unit 302 of the information processing apparatus 101 specifies information indicating the position and posture of the target marker 105 in the real space. Specifically, as described above, the target marker 105 is detected from the real space image captured by the HMD 102. Then, the position and orientation of the target marker 105 in the real space are estimated based on the shape of the target marker 105 shown in the real space image and the parallax specified from the two real space images acquired by the right eye / left eye video camera 221. Then, the position and orientation in the virtual space are specified from the position and orientation in the real space. In other words, the position in the virtual space corresponding to the position designated by the user wearing the HMD 102 in the real space is specified. The specified position and orientation are stored in the RAM 203 as target marker position / posture information. Since this information is the position and orientation of the HMD 102 in the current frame, this information is updated each time the frame is switched.

  In step S406, the information processing apparatus 101 executes processing for identifying and displaying a three-dimensional model of difference data existing in the vicinity of the user. FIG. 9 shows a detailed processing flow of the differential data identification processing.

  FIG. 9 is a flowchart illustrating an example of a detailed process flow of the differential data identification process. Steps S901 to S909 described below are processes executed by the CPU 201 of the information processing apparatus 101 operating each function unit.

  In step S901, the variable control unit 310 of the information processing apparatus 101 declares a counter variable i and substitutes 0 as an initial value. In step S902, the variable control unit 310 of the information processing apparatus 101 increments the variable i.

  In step S903, the data management unit 309 of the information processing apparatus 101 refers to the difference data of the record in the i-th row among the records in the difference list 700. Reference here means specifying as a processing target. Therefore, hereinafter, this difference data is referred to as reference difference data.

  In step S904, the position / posture specifying unit 302 of the information processing apparatus 101 acquires the display position of the three-dimensional model indicated by the reference difference data. The display position to be acquired is the coordinate in the world coordinate system of the center point of the three-dimensional model arranged in the virtual space. Although the center point is used in the present embodiment, the coordinates of the vertices of the bounding box surrounding the three-dimensional model may be acquired, or the coordinates of any other point may be acquired.

  In step S905, the distance calculation unit 313 of the information processing apparatus 101 calculates the distance from the position of the HMD 102 in the virtual space acquired in step S404 to the position acquired in step S904, and acquires this. That is, the distance from the user's viewpoint in the virtual space to the three-dimensional model indicated by the reference difference data is acquired.

  In step S906, the distance calculation unit 313 of the information processing apparatus 101 determines whether the acquired distance is equal to or less than a predetermined distance (or smaller than the predetermined distance). That is, it is determined whether or not a three-dimensional model of reference difference data exists within a predetermined range from the current position of the user in the virtual space. Or you may determine whether the acquired distance is not more than predetermined distance (or larger than predetermined distance). By extracting only the three-dimensional model of the difference data that exists in the vicinity of the user, even if there are a large number of difference data, only the difference data that you want to check is identified and displayed without impairing the appearance of the assembly. It becomes possible.

  As shown in FIG. 10A, the current position in the virtual space of the HMD 102, that is, the distance from the current position in the user's virtual space to the three-dimensional model of the reference difference data is calculated. Extract what is within the distance. For example, since the wheel of the car 1002 is a three-dimensional model of difference data, the distance from the user's viewpoint in the virtual space to the center point of the three-dimensional model is calculated. Then, it is determined whether or not the calculated distance is equal to or less than a predetermined distance, that is, whether or not the calculated distance exists within a range of 1000 spheres having a radius of 1001 which is the predetermined distance. In the example of FIG. 10A, it is determined that the wheel of the car that is the difference data is included in this range. Although 1003, 1004, and 1005 are also differential data, among them, the door mirror 1003 is within the predetermined range in FIG. 10A, but it is determined that the wing 1004 and the muffler 1005 are out of the predetermined range. In this way, a three-dimensional model of difference data existing in the vicinity of the user is extracted.

  As a modified example of step S906, a method of extracting difference data existing in the vicinity of the user in consideration of the user's visual field can be considered. As shown in FIG. 10B, the range of the visual field of the user in the virtual space is specified using the position and orientation of the HMD 102 (user) in the virtual space acquired in step S404. A range as indicated by 1010 is specified by combining the range of the visual field and the range of the predetermined distance. In FIG. 10B, 1011 is the predetermined distance, and 1010 is the range of the visual field of the user that falls within the predetermined distance. In this way, by extracting the difference data that exists within a predetermined distance within the user's viewable range, the difference data in the range that cannot be browsed by the user will not be identified and displayed in the process described later. The processing load on the information processing apparatus 101 can be reduced.

  Returning to FIG. In step S907, the data management unit 309 of the information processing apparatus 101 refers to the state of the reference difference data flag. That is, the states of the four flags from the addition flag 702 to the deletion flag 705 corresponding to the reference difference data are referred to.

  In step S908, the three-dimensional model control unit 305 of the information processing apparatus 101 determines what display state the three-dimensional model of the reference difference data has to be displayed according to the state of the flag referred to in step S907. Then, the determined display state is set for the three-dimensional model of the reference difference data. In other words, the display state of the specified component data that exists within a predetermined range from the current position of the user in the virtual space is set to a state that can be identified in the arranged three-dimensional model (setting unit). ). In FIG. 10A, since the wheel portion 1002 and the door mirror portion 1003 are three-dimensional models of difference data within a predetermined range, these are displayed so as to be distinguishable from other component data.

  For example, when “1” is stored in the addition flag 702, the three-dimensional model of the reference difference data is colored purple, and when “1” is stored in the shape change flag 703, the reference difference is stored. Color the three-dimensional model of the data blue. Further, when “1” is stored in the position / orientation change flag 704, the three-dimensional model of the reference difference data is colored yellow, and when “1” is stored in the deletion flag 705, it is referred to. The three-dimensional model of difference data is colored in red. Further, when “1” is stored in each of the shape change flag 703 and the position / orientation change flag 704, the three-dimensional model of the reference difference data is colored green, which is a color obtained by combining blue and yellow. . In this way, by determining the display state according to the contents of the design change, it is possible to make the user recognize what kind of change has been made.

  If a three-dimensional model of new assembly data is arranged in the virtual space, the deleted difference data is not displayed. Therefore, the edge of the difference data is represented by a broken line or the like at the position where the difference data originally exists, and the edge is colored to allow the user to recognize that the difference data has been deleted. Alternatively, if the three-dimensional model of the old assembly data is arranged in the virtual space, the added difference data is not displayed. Therefore, the edge of the difference data is expressed by a broken line or the like at the position where the difference data is added, and the edge is colored to allow the user to recognize that the difference data has been added.

  Further, the setting of the display state of the reference difference data is not limited to the color. For example, the design change location may be recognized by the user by changing the brightness or blinking.

  In step S909, the data management unit 309 of the information processing apparatus 101 determines whether or not the i-th row of the difference list 700 is the last row (last record). If it is determined that the current line is the last line, the difference data identification process is terminated, and the process proceeds to the next step of step S406. If it is determined that it is not the last line, the process returns to step S902. In this way, the processing from step S902 to step S909 is executed for all the difference data.

  Returning to FIG. In step S <b> 407, the information processing apparatus 101 executes a process for placing the three-dimensional model of difference data selected by the user on the target marker 105. FIG. 11 shows a detailed processing flow of the differential data arrangement processing.

  FIG. 11 is a flowchart illustrating an example of a detailed process flow of the differential data arrangement process. Each step of S1101 to S1112 described below is a process executed by the CPU 201 of the information processing apparatus 101 operating each function unit.

  In step S <b> 1101, the operation reception unit 312 of the information processing apparatus 101 specifies the operation content from the user using the target marker 105. In the present embodiment, the operation of the user wearing the HMD 102 is performed using the target marker 105. A cursor object is superimposed and displayed in the direction indicated by the target marker 105, and a selection operation is performed using this cursor object.

  In step S1102, the operation accepting unit 312 of the information processing apparatus 101 identifies the difference data in which the operation content specified in step S1101 is identified and displayed among the difference data constituting the new assembly data arranged in the virtual space. It is determined whether or not this is a selection operation. Specifically, when the cursor object superimposed on the target marker 105 is in contact with the three-dimensional model of the difference data being identified and displayed for a certain period of time, it is determined that the selection operation for the difference data has been performed (selection acceptance) means). The selection method is not limited to this, and a screen including a list of difference data may be displayed to accept a selection operation on the screen. If it is determined that the selection operation is for the differentially displayed differential data, the process proceeds to step S1103. Otherwise, the process proceeds to step S1108.

  In step S <b> 1103, the virtual space generation unit 304 of the information processing apparatus 101 determines whether any difference data is being displayed according to the position / posture of the target marker 105. If a three-dimensional model of difference data different from the selected difference data is displayed superimposed on the target marker 105, a plurality of difference data will be displayed superimposed and difficult to view unless this display is finished. Therefore, such a determination is made. If it is determined that any difference data is being displayed, the process advances to step S1104. If it is determined that none of the difference data is being displayed, the process proceeds to step S1105.

  In step S <b> 1104, the virtual space generation unit 304 of the information processing apparatus 101 hides the three-dimensional model of difference data that is superimposed and displayed on the target marker 105. Since it is only necessary for the user to be invisible, the three-dimensional model of the difference data arranged at the position of the target marker 105 may be removed, or the transparency may be set to 100% and the invisible state may be set.

  In step S1105, the data management unit 309 of the information processing apparatus 101 determines whether the shape change flag 703 of the selected difference data is “1”. That is, it is determined whether or not the design change content of the selected difference data is a shape change. If it is determined that the shape change flag 703 is “1”, the process proceeds to step S1106. If not, that is, if it is determined that the shape change flag 703 is “0”, the process proceeds to step S1107.

  In step S <b> 1106, the 3D model control unit 305 of the information processing apparatus 101 displays the 3D model before and after the design change of the selected difference data superimposed on the target marker 105 according to the position and orientation of the target marker 105. So as to be transparent. That is, the three-dimensional model of the difference data before and after the design change is superimposed and displayed transparently. In this way, it becomes easy to confirm what shape the shape has been changed before and after the design change. For example, as shown in FIG. 12A, when a cursor object as shown at 1201 is brought into contact with a wheel that is identified and displayed for a certain period of time, a three-dimensional model before and after the design change of the wheel is shown in FIG. ) 1202 and are transmitted through each other. The result of this arrangement is 1202 in FIG. In the present embodiment, the image is transmitted and superimposed, but it may be configured such that one before and after the design change is transmitted and the other is not transmitted. Further, the difference data before and after the design change may be transmitted and colored with different colors.

  In step S <b> 1107, the 3D model control unit 305 of the information processing apparatus 101 superimposes the 3D model before or after the design change of the selected difference data on the target marker 105 according to the position and orientation of the target marker 105. Arrange it so that it is visible. When the addition flag 702 is “1” difference data, the difference data after the design change is arranged. When the deletion flag 705 is “1” difference data, the difference data before the design change is arranged. For the difference data with the position / orientation change flag 704 of “1”, any data before and after the design change may be arranged. When the process of step S1106 or step S1107 is completed, the differential data arrangement process is terminated, and the process proceeds to the next step of step S407.

  In step S1108, the operation reception unit 312 of the information processing apparatus 101 determines whether an instruction to switch the display state of the difference data before and after the design change transparently displayed in step S1106 has been received. This switching instruction determines that there has been a switching instruction by receiving again a selection operation for the three-dimensional model of the difference data. Not limited to this, it may be determined that there is a switching instruction when the target marker 105 is moved along a predetermined locus, or it is determined that there is a switching instruction by accepting a press on a physical or virtual button. Also good. If it is determined that a switching instruction has been received, the process proceeds to step S1109. Otherwise, the difference data placement process is terminated, and the process proceeds to the next step after step S407.

  In step S1109, the data management unit 309 of the information processing apparatus 101 determines whether or not the shape change flag 703 of the difference data for which a switching instruction has been issued is “1”. That is, it is determined whether or not the design change content of the difference data for which a switching instruction is given is a shape change. If it is determined that the shape change flag 703 is “1”, the process proceeds to step S1110. If not, that is, if it is determined that the shape change flag 703 is “0”, the difference data arrangement process is terminated, and the process proceeds to the next step of step S407.

  In step S1110, the 3D model control unit 305 of the information processing apparatus 101 determines whether or not the 3D model of the difference data superimposed on the target marker 105 is being transparently displayed. What is necessary is just to confirm the transparency of the said three-dimensional model. If it is determined that transparent display is being performed, the process proceeds to step S1112. If it is determined that the transparent display is not being performed, the process proceeds to step S1111.

  In step S <b> 1111, the 3D model control unit 305 of the information processing apparatus 101 displays the 3D model before and after the design change of the selected difference data superimposed on the target marker 105 according to the position and orientation of the target marker 105. So as to be transparent. That is, the same processing as step S1106 is performed.

  On the other hand, in step S <b> 1112, the virtual space generation unit 304 of the information processing apparatus 101 superimposes and displays the three-dimensional model before and after the design change of the selected difference data on the target marker 105 according to the position and orientation of the target marker 105. To be arranged. This is arranged without transmitting. In addition, three-dimensional models of difference data before and after the design change are arranged side by side. For example, as shown in FIG. 13, the three-dimensional model of the difference data before the design change and the three-dimensional model of the difference data after the design change are displayed side by side without being transparent, as indicated by 1301 and 1302. In this way, it is possible to help the user confirm the difference by displaying the user before and after the design change for easy comparison. In this embodiment, it is arranged without transmitting, but it may be transmitted. However, since the visibility is lowered, it is desirable not to transmit.

  When the process of step S1111 or step S1112 is completed, the difference data arrangement process is terminated, and the process proceeds to the next step of step S407.

  Returning to FIG. In step S408, the virtual space image acquisition unit 306 of the information processing apparatus 101 acquires a virtual space image by imaging the virtual space from the viewpoint set on the virtual space, and stores this in the RAM 203 or the like. Note that two images, a right-eye virtual space image and a left-eye virtual space image, are acquired for display on the right-eye and left-eye displays 222 of the HMD 102.

  In step S409, the mixed reality image generation unit 307 of the information processing apparatus 101 reads the real space image acquired in step S403 and the virtual space image acquired in step S408 from the RAM 203 or the like. Then, the virtual space image is superimposed on the real space image to generate a mixed reality image. The generated mixed reality image is stored in the RAM 203 or the like. As described above, since the real space image and the virtual space image are stored in the RAM 203 or the like for each of the right eye and the left eye, the virtual space image for the right eye is superimposed on the real space image for the right eye. The virtual space image for the left eye is superimposed on the real space image for the left eye.

  In step S410, the display control unit 308 of the information processing apparatus 101 reads the mixed reality image generated in step S409 from the RAM 203 or the like and displays the mixed reality image on the right-eye / left-eye display 222 of the HMD 102 through the video controller 206. The mixed reality image stored in the RAM 203 or the like has two sheets for the right eye and the left eye. Therefore, control is performed so that the mixed reality image for the right eye is displayed on the right eye display of the right eye / left eye display 222, and the mixed reality image for the left eye is displayed on the left eye display of the right eye / left eye display 222.

  In step S411, the operation reception unit 312 of the information processing apparatus 101 determines whether there is an instruction to end the process of presenting mixed reality to the user wearing the HMD 102. For example, it is determined whether or not there has been an instruction to stop or end the application of the information processing apparatus 101 that executes the processes in steps S401 to S411 described above. If it is determined that an end instruction has been issued, the series of processes ends. If it is not determined that there is an end instruction, that is, if there is no end instruction, the process returns to step S403, and the processes of steps S403 to S411 are repeated until an end instruction is issued.

  As described above, even when a plurality of part data whose design has been changed is included in the three-dimensional model, the part data can be easily viewed in the virtual space.

  The present invention can be implemented as a system, apparatus, method, program, storage medium, or the like, and can be applied to a system including a plurality of devices. You may apply to the apparatus which consists of one apparatus.

  Note that the present invention includes a software program that implements the functions of the above-described embodiments directly or remotely from a system or apparatus. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Therefore, in order to realize (executable) the functional processing of the present invention by a computer, the program code itself installed in the computer also realizes the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let me. It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

101 Information processing apparatus 102 HMD
103 Optical marker 104 Infrared camera 105 Target marker

Claims (7)

A virtual space control device for presenting a virtual space in which a three-dimensional model composed of a plurality of component data is arranged,
A specifying means for comparing the plurality of part data constituting the three-dimensional model before the design change with the plurality of part data constituting the three-dimensional model after the design change, and identifying the part data on which the design change has been made; ,
Arrangement means for arranging the three-dimensional model in the virtual space;
Setting means for setting a display state of the component data specified by the specifying means existing within a predetermined range from the current position of the user in the virtual space to a state identifiable in the three-dimensional model arranged by the arranging means; A virtual space control device comprising:   The virtual space control apparatus according to claim 1, wherein the setting unit sets a display state corresponding to the content of the design change for the component data specified by the specifying unit.   The setting means displays the display state of the component data specified by the specifying means that exists within a predetermined range from the current position of the user in the virtual space and that can be browsed by the user. The virtual space control device according to claim 1, wherein the virtual space control device is set in an identifiable state in the three-dimensional model arranged in (1). A selection receiving means for receiving a selection for the component data for which the display state is set by the setting means;
4. The virtual space control device according to claim 1, wherein the arrangement unit arranges, in the virtual space, component data for which selection has been received by the selection reception unit before and after a design change. 5.   5. The virtual space control device according to claim 4, wherein the arrangement unit arranges the component data received by the selection reception unit before and after the design change in the virtual space in a superimposed manner. A control method of a virtual space control device for presenting a virtual space in which a three-dimensional model composed of a plurality of component data is arranged,
The identification unit of the virtual space control device compares a plurality of part data constituting the three-dimensional model before the design change with a plurality of part data constituting the three-dimensional model after the design change, and the design change Specific steps to identify the part data made,
An arrangement step of arranging the three-dimensional model in the virtual space by an arrangement unit of the virtual space control device;
In the three-dimensional model arranged in the arrangement step, the setting unit of the virtual space control device displays the display state of the part data identified in the identification step existing within a predetermined range from the current position of the user in the virtual space. A control method for a virtual space control device, comprising: a setting step for setting to an identifiable state. A program capable of executing a control method of a virtual space control device for presenting a virtual space in which a three-dimensional model composed of a plurality of component data is arranged,
The virtual space control device;
A specifying means for comparing the plurality of part data constituting the three-dimensional model before the design change with the plurality of part data constituting the three-dimensional model after the design change, and identifying the part data on which the design change has been made; ,
Arrangement means for arranging the three-dimensional model in the virtual space;
Setting means for setting the display state of the component data specified by the specifying means existing within a predetermined range from the current position of the user in the virtual space to a state that can be identified in the three-dimensional model arranged by the arranging means. A program characterized by functioning.

Images