JP2006293605A - Information processing method and system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that the viewpoint of an instructor is restricted to the viewpoint of an operator in a system through which the remote instructor assists in operation by using video obtained by photographing an operator mixed reality space by an operator viewpoint camera. <P>SOLUTION: This information processing method includes a mixed reality space image acquisition step of acquiring a mixed reality space image based upon video of a first imaging part mounted on the operator and a virtual body image based upon the position attitude of a first imaging part, an event information acquisition step of acquiring event information on a virtual body that the operator operates, a second user specified position attitude information acquisition step of acquiring viewpoint position attitude information of the instructor, and a generation step of generating the virtual body image corresponding to the event information based upon the viewpoint position attitude information of the instructor, and has a first mode wherein the first image is presented to the instructor and a second mode wherein the virtual body image is presented to the indicator. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a case where a plurality of users in different places share a mixed reality space and perform cooperative work.

  In recent years, technological development relating to mixed reality (MR) aimed at seamless integration of the real world and the virtual world has become active. MR is attracting attention as a technology for enhancing VR for the purpose of coexistence between the virtual reality (VR) world and the real world, which can only be experienced in a situation separated from the real space.

  A typical device that realizes mixed reality is a head-mounted display device (HMD = Head Mounted Display). That is, the mixed reality is realized by combining the real space and the virtual space with the HMD and displaying it.

  In a system that supports conferences and various collaborative work among remote participants using VR, a world separated from the real world is built and shared in a computer as described above, whereas remote in MR The work support system can support work in a work space in which a virtual world is superimposed based on the real world.

  There has been proposed an example of a technique capable of observing / manipulating a stereoscopic image of a participant photographed by a stereo camera together with a virtual object shared between the participants using a head-mounted display device (HMD). For example, a stereoscopic video in which a participant is photographed with two cameras and only a person portion is cut out can be transmitted over a network, and the other multiple participants can observe the stereoscopic video. The virtual object can be operated because the position of the worker and the position of the hand for virtual object operation are measured, but the stereo image that the participant is observing is the image of the participant with the background removed The VR space is an image of the virtual object and does not have information on the real space of the participant.

An example of a technique for supporting remote work based on an operator's HMD video has been proposed (Patent Document 1). In this example, there is a technique in which an operator wears an HMD with a camera, shares an image from the camera with the operator and a remote instructor, and superimposes the instructor's instruction on the camera image to instruct the operation. It is disclosed. The operator's camera can be set in stereo so that both the operator and the instructor can observe the same stereoscopic image. The hand of the instructor indicating the work object in the image is cut out by the chroma key composition and synthesized with the image from the camera. The technology is described. In this system, since coordinates are not set in the worker's space, a virtual object other than the work target cannot be placed at any position in the worker space for pointing or interaction. Further, when the viewpoint position of the operator moves, there is a problem that the image of the hand moves even if the instructor does not move the hand.
JP 2002-132487 A

  As described above, according to the prior art, the instruction point of view is restricted to the worker's point of view, so that the work instruction cannot be performed smoothly.

  It is an object of the present invention to solve the above-described problems and provide a work instruction smoothly by providing a mode in which the instructor's viewpoint is not constrained by the operator's viewpoint.

  In order to achieve the above-described object, the present invention has the following configuration.

  The information processing method according to claim 1 is an information processing method in which a second user shares a mixed reality space image in which a virtual object is superimposed and displayed in a space in which the first user exists. A mixed reality space image acquisition step of acquiring a mixed reality space image based on a video of a first imaging unit worn by a user and a virtual object image based on a position and orientation of the first imaging unit; and the first user Event information acquisition step for acquiring event information for the virtual object, a second user-specified position and orientation information acquisition step for acquiring viewpoint position and orientation information of the second user, and a viewpoint of the second user Generating a virtual object image corresponding to the event information based on the position and orientation information, a first mode for presenting the first image to the second user, and the virtual object image And having a second mode of presenting the serial second user.

  The invention according to claim 4 is an information processing system for remotely sharing a worker mixed reality space in which a virtual object is superimposed and displayed in a space where a worker exists, and the worker mixed reality from the viewpoint of the worker. Stereo video transmission means for transmitting a stereo image of space to the instructor, virtual object sharing means for sharing the virtual object between the operator and the instructor, and stereoscopic means for enabling the operator and the instructor to stereoscopically view the stereo video The instructor has a mode for sharing a stereo image from the worker viewpoint and a mode for operating the virtual object from a viewpoint independent of the worker viewpoint.

  According to the present invention, smooth work support is possible, and the convenience of cooperative work between remote locations can be improved.

(First embodiment)
Hereinafter, with reference to an accompanying drawing, it explains in detail according to an embodiment.

  FIG. 1 shows a functional configuration of the remote mixed reality sharing system in the present embodiment. The configuration is composed of an operator mixed reality apparatus 10a shown in the upper part of FIG. 1 and an instructor mixed reality apparatus 10b shown in the lower part of FIG. 1, which are connected via a management server 60 via a network. . Each device has a head-mounted device (HMD = Head Mounted Display) 20a, 20b, and both the worker and the instructor can see a mixed reality image of the space where the worker is located through the HMD. Further, microphones 28a and 28b and speakers 27a and 27b for carrying out a conversation between the two are provided.

  The worker HMD 20a includes a three-dimensional position and orientation sensor 21a that measures the position and orientation of the HMD using magnetism, and a stereo camera 22 that can capture a real space and obtain an image (L for the left eye and L for the right eye). R), and a stereo display 23a (L for the left eye and R for the right eye) that can display an image. The three-dimensional position and orientation sensor 21a and the camera 22 Assume that the positional relationship is measured and fixed in advance. A position / orientation signal as a measurement result is output from the three-dimensional position / orientation sensor 21a to a position / orientation measurement unit 11a described later, and an imaging result is output from the camera 22 to an image input unit 12 described later. The display device 23a receives the image signals for the left eye and the right eye as input from the image composition unit 15, and displays the images on the display device 23aR for the left eye and the right eye, respectively.

  The position / orientation measurement unit 11a in the mixed reality apparatus 10a for the worker includes a three-dimensional position / orientation signal output from the three-dimensional position / orientation sensor 21a of the HMD 20a and a three-dimensional stylus that is a three-dimensional pointing device in the mixed reality space. In response to the position and orientation signal, these data are output to the virtual object manager 16a. The virtual object management unit 16a receives worker stylus and HMD position / orientation data from the position / orientation control unit 11a and accumulates them together with data of all virtual objects used in the work. The virtual object transmission / reception unit 31a transmits / receives all virtual object information shared with the instructor mixed reality apparatus 10b to / from the management server 60. The event information for the virtual object received from the management server 60 is sent to the virtual object management unit 16a, and the scene graph of the virtual object stored therein is changed. In addition, the image generation unit 13 creates virtual spaces CG for the left eye and the right eye as viewed from the worker viewpoint information (HMD position / orientation information) of the scene graphs of all virtual objects stored in the virtual object management unit 10a. Draw and generate an image.

  Reference numeral 15 denotes an image synthesis unit, which inputs captured images from the left-eye camera 22L and the right-eye camera 22R through the image input unit 12, and generates the left-eye and right-eye images generated by the image generation unit 13a on the input image. It combines with the virtual space CG image for eyes. As a result, in the display device for the left eye and the right eye of the display device 23a, the virtual space data from the operator's viewpoint is superimposed and displayed on the image captured by the camera. The superimposed image is compressed and encoded by the video encoding unit 32a and transmitted to the management server 60 through the video transmission unit 33a.

  The configuration of the mixed reality apparatus 10b for the instructor is almost the same as that of the mixed reality apparatus 10a for the worker, except that the HMD 20b is not equipped with a camera and no image is input from the camera. As for the mixed reality video in the worker space, a stereo video from the camera of the worker HMD is received by the video receiving unit 33b via the management server 60, decoded by the video decoding unit 32b, and 23bL for the left eye of the display 23b. Are displayed on the display device for the right eye 23bR. The position / posture of the instructor's HMD is acquired from the three-dimensional position / posture sensor 21b, is input to the position / posture measurement unit 11b together with the three-dimensional position / posture of the stylus 41b, and is sent to the virtual object management unit 16b. Similar to the virtual object management unit 16a of the worker mixed reality device 10b, the virtual object 16b stores all the virtual object data shared between the worker instructors. An event for the virtual object from the instructor side is sent to the management server 60 through the virtual object transmission / reception unit 31b, and an event to the virtual object sent from the management server 60 is sent to the virtual object management unit 16b through this event. Changes are made. Since the function of the image generation unit 13b is the same as 13a, a description thereof will be omitted. The image output unit 17 selectively switches between the worker mixed reality video sent from the video encoding unit 32b and the virtual object CG image sent from the image generation unit 13b, and outputs it to the display 23b.

  Further, after the connection between the mixed reality device for the worker and the mixed reality device for the instructor is established, the video communication module and the audio communication module are activated, and the worker mixed reality space video is transmitted from the video transmission unit 33a to the video reception unit. One-way communication is transmitted to 33b, and bidirectional audio data is exchanged as audio information. As a result, the operator can talk using the 27a speaker and the 27b microphone, and the instructor can talk using the 28a speaker and the 28b microphone.

  Reference numeral 60 denotes a management server that manages virtual object information. The virtual object communication management unit 61 performs information communication management exchanged between the mixed reality device for the worker and the mixed reality device for the instructor. The virtual object management unit 62 manages the scene graph information of the shared virtual object including the operator and the instructor stylus and the HMD and the operation right of the shared virtual object. All changes in the virtual object information from the operator or the instructor are transmitted to the virtual object management unit 62 as events, and after changing the shared scene graph, the instructor mixed reality apparatus 10a and the mixed reality for the instructor are performed. The same event is distributed to the sensing device 10b, and the virtual object scene graph stored in each device is changed.

  An operation example with the above-described configuration will be described with reference to FIG. FIG. 2A shows a situation where the worker 40 wearing the HMD works in the mixed reality space. 42 is a real work object, and 43 is a stereoscopic display of CG which is a 3D model of the work object. The virtual object 43 is shared with the instructor, and work instructions from the instructor and examples of work are performed through the virtual object 43. A world coordinate system (x, y, z) as shown in the figure is set for the real space where the worker is present, and the virtual object 43 as an exemplary model placed in this coordinate system is real when viewed through the HMD. It is displayed in a positional relationship as if it was placed next to the work object 42. By using a stylus equipped with a magnetic sensor, an interaction in which a part of these virtual objects is selected and moved is possible. Reference numeral 41a denotes a pointer in which CG is superimposed on the worker's stylus, and 41b denotes the stylus of the instructor in CG.

  FIG. 2 (2) shows the space of the remote instructor. The instructor 50 is wearing the HMD 20b, and the display of the HMD 20b displays the work object 42 and the image of the 3D model CG 43 in a stereoscopic view as shown by 23b. This is exactly the same image that the worker 40 is viewing. 41a is a pointer indicating the position of the worker's stylus, and 41b is a pointer indicating the position of the stylus of the instructor. In the space where the instructor exists, the instructor world coordinate system (x, y, z) corresponding to the world coordinate system of the worker mixed reality space is set as shown in FIG. In these two world coordinate systems, the position and orientation of the virtual object are represented by coordinate values common to the operator / instructor. Since the instructor's HMD 20b and stylus 41b are equipped with magnetic sensors, the relative positional relationship between the viewpoint position and the stylus held in the hand can be measured. Since this positional relationship can be converted into a positional relationship from the worker's viewpoint to determine the position of the stylus of the instructor from the worker's viewpoint, the instructor feels like operating the pointer from the worker's viewpoint. Can be given. The state in which the worker viewpoint mixed reality video is shared between the worker / instructor and the pointer of the instructor is displayed based on the worker viewpoint position is referred to as a workspace mode.

  The instructor 50 instructs the worker by pointing the virtual object 43 with the pointer 41b while moving the virtual object 43 while viewing the same video 23b as the worker.

  The movement of the virtual object moves the stylus 41b to a position where it touches the virtual object, and presses the first button 47 of the stylus shown in FIG. Thus, the virtual object gripping mode is set, and the gripped virtual object moves together with the stylus. When the stylus button is pressed in the grip mode, the grip mode ends and the virtual object is released. The operator can also operate the virtual object by the same operation. However, the operator's stylus has only one button (not shown). Only one of them can enter the gripping mode and move the virtual object so that the operations of the operator and the instructor do not occur simultaneously.

  In this worker space mode, the relative position relationship between the head position of the indicator and the stylus held in the hand is determined from the position and orientation of the operator's head as the position where the pointer of the indicator is displayed in the mixed reality space of the operator. By converting and displaying the relative positional relationship, you can feel as if you are pointing and extending your hand from your point of view, but if the operator changes the position and orientation of the head, the pointer changes accordingly. The position of will also move. For this reason, there is a possibility that the pointer moves to a part not intended by the instructor. Therefore, when the instructor presses the stylus second button 48 in FIG. 3, a virtual space (FIG. 4B) composed only of the virtual object from the instructor's viewpoint is displayed on the display 23b. In this example, a virtual space composed of the virtual object 43, the operator pointer 41a, and the instructor pointer 41b is displayed from the instructor viewpoint. While viewing this space, operations such as pointing and moving to the virtual object are performed, and the result is reflected in the mixed reality apparatus for the worker, so that the virtual object can be shared with the worker. That is, the instructor can see and operate the virtual object from his / her viewpoint regardless of the movement of the operator's head. This state is referred to as a shared virtual mode. At this time, the worker is looking at the mixed reality space on which the virtual object changed by the instructor is superimposed.

  Further, when the instructor presses the stylus second button 48 in this shared virtual mode, the video in the single virtual mode shown in FIG. 4 (3) is displayed. In this state, the virtual object 43 is not shared with the worker, and only the pointer and the instructor pointer 41b are displayed. In this state, the pointing of the instructor and the change to the virtual object are performed only by the instructor mixed reality device and are not reflected in the worker mixed reality device. The instructor can try and operate the virtual object alone. When the instructor presses the stylus second button in this mode, the virtual object edited in the single virtual mode is uploaded to the management server, and further downloaded to the worker mixed reality device, and the result is reflected in FIG. A video like (1) is displayed on the instructor HMD display device, and the operation mode is restored to the work space mode described first. As described above, the instructor can perform an efficient work instruction by appropriately switching the mode according to the instruction content.

  FIG. 5 is a flowchart for explaining the operation of the mixed reality apparatus 10a for workers for performing work in the mixed reality workspace. Although not shown in FIG. 2, it is assumed that the present apparatus 10 a is in a position where it can be connected to the worker's HMD 20 a and stylus 41 a and is connected to the instructor mixed reality apparatus 10 b through the management server 60 via a network.

  In step S100, the worker mixed reality apparatus 10a is initialized. In this step, the world coordinate system of the mixed reality workspace is set. An output from a sensor, which will be described later, is expressed as data (x, y, z, α, β, γ) as a set of six parameters in this coordinate system. Here, α represents a rotation angle about the x axis, β represents a rotation angle about the y axis, and γ represents a rotation angle about the z axis. In addition, initial data of the virtual object 43 serving as a reference for the real object 42 is arranged in the world coordinate system and is stored in the virtual object management unit 16a as scene graph data.

  In the next step S110, a network connection is established with the management server 60 so that both devices can transmit and receive data, and the virtual object information set in step S100 is transmitted via the virtual object transmitting / receiving unit 31a. Upload to the management server.

  In step S120, a voice communication connection is started between the worker mixed reality device 10a and the instructor mixed reality device 10b. After the voice communication connection is set, voice output is performed from the speakers 27a and 27b, and voice input is performed from the microphones 28a and 28b, respectively. This enables voice conversation between the worker and the instructor. Also, a connection for video communication is set, and video transmission from the video transmission unit 33a to the video reception unit 33b is enabled.

  In step S130, processing is performed to capture the video from the camera 22 attached to the worker HMD into the image composition unit 15 via the image input unit 12. The camera 22 includes two left-eye (L) and right-eye (R) cameras corresponding to the eyes of the worker, and each video is stored in a separate buffer of the image composition unit 15.

  In the next step S140, the position and orientation of the operator's head is input to the position and orientation measurement unit 11a as a value from the HMD three-dimensional position sensor 21a, and data as a set of six parameters in the world coordinate system is generated.

  In the next step S150, the three-dimensional position / orientation information of the worker's stylus 41a is input to the position / orientation measurement unit 11a in the same format as the data from the HMD three-dimensional position sensor and is held in the virtual object management unit 16a.

  In step S160, the worker HMD and the stylus position and orientation information obtained in step S150 are transmitted as events to the management server 60 through the virtual object transmitting / receiving unit 31b.

  In step S170, it is checked whether the operator stylus button is pressed. If the button is pressed, the process proceeds to step S180. Otherwise, the process proceeds to step S190.

  In step S180, the pressed button event is transmitted to the management server 60 through the virtual object transmitting / receiving unit 31a.

  In step S190, information regarding the change of the worker pointer, the pointer pointer, and the virtual object is received as an event from the management server 60 through the virtual object transmitting / receiving unit 31a.

  In step S210, the scene graph stored in the virtual object management unit 16a is changed based on the change information obtained in step S190.

  In step S220, the left eye CG image and the right eye CG image from the view of the worker HMD position and orientation are obtained from the scene graph of the virtual object reflecting the operation result in the image generation unit 13a, the worker pointer 41a, and the pointer pointer 41b. Create CG images respectively.

  In the next step S240, the left-eye CG image and the right-eye CG image generated in step 220 are overwritten on the left-eye captured image and the right-eye captured image from the camera of the image composition unit 15, respectively. A composite image of an object and an actual image can be obtained. The positional relationship among the three-dimensional position / orientation sensor 21a, the left-eye camera 22L, and the right-eye camera 22R is fixed, and a conversion equation can be obtained by performing calibration in advance. Is determined. The left-eye composite image and the right-eye composite image are displayed on the left-eye display 23aL and the right-eye display 23aR of the worker HMD, respectively.

  In step S250, the same binocular image displayed on the worker HMD display 23a is encoded by the image encoding unit 32a and transmitted to the video receiving unit 33b of the mixed reality apparatus for the instructor through the video transmitting unit 33a. Is done.

  In the next step S260, if the termination command for the mixed reality apparatus for workers has been input, the process returns to step S130 if the voice communication termination process is not proceeded to in step S270.

  In step S270, the audio communication process is terminated, the audio connection is disconnected, and the video communication connection is also disconnected.

  In step S280, the communication with the management server 60 is disconnected and the process ends.

  FIG. 6 is a flowchart for explaining the operation of the instructor mixed reality apparatus 10b that instructs / supports work in the mixed reality workspace. First, processing in the work space mode is executed from step 300.

  In step S300, the mixed reality apparatus 10b for the instructor is initialized. In this step, the world coordinate system of the space in which the instructor exists is set in the position / orientation measurement unit 11b, and the output from the three-dimensional sensor described later is a six-parameter output similar to the above-described mixed reality apparatus for workers 10a. Expressed as a set of data. In addition, a table of an instructioner operation article ID, which will be described later, is cleared.

  In the next step S302, a network connection with the management server 60 is established so that data can be transmitted / received between both devices, and data such as virtual object information is downloaded through the virtual object transmitting / receiving unit 31b and is transmitted to the virtual object managing unit 16b. Accumulated.

  In step S304, an audio communication connection and a video communication connection are set with the worker mixed reality apparatus 10a, and audio communication is started. The voice of the instructor is output from the speaker 27b and the voice is input from the microphone 28b. As a result, a conversation by voice during work becomes possible. Video from the mixed reality device for workers is made possible via the video receiver 33b.

In step S306, the three-dimensional position / orientation information (x _s , y _s , z _s , α _s , β _s , γ _s ) of the instructor's stylus 41b is read out and input to the position / orientation measurement unit 11b. 16b.

In step S308, the position / posture (x _h , y _h , z _h , α _h , β _h , γ _h ) of the viewpoint of the instructor is read from the HMD three-dimensional position sensor 21b and input to the position / posture measurement unit 11b. It is held in the management unit 16b.

  In step S310, the stylus and HMD three-dimensional position and orientation data obtained in steps 306 and 308 are transmitted to the management server 60.

  In step S312, the left and right eye images are received by the image receiving unit 33b from the image transmitting unit 33a of the worker mixed reality apparatus, and the left and right eye images are respectively decoded by the image decoding unit 32b.

  In step S314, the decoded left-eye and right-eye images are written into the image output unit 17 and displayed on the left-eye 23bL and the right-eye display 23bR of the instructor HMD 20b, respectively.

  In the next step S316, if the end command of the mixed reality device for the instructor is input, the process returns to step S306, and the process is repeated unless the process proceeds to the voice communication end process in step S318.

  In step S318, the audio communication connection and the video communication connection are disconnected, and the audio processing ends. In step S320, the communication with the management server 60 is disconnected and the process ends.

  FIG. 7A shows button event processing when the stylus button of the instructor is pressed in this workspace mode. Step S400 is activated when the first button 47 of the stylus is pressed. In step S400, the position and orientation of the instructor's stylus and HMD are read from the virtual object management unit 10b and transmitted to the management server 60, and the flow returns to the work space mode flow of FIG. When the second stylus button 48 is pressed, the process proceeds to the shared virtual mode process described later. The shared virtual mode process will be described with reference to the flowchart of FIG.

In step S330, the three-dimensional position / orientation information (x _s , y _s , z _s , α _s , β _s , γ _s ) of the instructor's stylus 41b is read out and input to the position / orientation measurement unit 11b. 10b.

  In step S334, the stylus three-dimensional position and orientation data obtained in steps 330 and 332 is transmitted to the management server 60.

  In step S336, information regarding the change of the worker pointer, the instructor pointer, and the virtual object is received as an event from the management server 60 through the virtual object transmitting / receiving unit 31b.

  In step S338, the scene graph stored in the virtual object management unit 16b is changed based on the change information obtained in step S336.

  In step S340, a left-eye CG image and a right-eye CG image from the instructor viewpoint (instructor HMD position and orientation) are generated by the image generation unit 13b from the scene graph changed in step S338, and the image output unit 17 is processed. The output is displayed on the display 23. Next, the process returns to step S330.

  Through the above processing, a virtual space image consisting only of a virtual object from the instructor's viewpoint is displayed on the instructor HMD. FIG. 7B shows the processing when the stylus button of the instructor is pressed in this shared virtual mode.

  Step S410 is activated when the first button 47 of the stylus is pressed. In step S410, the position and orientation of the instructor's stylus is read from the virtual object management unit 10b and transmitted to the management server 60, and the flow returns to the flow of FIG. 6 (2) shared virtual mode. When the second button 48 of the stylus is pressed, in step S420, a request for clearing the corresponding column of the instructor operation article ID table stored in the virtual object management unit 62 of the management server is transmitted to the management server. Then, the process proceeds to the single virtual mode processing described later.

  The processing in the single virtual mode will be described with reference to the flowchart of FIG.

In step S350, the three-dimensional position / orientation information (x _s , y _s , z _s , α _s , β _s , γ _s ) of the instructor's stylus 41b is read out and input to the position / orientation measurement unit 11b. 10b.

In step S352, the position and orientation (x _h , y _h , z _h , α _h , β _h , γ _h ) of the instructor's viewpoint is read from the HMD three-dimensional position sensor 21b and input to the position / orientation measurement unit 11b. It is held in the management unit 10b.

  In step S354, it is determined whether or not the instructor is currently operating the virtual object by checking the instructor operation article ID table shown in FIG. This table is held in the virtual object management unit 16b of the worker mixed reality device, and stores a value indicating which virtual object is being operated by the instructor. In the example of FIG. 13, the operator is operating the virtual object A. If there is no virtual object being operated, null is stored. If this table is checked and the virtual object is being operated, the process proceeds to step S356; otherwise, the process proceeds to step S358.

  In step S356, an event for moving the virtual object to the stylus position of the instructor is issued.

  In step S358, if there is a change in the stylus position or virtual object of the instructor, it is reflected in the scene graph, and the position and orientation data of the HMD is set as the viewpoint position.

  In step S360, the left-eye CG and the right-eye CG are generated by the image generation unit 13b in the image graph generation unit 13b and written to the image output unit 17, and these CGs are displayed on the display unit 23b.

  In addition, in the mixed reality apparatus 10b for the instructor, in addition to the above flow, an event from the management server is received in the background, and a scene graph is appropriately changed.

  FIG. 7 (3) shows processing when the stylus button of the instructor is pressed in this single virtual mode. Step S430 is activated when the first button 47 of the stylus is pressed. In Step 430, it is determined whether or not the instructor is operating the virtual object by checking the instructor operation article ID table in FIG. If the virtual object is being operated, the process proceeds to step S432; otherwise, the process proceeds to step S434.

  In step S432, the instructor operation article ID table of FIG. 13 is cleared, null is stored, and the flow returns to the single virtual mode flow of FIG.

  In step S434, the distance between the stylus position of the instructor and all the virtual objects existing in the virtual space is compared, and the process proceeds to the next step S436.

  In step S436, if there is a virtual object whose distance is equal to or smaller than a certain threshold (select a shorter distance in a plurality of cases), the process proceeds to step S438 using that virtual object as an operation target. Return to the single virtual mode flow.

  In step S438, the operation object ID obtained in step S436 is written into the instructor operation object ID table in FIG. 13 and the first button event process is terminated.

  Step S440 is activated by pressing the second button 48 of the stylus. In step S440. The instructor operation article ID table in FIG. 13 is cleared and null is stored.

  In step S442, a request to upload the entire scene graph of the virtual object stored in the virtual object management unit 16b to the virtual object management unit 62 of the management server is sent to the management server, the data is uploaded, and the flow of the workspace mode The process from step S306 is performed.

  Next, processing in the management server 60 will be described with reference to the flowchart of FIG. The management server accepts requests and events from the worker mixed reality apparatus 10a and the instructor mixed reality apparatus 10b and processes them.

  The worker stylus processing is activated when a stylus / HMD position event is received from the worker mixed reality device. In step S500, the position / orientation of the worker stylus / HMD is reflected in the scene graph stored in the virtual object management unit 62.

  In step S502, it is determined whether or not the operator is operating the virtual object by checking the operator / instructor operation article ID table of FIG. The operator / instructor operation article ID table is in the virtual object management unit 62, and stores the virtual object operated by the operator and the ID of the virtual object operated by the instructor. In the example of FIG. 12, since the operator does not operate any virtual object, null is stored, and the instructor operates the virtual object C. If the operator operation article ID has a virtual object ID, the process proceeds to step S504. Otherwise, the process proceeds to step S506.

  In step S504, the virtual object being operated is moved to the stylus position updated in step 500, the scene graph stored in the virtual object management unit 62 is changed, and the process proceeds to step S506.

  In step S506, the updated worker stylus, HMD, and virtual object information are transmitted to the worker host, and the worker stylus processing ends.

  The worker button process is started when the worker presses the stylus button. In step S510, the operator operation object ID in FIG. 12 is checked to determine whether or not the worker is currently operating the virtual object. If it is null, it is determined that the operation is not being performed, and the process proceeds to step S514. If the operator operation article ID is stored, it is determined that the operation is being performed and the process proceeds to step S512.

  In step S512, the content of the worker operation item ID in FIG. 12 is replaced with null, and the processing of the worker button event ends.

  In step S514, the current stylus position of the transmitted worker and the positions of all virtual objects in the worker mixed reality space are compared, and the process proceeds to the next step.

  In step S516, if there is a virtual object whose distance is equal to or smaller than a certain threshold (in the case of multiple virtual objects, the shorter distance is selected), the virtual object is set as the operation target, and the process proceeds to step S518. Exit.

  In step S518, the operation object ID obtained in the previous step and the instructor operation object ID in FIG. 12 are checked. If they are the same, it is determined that the operation object is being operated by the instructor, and the operator button event process is terminated. If not, the process proceeds to step S520.

  In step S520, the operation object ID obtained in step S516 is written in the worker operation object ID of FIG. 12, and the worker button event process is terminated.

The instructor stylus processing I is an event processing flow transmitted in step 310 of FIG. 6 (1) workspace mode. In step S530, the position / posture (x _h , y _h , z _h , α _h , β _h , γ _h ) from the HMD three-dimensional position sensor 21b, which is the viewpoint position of the instructor, and the three-dimensional position / posture of the instructor's stylus 41b. Information (x _s , y _s , z _s , α _s , β _s , γ _s ) is received. Since these are values in the world coordinate system of the space where the instructor exists, the position and orientation of the stylus is converted into a relative position from the viewpoint position and orientation, and the instructor stylus relative position (x _d , y _d , z _d , _{α d, β d, γ d} ) = calculating the _{(x s -x h, y s} -y h, z s -z h, α s -α h, β s -β h, γ s -γ h) .

  In step S532, the scene graph of the virtual object management unit 62 is changed using the relative position of the instructor stylus calculated in the previous step as a new stylus event.

  In step S534, it is determined whether or not the instructor is operating the virtual object by checking the instructor operation object ID in the worker / instructor operation object ID table of FIG. If the instructor operation article ID is null, it is determined that the virtual object is not being operated, and the process proceeds to step S538. Otherwise, it is determined that the operating object is being operated, and the process proceeds to step S536.

  In step S536, the virtual object being operated is moved to the stylus position updated in step 532, the scene graph stored in the virtual object management unit 62 is changed, and the process proceeds to step S538.

  In step S538, the updated instructor stylus, HMD, and virtual object information are transmitted to the worker mixed reality apparatus 10a, and the instructor stylus processing I is terminated.

  The instructor stylus processing II is an event processing flow transmitted in step 334 of FIG. 6 (2) shared virtual mode. In step S540, the position / posture from the HMD three-dimensional position sensor 21b, which is the viewpoint position of the instructor, and the three-dimensional position / posture information event of the instructor's stylus 41b are received, and the scene of the virtual object management unit 62 is received based on this information. Change the graph.

  In step S542, it is determined whether or not the instructor is operating the virtual object by checking the instructor operation object ID in the worker / instructor operation object ID table of FIG. If the instructor operation article ID is null, it is determined that the virtual object is not being operated, and the process proceeds to step S546. Otherwise, it is determined that the operating object is being operated, and the process proceeds to step S544.

  In step S544, the virtual object being operated is moved to the stylus position updated in step 540, the scene graph stored in the virtual object management unit 62 is changed, and the process proceeds to step S546.

  In step S546, the updated information on the instructor stylus and the virtual object is transmitted to the worker mixed reality apparatus 10a and the instructor mixed reality apparatus 10b, and the instructor stylus processing II is terminated.

The instructor first button I process is an event process transmitted to the server by the process of step S400 in FIG. 7A that is started when the instructor presses the first stylus button in the work space mode. In step S550, the position / posture (x _h , y _h , z _h , α _h , β _h , γ _h ) from the HMD three-dimensional position sensor 21b that is the viewpoint position of the instructor and the three-dimensional position / posture of the instructor's stylus 41b. Information (x _s , y _s , z _s , α _s , β _s , γ _s ) is received. Since these are values in the world coordinate system of the space where the instructor exists, the position and orientation of the stylus is converted into a relative position from the viewpoint position and orientation, and the instructor stylus relative position (x _d , y _d , z _d , _{α d, β d, γ d} ) = calculating the _{(x s -x h, y s} -y h, z s -z h, α s -α h, β s -β h, γ s -γ h) .

  In step S552, it is determined whether or not the instructor is currently operating the virtual object by checking the instructor operation article ID in FIG. 12, and if it is null, it is determined that the operation is not in progress, and the process proceeds to step S556. If it is stored, it is determined that the operation is being performed, and the process proceeds to step S554.

  In step S554, the content of the instructor operation article ID in FIG. 12 is replaced with null, and the processing of the operator button event ends.

  In step S556, the current stylus position of the instructor stored in the virtual object management unit 62 is compared with the positions of the virtual objects in all worker mixed reality spaces, and the process proceeds to the next step.

  In step S558, if there is a virtual object whose distance is equal to or smaller than a certain threshold value (if a plurality of virtual objects are selected, the shorter distance is selected), the virtual object is set as the operation target, and the process proceeds to step S560. Exit.

  In step S560, the operation object ID obtained in the previous step and the operator operation object ID in FIG. 12 are checked. If they are the same, the operation object is determined to be in operation by the operator, and the instructor button event process I ends. If not, the process proceeds to step S562.

  In step S562, the operation object ID obtained in step S558 is written into the instructor operation object ID in FIG. 12, and the instructor button event process I is terminated.

  The instructor first button II process is an event process transmitted to the server by the process of step S410 in FIG. 7B that is started when the instructor presses the first stylus button in the shared virtual mode. In step S570, it is determined whether or not the instructor is currently operating the virtual object by checking the instructor operation object ID in FIG. 12, and if it is null, it is determined that the operation is not in progress, and the process proceeds to step S574. If it is stored, it is determined that the operation is in progress, and the process proceeds to step S572.

  In step S572, the content of the instructor operation article ID in FIG. 12 is replaced with null, and the processing of the operator button event ends.

  In step S574, the transmitted current stylus position of the instructor is compared with the positions of all virtual objects in the worker mixed reality space, and the process proceeds to the next step.

  In step S576, if there is a virtual object whose distance is equal to or smaller than a certain threshold value (in the case of a plurality of objects, the shorter distance is selected), the virtual object is set as the operation target, and the process proceeds to step S578. Exit.

  In step S578, the operation object ID obtained in the previous step and the operator operation object ID in FIG. 12 are checked. If they are the same, it is determined that the operation object is being operated by the operator, and the instructor button event process II is terminated. If not, the process proceeds to step S580.

  In step S580, the operation object ID obtained in step S576 is written into the instructor operation object ID in FIG. 12, and the instructor button event process II is terminated.

  The instructor operation article ID clear process is an event process transmitted to the server by the process of step S420 in FIG. 7B that is started when the instructor presses the stylus second button in the shared virtual mode. In step S584, the content of the instructor operation article ID in the instructor / worker operation article ID table stored in the virtual object management unit 62 is replaced with null, and the instructor operation article ID clear process ends.

  The instructor scene graph upload process is an event process transmitted to the server by the process of step S442 in FIG. 7 (3) that is started when the instructor presses the stylus second button in the single virtual mode. In step S588, the scene graph stored in the virtual object management unit 62 is replaced with the virtual object scene graph uploaded from the instructor mixed reality apparatus. In step S590, the replaced scene graph information is downloaded to the worker mixed reality apparatus, and the process ends.

  In the present embodiment, the operator's mode switching button is the second button of the stylus shown in 48 of FIG. 3, but is not limited to this, and is assigned to two buttons for moving to the next mode and returning to the previous mode. Alternatively, buttons may be prepared for the number of modes and a function of proceeding to each mode may be assigned.

  In addition, the display device presented to the operator when the instructor is in the shared virtual mode displays a video in which a virtual object as shown in FIG. 4 (2) is arranged in the background of the virtual space. You may use the background which freezes the image | video (for example, image | video as shown in FIG. 4 (1)) of the worker mixed reality space at the moment of entering virtual mode. In this case, even if the worker changes the viewpoint, the background video does not change, but since the viewpoint of the shared virtual object can be freely changed, the gist of the invention is not impaired.

(Second embodiment)
In the first embodiment, the instructor can shift to the work space mode, the shared virtual mode, and the single virtual mode at any time. However, the image on the operator display device 23a does not reflect the mode change of the instructor. For this reason, since it is impossible for the operator to determine which mode the instructor is in, there is a possibility that communication is not smoothly performed. Thus, in the second embodiment, the operator can identify the mode of the instructor while viewing the worker mixed reality image. Specifically, the color of the pointer 41b of the instructor is changed depending on the mode. For example, in FIG. 2A, when the instructor is in the work viewpoint mode, the instructor pointer 41b is green, in the case of the shared virtual mode, the instructor pointer 41b is blue, and in the case of the single virtual mode, the instruction is performed. The person pointer 41b is browned. In this way, the operator can determine which mode the instructor is in based on the color of the pointer of the instructor in the operator mixed reality space. Specific processing of this operation will be described below.

  Assume that the pointer color in the work space mode is set at the time of the initialization process in step S300 of the instructor process flow shown in FIG. In the above example, blue is set. The second button process activated when the operator presses the second button of the stylus in each mode shown in FIG. 7 is different from that in the first embodiment, and will be described with reference to FIG. The second button process in the workspace mode is shown in FIG. Step S402 for transmitting an operator pointer color change event (change event to green in the above example) to the management server is added to the first embodiment. Similarly, in the second button process in the shared virtual mode, the process of step S422 (change event transmission to brown in the above example) is performed, and in the single virtual mode, the process of step S444 (change event transmission to blue in the above example) is performed. Is added. FIG. 9 (4) shows the process when the management server receives the color change process. In step S592, the pointer in the scene graph stored in the virtual object management unit 62 is changed to the designated color, and in the next step S594, the color change event is changed to the worker mixed reality device and the instructor mixed reality device. Send to. The worker mixed reality device and the instructor mixed reality device change the color of the pointer pointer 41b in the scene graph accumulated by this color change event. By these processes, a worker pointer of a different color can be displayed for each worker mode.

  In the present embodiment, the color of the pointer pointer allows the operator to identify the mode of the pointer. However, the present invention is not limited to the color, and any pointer can be used as long as it can be visually identified.

(Third embodiment)
In the first embodiment, when the instructor is in the worker space mode, the both viewpoints of the operator are shared. However, when the instructor is in the shared virtual mode, the operator and the instructor operate from independent viewpoints. It will be. At this time, the pointers of each other are shown in the video images of the display devices that are being viewed, but the viewpoint positions of each other cannot be known. In such a situation, work instructions may be given while looking at the shared virtual object from a completely different viewpoint, which may cause misunderstanding of mutual communication. Therefore, in the third embodiment, when the instructor is in the shared virtual mode, the viewpoint of the instructor is displayed on the display 23a of the operator, and the viewpoint of the operator is displayed on the display 23b of the instructor, so that the viewpoints of each other can be obtained. Make sure you can check. FIG. 10 is a screen example of the display when the instructor is in the shared virtual mode. (1) is the screen of the worker display device 23a, and the viewpoint of the instructor is shown as 55b in addition to the virtual object 43, the worker pointer 41a, and the instructor pointer 41b. Similarly, FIG. 9 (2) shows the operator's viewpoint 55a. In this way, both can confirm each other's viewpoint. The specific processing of this operation is described below as processing different from the first embodiment.

  In the instructor process flow in the instructor virtual reality apparatus, the process in the shared virtual mode shown in FIG. 6B is as shown in FIG. For the different part, in step S332, the three-dimensional position and orientation of the HMD are acquired from the position and orientation measurement unit 11b. In step S335, the stylus position and orientation obtained in step S332 and the HMD position and orientation event obtained in step 335 are transmitted to the management server 60. In step S336, the CG indicating the viewpoint as shown in FIG. 10 can be displayed by receiving the HMD information in addition to the pointer information. In addition, in step S190 of the worker processing flow in the mixed reality apparatus for workers shown in FIG. 5, the CG for displaying the viewpoint can be displayed by receiving the HMD information in addition to the pointer and virtual object information. In the processing at the management server 60, the instructor stylus processing II processing shown in FIG. 8 (4) is changed as shown in FIG. 11 (2). In step S541, in addition to the processing in step 540 in FIG. In step S547, in addition to step S546 of FIG. 8 (4), the information on the HMD of the instructor is also transmitted. Further, the instructioner operation article ID clear process in FIG. 8 (7) is changed as shown in FIG. 11 (3). In step S596, the contents of the instructor operation object ID in the instructor / worker operation object ID table stored in the virtual object management unit 62 are replaced with null, and a CG representing the viewpoint of the instructor / worker is obtained from the scene graph. delete. In step S598, the instructor / worker viewpoint CG clear event is transmitted to the mixed reality apparatus for workers and the mixed reality apparatus for instructions, and the process ends.

  In this embodiment, a CG representing a face is used as an image indicating a viewpoint. However, an arrow indicating a line of sight or a live-action image of an operator / instructor can be used to visually recognize a viewpoint position. Then, the gist of the present invention can be achieved.

It is a block diagram which shows the function structure of the mixed reality remote cooperation work assistance system in 1st embodiment. It is a figure which shows the example of a screen display of the mode of a worker mixed reality space, and the mixed reality apparatus for remote directions. It is a figure which shows the button arrangement example of the stylus for instructions used in the mixed reality apparatus for instructions. It is a figure which shows the example of a screen display at the time of the mode change in the mixed reality apparatus for instructions. It is a flowchart of the worker process in the mixed reality apparatus for workers in 1st embodiment. It is a flowchart of an instructor process in the mixed reality apparatus for instructors in 1st embodiment. It is a flowchart of the button event process in the mixed reality apparatus for instructions in 1st embodiment. It is a flowchart of the reception event process in the management server in 1st embodiment. It is a flowchart of a part of instruction | indication process in the mixed reality apparatus for instructions in 2nd embodiment, and the reception event process in a management server. It is a figure explaining the example of a screen display of the mixed reality apparatus for workers and the mixed reality apparatus for instructions in 3rd embodiment. It is a flowchart of a part of instruction | indication process in the mixed reality apparatus for instructions in a 2nd embodiment, and the reception event process in a management server. It is a figure which shows an example of an operator / instructor operation thing ID table. It is a figure which shows an example of an instructor operation thing ID table.

Claims (8)

An information processing method in which a second user shares a mixed reality space image in which a virtual object is superimposed and displayed in a space where the first user exists,
A mixed reality space image acquisition step of acquiring a mixed reality space image based on a video of the first imaging unit worn by the first user and a virtual object image based on a position and orientation of the first imaging unit;
An event information acquisition step of acquiring event information for the virtual object performed by the first user;
A second user-specified position / orientation information acquisition step of acquiring viewpoint position / orientation information of the second user;
Generating a virtual object image corresponding to the event information based on the viewpoint position and orientation information of the second user,
An information processing method comprising: a first mode for presenting the first image to the second user; and a second mode for presenting the virtual object image to the second user.   The information according to claim 1, further comprising a third mode in which a virtual object image corresponding to the second viewpoint position and orientation information is presented to the second user without being based on the event information. Processing method.   A program for realizing the information processing method according to claim 1 or 2 on a computer. An information processing system for remotely sharing a worker mixed reality space in which a virtual object is superimposed and displayed in a space where a worker exists,
Stereo video transmission means for transmitting a stereo video of the worker mixed reality space from the worker's viewpoint to the instructor,
Virtual object sharing means for sharing the virtual object between the operator and the instructor;
A stereoscopic means for enabling the operator and the instructor to stereoscopically view the stereoscopic video;
An information processing system comprising: a mode in which the instructor shares a stereo video from the worker viewpoint; and a mode in which the virtual object is operated from a viewpoint independent of the worker viewpoint.   The information processing system according to claim 4, wherein the mode for operating the virtual object from a viewpoint independent of the worker includes a mode for sharing the virtual object and a mode for not sharing the virtual object.   5. The information processing system according to claim 4, further comprising switching means for switching between a mode in which the worker shares a stereo video from the worker viewpoint and a mode in which the virtual object is operated from a viewpoint independent of the worker.   The information processing system according to claim 4, further comprising identification means for identifying the mode of the instructor using the worker's stereoscopic vision means.   5. The information according to claim 4, further comprising display means for displaying the viewpoint position of the instructor using the operator's stereoscopic means and displaying the worker's viewpoint position using the indicator's stereoscopic means. Processing system.

Images