JP4991515B2 - Image processing system, image processing system control method, and computer program - Google Patents

Description

  The present invention relates to an image processing system, an image processing system control method, and a computer program that can be applied to a mixed reality technology using HMD (Head Mounted Display).

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology has been known as a technology for seamlessly combining the real world and the virtual world in real time. One of MR techniques uses a video see-through HMD. There, there is a technology that allows a HMD wearer to observe an image in which a subject that is substantially coincident with the subject observed from the pupil position of the HMD wearer is picked up by a video camera or the like, and an image in which CG (Computer Graphics) is superimposed and displayed on the picked-up image. Are known.

  FIG. 9 is a functional block diagram of a wireless mixed reality system (hereinafter referred to as a wireless MR system) using a video see-through HMD. The outline will be described with reference to this figure.

  The video see-through HMD 901 includes an imaging unit 903 that images the outside, a display unit 904 that displays an MR image on which a CG is superimposed, and a three-dimensional position and orientation sensor 905 that outputs a value for calculating a CG drawing position. . Furthermore, a position / orientation information generation unit 906 that calculates a CG drawing position based on the captured image and sensor information, an image composition unit 907 that combines the captured image and the CG image to generate an MR image, and an image processing device for rendering the CG And a wireless I / F 908.

  Hereinafter, the function of each component will be described. First, the imaging unit 903 functions as an imaging unit that captures an external observation image that substantially matches the line-of-sight position of the HMD wearer. The imaging unit 903 is generally composed of two imaging devices for the right eye and left eye to generate a stereo image, an optical system, and a DSP (Digital Signal Processor) for performing subsequent image processing.

  The display unit 904 functions as display means for displaying the combined MR image. The display unit 904 is also composed of two display devices for the right eye and left eye and an optical system. As the display device, a small liquid crystal display or a retina scan type device using MEMS is used.

  The three-dimensional position and orientation sensor 905 is a sensor for obtaining three-dimensional position and orientation information of the HMD wearer. A magnetic sensor or a gyro sensor (acceleration, angular velocity) can be used. The position / orientation information generation unit 906 generates the 3D position / orientation information of the HMD wearer based on the image captured by the imaging unit 903 and the information acquired by the 3D position / orientation sensor 905.

  The image synthesis unit 907 synthesizes the captured image and a CG image generated by a CG drawing unit 910 described later to generate an MR image. The MR image generated by the image composition unit 907 is sent to the display unit 904 and displayed on the wearer. The wireless interface (I / F) 908 is an interface for transmitting the three-dimensional position / orientation information generated by the position / orientation information generation unit 906 to the image processing apparatus 902 and transmitting the rendered CG image to the HMD 901 wirelessly. It is.

  Next, the image processing apparatus 902 is an information processing apparatus for drawing a CG based on the three-dimensional position and orientation information received from the HMD 901 and performing a synthesis process with a captured image. As the image processing apparatus 902, an apparatus having a high-performance arithmetic processing function or graphic display function such as a personal computer or a workstation can be generally used.

  The image processing apparatus 902 includes a wireless I / F 909 and a CG drawing unit 910. The wireless I / F 909 is a wireless I / F on the image processing apparatus side and has the same configuration as the wireless I / F 908. The CG drawing unit 910 draws a CG based on the position / orientation information and the content.

  With the above configuration and processing process, the user can wear the video see-through HMD and experience an MR space in which the real world and the virtual world are seamlessly fused in real time.

  In FIG. 9, the transmission data from the HMD 901 to the image processing apparatus 902 is only the position and orientation information, and the three-dimensional position and orientation sensor 905, the position and orientation information generation unit 906, and the image composition unit 907 are used to reduce the transmission data amount of the entire system. Was placed on the HMD901 side. However, these functions may be provided on the image processing apparatus 902 side. In this case, since the captured image is transmitted from the HMD 901 to the image processing apparatus 902, the amount of transmission data increases, but effects such as simplification of the hardware configuration on the HMD side and reduction of processing load can be expected.

  The wireless MR system using a video see-through HMD experiences an MR space in a wider area by arranging a plurality of markers, a three-dimensional position and orientation sensor such as an objective camera, and an image processing device at appropriate intervals. It becomes possible. In the wireless MR system, the HMD and the image processing apparatus basically perform one-to-one communication. Therefore, when the wearer of the HMD 901 moves and the distance from the image processing apparatus 902 increases, the radio field intensity decreases. , Stable communication may not be possible.

  Therefore, when the distance from the image processing apparatus 902 in communication is increased, the communication state can be maintained by performing a process of switching communication from the current position of the HMD wearer to the image processing apparatus 902 that is closer to the distance. . In order to perform the switching process, the 3D position and orientation information of the marker and the 3D position and orientation sensor, the information regarding the installation position of the image processing device 902, the radio frequency band (channel) and connection used by each image processing device It is necessary to manage information related to connections such as IDs in the system. However, since all the information is manually registered in advance by the user, the load on the user for building the MR space is very large.

  In view of this, a technique has been proposed in which user information when performing wireless connection is reduced by adding unique information necessary for a wireless communication connection request to the marker (see Patent Document 1).

  In this proposed technique, a marker including unique information of a wireless terminal that is to perform communication is photographed, unique information related to wireless communication is acquired from image information, and a connection request signal including the acquired unique information is transmitted. This eliminates the time for searching for wireless terminals that can communicate with each other and the time for selecting a wireless terminal that performs communication from the search result, and quickly starts communication with a desired wireless terminal.

In addition, in an MR system using an HMD, there has been proposed a method for performing additional registration of a marker used for alignment between a real space and a virtual space while the HMD is mounted (see Patent Document 2). In this proposed technique, when a user designates a newly added marker from the operation input unit in a real-world captured image captured by the HMD imaging unit, the world coordinates of the corresponding marker are known in the captured image. Calculation is performed using the world coordinate information of the marker group. Then, additional markers and calculated world coordinates are registered and managed.
JP 2005-142807 A JP-A-2005-327204

  However, the above-described conventional technique has the following problems.

  In the proposed method of Patent Document 1, when switching the connection partner, the marker attached to the wireless terminal that is the communication partner must be visible. Therefore, the user needs to know the position of the wireless terminal and the installation position of the marker. In addition, in a system in which a plurality of users can freely move in a space, such as a wireless MR system, it is not always easy to reliably capture the marker because the user may block the marker.

  Further, with the proposed method of Patent Document 2, it is possible to reduce the user load regarding the acquisition of the position and orientation of the additional marker. On the other hand, it is still necessary for the user to manually register the unique information related to the connection of the wireless terminal while associating it with the acquired position and orientation information.

  The present invention has been made in view of such a problem, and an object of the present invention is to easily configure an MR space and to reduce a user load by providing information used for alignment and information related to wireless connection to a marker. And Further, by switching the three-dimensional position / orientation information given to the marker and the unique connection information of the wireless terminal and registering them, switching control of the communication partner using the position / orientation information becomes possible.

In order to solve the above problems, an embodiment of the present invention includes an imaging unit that captures a real space and generates captured image data, and combines the captured image data with a CG image and displays the display unit on the display unit; An image processing system comprising a plurality of image processing devices that generate the CG image and provide the display device with wireless communication,
The first position and orientation information of the first index is registered for the first index arranged in the real space, and the second index of the second index is attached to the second index attached to the image processing apparatus. Registration means for registering a list for registering the position and orientation information of 2 and the connection information for the display device to wirelessly communicate with the image processing device;
When the communication status with any of the plurality of image processing devices communicating with the display device deteriorates, the first index included in the captured image data generated by the imaging unit is registered in the list. Selection means for selecting an image processing apparatus closest to the display device based on first position and orientation information and the second position and orientation information registered in the list;
The display device switches an image processing device to be communicated to the closest image processing device selected by the selection unit.

Further, the connection information is given to the second indicator, and the display device
Index detecting means for detecting the first index or the second index from the captured image data;
Position and orientation information generating means for generating the position and orientation information of the index detected by the index detection means by using the position and orientation information of other indices registered in the list;
Determination means for determining whether or not the detected index is registered in the list,
When it is determined that the detected index is not registered in the list, and the detected index is the second index, the list includes the second index for the detected second index. The second position / orientation information generated for the second index and the connection information extracted from the second index are added.

  According to the present invention, by using the described system configuration, it is possible to associate the position information used for alignment with the connection information related to the wireless connection with the marker, to easily configure the MR space, and to reduce the user load. Can be reduced.

  Further, by switching the three-dimensional position / orientation information given to the marker and the unique connection information of the wireless terminal and registering them, switching control of the communication partner using the position / orientation information becomes possible.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
A first embodiment for realizing the present invention will be described with reference to the drawings. In the following description, a case where the image processing system according to the embodiment of the present invention is applied to a wireless MR system in which wireless communication is applied to the MR system will be described.

  FIG. 2 is a diagram illustrating a configuration of the wireless MR system according to the first embodiment.

  The video see-through type HMD 201 is an image for obtaining images of the left eye and the right eye that the wearer is observing, and an image for providing the wearer with stereo images for the left eye and the right eye constituting the MR image. It has a display means. The HMD 201 performs wireless communication with the controller 203 via a repeater 202 connected by wire. The wireless communication system used in this system assumes a system that configures a small-scale network such as a WLAN (Wireless Local Area Network) or a WPAN (Wireless Personal Area Network).

  The repeater 202 is used by being attached to a part of the user's body in the same manner as the HMD 201, and the HMD 201 and the repeater 202 can be driven by a battery. The PC 204 wired to the controller 203 has image processing means for drawing a CG image. The PC 204 performs wireless communication with the repeater 202 via the controller 203. In the HMD 201, the CG image received by the repeater 202 and the background image captured by the HMD 201 are combined, and the combined MR image is displayed to the HMD wearer by the image display means. In the present embodiment, the HMD 201 and the repeater 202 are used as individual hardware. However, all the functions of the repeater 202 can be mounted in the HMD 201 to form an integrated apparatus.

  Similarly, with respect to the PC 204 and the controller 203, all the functions of the controller 203 are mounted and integrated in the PC 204, or the functions possessed by the PC 204 and the controller 203 are collected to form a dedicated image processing apparatus. it can. The image composition function does not necessarily have to be provided on the HMD 201 side, and can be configured to be installed on the PC 204 or the controller 203 side.

  In the following description, from a functional viewpoint, a combination of the functions of the HMD 201 and the repeater 202 is classified as an HMD, and a combination of the functions of the PC 204 and the controller 203 is classified as an image processing apparatus.

  FIG. 1 is a diagram showing an example of functional blocks of the entire wireless MR system according to the first embodiment for explaining the features of the present invention.

  In the system according to the first embodiment, the HMD 10 includes an imaging unit 101, a marker detection unit 102, a position / orientation information generation unit 103, a three-dimensional position / orientation sensor 104, a wireless I / F 105, an image composition unit 108, a display unit 109, and a connection. A management unit 117 is included. The image processing apparatus 11 includes a wireless I / F 106, a CG image drawing unit 107, a connection management unit 110, and a wired I / F 111. Further, the system management server 12 includes a hub 112, a wired I / F 113, a list management unit 114, a marker list 115, and a switching control unit 116. All the image processing apparatuses 11 and the system management server 12 are connected via a hub 112 by wire.

  Hereinafter, the function of each component will be described. First, the imaging unit 101 functions as an imaging unit that captures an observation image of the outside world that substantially matches the line of sight of the HMD wearer. The imaging unit 101 is generally composed of two imaging elements for the right eye and left eye for generating a stereo image, an optical system, and a DSP for performing subsequent image processing.

  The marker detection unit 102 detects a marker used for calculating position and orientation information from the captured image. The marker detection process will be described later with reference to FIGS. 3 and 5a to 5b. The position / orientation information generation unit 103 generates the three-dimensional position / orientation information of the HMD wearer based on the image captured by the imaging unit 101 and the information acquired by the three-dimensional position / orientation sensor 104, or the position of the marker. Generate posture information.

  The three-dimensional position and orientation sensor 104 is a sensor for obtaining three-dimensional position and orientation information of the HMD wearer. A magnetic sensor or a gyro sensor (acceleration, angular velocity) can be used. The wireless interface (I / F) 105 transmits the three-dimensional position / orientation information generated by the position / orientation information generation unit 103 to the image processing apparatus 11 and wirelessly transmits the rendered CG image to the HMD 10. Means. As a wireless communication protocol, Bluetooth, IEEE802.11, or the like can be used.

  The image synthesis unit 108 synthesizes the captured image and the CG image generated by the CG image drawing unit 107 described later to generate an MR image. The MR image generated by the image composition unit 108 is sent to the display unit 109 and displayed to the wearer. The display unit 109 functions as display means for displaying the combined MR image. The display unit 109 is also composed of two display devices for right eye and left eye and an optical system. As the display device, a small liquid crystal display or a retina scan type device using MEMS is used.

  The connection management unit 117 is a management unit for managing wireless communication with the image processing apparatus 11 and changes a wireless frequency channel used when a switching instruction is received from the system management server 12.

  Next, in the image processing apparatus 11, the wireless interface (I / F) 106 receives the three-dimensional position / orientation information generated by the position / orientation information generation unit 103 from the HMD 10 and wirelessly renders the drawn CG image to the HMD 10. It is a communication means for transmitting by. As a wireless communication protocol, Bluetooth, IEEE802.11, or the like can be used.

  The CG image drawing unit 107 draws a CG using the 3D position and orientation information received from the HMD 10 and the content image. The connection management unit 110 is a management unit that manages wireless communication in the wireless I / F 106 and communication in the wired I / F 111. A wired interface (I / F) 111 is a communication unit for communicating with the system management server 12 via the hub 112.

  Next, in the system management server 12, a wired interface (I / F) 113 is a communication unit for communicating with the image processing apparatus 11 via the hub 112. The list management unit 114 is a list management unit that manages creation and update of the marker list 115. The marker list 115 is a list for managing information on markers as indices arranged in the real space. A detailed data structure will be described with reference to FIG.

  In the system configuration of FIG. 1, only one HMD 10 and three image processing apparatuses 11a, 11b, and 11c are shown. However, in the present embodiment, these ranges can be used as long as real-time performance from imaging to display can be maintained to some extent. It is possible to increase the number. Therefore, a plurality of HMDs 10 and image processing apparatuses 11 that are not shown may exist in the system. However, since the HMD 10 and the image processing apparatus 11 are premised on one-to-one communication, the number of HMDs 10 is equal to or less than the number of image processing apparatuses 11.

  In the wireless MR system of the present embodiment, two operations are largely performed when marker information is registered and when a user experiences MR. First, an operation at the time of registering marker information, which is necessary when configuring an area, will be described.

  The HMD 10 acquires a real space image by the imaging unit 101, detects a marker from the captured image by the marker detection unit 102 as an index detection unit, and extracts ID information from the detected marker. In the position / orientation information generation unit 103, the marker is passed through the wireless I / F 105 of the HMD 10, the wireless I / F 106a of the image processing apparatus, the wired I / F 111a, the hub 112, the wired I / F 113 of the system management server 12, and the list management unit 114. Refer to Listing 115.

  Of the markers detected from the captured image by the marker detection unit 102, there may be a marker whose position and orientation information is not registered in the marker list. The marker calculates the position and orientation information using a known marker in which the position and orientation information detected in the captured image is registered and the sensing information acquired by the three-dimensional position and orientation sensor 104 as a new marker. Registration to the marker list 115. The above is description of operation | movement in the wireless MR system at the time of marker information registration.

  Next, the user's MR experience will be described. The HMD 10 acquires a real space image with the imaging unit 101, and detects a marker used for calculation of position and orientation information from the captured image with the marker detection unit 102. The position / orientation information generation unit 103 generates three-dimensional position / orientation information of the HMD 10. The three-dimensional position / orientation information is generated using the marker list 115 in the system management server 12, marker information detected from the captured image by the marker detection unit 102, and sensing information acquired by the three-dimensional position / orientation sensor 104. Do.

  The three-dimensional position / orientation information generated by the position / orientation information generation unit 103 is transmitted to the image processing apparatus 11 a via the wireless I / F 105 of the HMD 10. When the image processing apparatus 11a receives the three-dimensional position / orientation information from the HMD 10 via the wireless I / F 106a, the image processing apparatus 11a draws a CG image that is a virtual image of the virtual space based on the three-dimensional position / orientation information. At the same time, the 3D position and orientation information received from the HMD 10a is transmitted to the system management server 12 via the wired I / F 111a.

  The CG image, which is virtual image data generated by the image processing apparatus 11a, is transmitted to the HMD 10 via the wireless I / F. The image synthesis unit 108 synthesizes the captured image data that is a real space image and the received CG image, and the display unit 109 displays an MR image as the synthesized image data. In the switching control unit 116 of the system management server 12, the 3D position and orientation information of the HMD 10 transmitted from the image processing apparatus 11 a and the position of the connection information marker attached to the image processing apparatus 11 registered in the marker list 115. Switching control is performed with reference to the posture information.

  For example, when it is detected that the HMD 10 has moved and the distance from the image processing apparatus 11a has been increased, the switching control unit 116 has the most current position of the HMD 10 among the connection information markers registered in the marker list 115. A near image processing apparatus 11 is selected. Then, the switching instruction is transmitted to the connection management unit 110a of the image processing apparatus 11a in communication with the HMD 10 and to the connection management unit 110 of the image processing apparatus 11 that performs communication next. When the connection management unit 110a of the image processing apparatus 11a receives a switching instruction from the system management server 12, the connection management unit 110a transmits the instruction to the connection management unit 117 of the HMD 10 via the wireless I / F. The connection management unit 117 of the HMD 10 can switch the connection with another image processing apparatus 11 by changing the radio frequency channel to be used based on the received switching instruction.

  Next, the concept of the three-dimensional position / orientation information in the present embodiment will be described with reference to FIG. FIG. 3 is a diagram for describing three-dimensional position and orientation information using a captured image of the HMD in the first embodiment.

  In the captured image 30 acquired by the imaging unit 101 of the HMD 10, absolute position and orientation information in the MR space or relative position and orientation information with respect to an arbitrary absolute position and orientation is given in advance to the marker 302 installed in the real space. The marker detection unit 102 detects the marker 302 from the captured image data. The position / orientation information generation unit 103 calculates three-dimensional position / orientation information from information such as the size and shape of the detected marker 302 and a fill pattern. In the present embodiment, the three-dimensional position / orientation information is information indicating the relative positional relationship of the HMD 10 main body based on the position / orientation information of the marker 302 and the direction in which the HMD wearer is observing the marker.

  In FIG. 3, as an example, a three-dimensional coordinate system (303a, 303b, 303c) having the origin at the center of the marker 302 is assumed as a coordinate system in the MR space. It is not necessary to set. For example, an arbitrary position may be set by associating the relative positional relationship between the origin of the coordinate system and the marker 302. In this embodiment, a plurality of markers can be used simultaneously as shown in FIG. By using a plurality of markers, it is possible to enlarge an area where position and orientation information can be calculated.

  When a plurality of markers are used at the same time, the relationship between the relative positions and orientations of the markers is calculated in advance, and the position and orientation information of each marker is registered, so that the three-dimensional position and orientation information of the HMD 10 with higher accuracy can be obtained. Can be calculated. Even when a plurality of markers are used at the same time, it is not necessary to use a marker that can be identified in the direction by the internal filling pattern as shown in FIG. For example, a color marker using a specific color or a marker having no directionality such as a light emitting element such as an LED can be used. In addition, natural features included in the captured image, such as table contour line 304, wall / floor boundary line 305, and corner feature point 306, and any color in the image, not markers prepared in advance. Can be extracted and used as an index.

  By using multiple markers of the same type as described above, using several types of markers at the same time, or using a combination of marker information and natural feature information in the image, the 3D position and orientation can be improved with higher accuracy. Information can be generated. In addition, since the relative positional relationship between multiple markers and natural features is calculated and registered in the system, even if not all markers and feature points are displayed in the image, the position of each marker or natural feature It is possible to estimate the posture.

  In order to generate the three-dimensional position and orientation information of the wireless MR system of the present embodiment, in addition to the method using image information captured by the subjective camera (internal image sensor) built in the HMD 10 described above, sensing information by various sensing devices is used. There is a method of using. As an example of the sensing device, there is a type of device that is used by being attached to the HMD 10 such as a magnetic sensor or a gyro sensor. In addition, there is a type of device that is attached and used in addition to the HMD 10, such as an objective camera (external image sensor). When using an objective camera, a marker is also attached to the HMD main body as a detection target.

  These sensing devices can be used alone, or a plurality of types of devices can be used in combination. Further, by generating 3D position and orientation information by combining image information captured by a subjective camera built in the HMD 10 and sensing information acquired by a sensing device, more accurate 3D position and orientation information is obtained. Can do. Further, it is possible to acquire the three-dimensional position / orientation information even when the marker is not visible with the subjective camera.

  Next, an area configuration method of the wireless MR system in the present embodiment will be described with reference to FIGS. 5a to 5e. FIG. 5a is a diagram for explaining an area configuration method of the wireless MR system in the first embodiment for explaining the features of the present invention.

  The wireless MR system shown in FIG. 5A includes an HMD 501, image processing apparatuses 502 a, b, c, d, and e, a hub 503, and a system management server 504. Also, in the space, a reference marker 505 given absolute position and orientation information in the space, and position information markers (first indices) 506a, b, c, d, and 506 for use in calculating the position and orientation information. e is arranged. The image processing apparatus 502 is attached with a connection information marker (second index), and the connection information marker is attached with a marker ID and information related to wireless connection such as a MAC address and a use frequency channel. Therefore, the information of the marker ID, the MAC address, and the used frequency channel can be acquired from the image data by the imaging unit 101 of the HMD 501. The marker ID, the MAC address, and the used frequency channel included in the image data may be recognized by characters. When the information is expressed by a two-dimensional barcode, it can be read from the two-dimensional barcode.

  In FIG. 5a, the patterns of the position information marker 506 and the connection information marker are set so as to have numbers and alphabets, respectively, so that the description can be easily understood visually. Information about the marker is stored in the marker list 115 shown in FIG. 5b. The marker list 115 is managed by the system management server 504. In the marker list 115, ID 511, type 512, position and orientation information 513, and wireless connection information 514 are registered in association with each other.

  ID511 is identification information for uniquely identifying a registered marker. The type 512 includes “connection information” and “position and orientation information” as information for determining whether the marker is a position information marker as a first index or a connection information marker as a second index. One of these is registered. The position and orientation information 513 is information on the position and orientation of the marker in the MR space. The position and orientation information of the position information marker and the position and orientation information of the connection information marker are registered as the second position and orientation information as the first position and orientation information. In the wireless connection information 514, information on the MAC address of the image processing apparatus 502 and the used frequency channel is registered.

  In this embodiment, first, the ID 511, type 512, and wireless connection information 514 (only the connection information marker) acquired from the marker are stored in the marker list 115, and then calculated by the image processing apparatus 502 as the position / orientation information 513. A value is registered.

  In the area configuration, first, the HMD 501 broadcasts a connection request to the image processing apparatuses 502 around it and starts wireless communication with one of the image processing apparatuses 502 that has responded. At this time, the HMD 501 may not be able to determine which of the image processing devices 502a, b, c, d, and e is communicating with the image processing device 502 installed.

  Subsequently, a captured image including the reference marker 505 is acquired using the HMD 501 in a state where the HMD 501 and any one of the image processing apparatuses 502 are communicating. A surrounding line 507 in the figure indicates a marker included in the captured image. In the captured image, if the marker list 115 managed by the system management server 504 includes an unregistered marker for which position and orientation information is not registered, an unregistered marker is obtained from the position and orientation information of the reference marker 505. Is calculated and registered in the list.

  Here, the position / orientation information of the connection information marker attached to the image processing apparatus 502a is calculated using the reference marker 505, and the calculated position / orientation information is registered in “A” of the ID 511 in the list of FIG. 5B. It becomes. By such a processing flow, the ID 511, the position / orientation information 513, and the wireless connection information 514 are associated with the marker attached to the image processing apparatus 502a.

  Next, consider a case where an unregistered marker is detected in the captured image by moving the HMD 501 as shown in FIG. 5c. In this case, the position and orientation information of the unregistered marker is calculated from the relative relationship with the position and orientation information of a known marker in which the position and orientation information 513 is registered in the marker list 115 and registered in the marker list 115.

  In FIG. 5c, the connection information marker attached to the image processing apparatus 502b and the position information marker 506a correspond to unregistered markers. Therefore, the position and orientation information of each marker is calculated with reference to the connection information marker attached to the image processing apparatus 502a. As a result, as shown in FIG. 5d, the calculated values are registered in the position / orientation information 513 of the marker list 115.

  When the above process is repeated and registration of the position / orientation information 513 of all the markers in the list is completed, a marker list 115 as shown in FIG. 5E is completed. Since all marker IDs are registered in the marker list 115, it is possible to determine whether or not registration of position and orientation information of all markers has been completed. Therefore, when there is an unregistered marker in the area, the contents are displayed on the display unit 109 of the HMD 501 and notified to the user, so that omission of marker registration can be eliminated.

  By managing the marker list in FIG. 5e in the system, the correspondence between the connection information and the position / orientation information of the image processing apparatus 502 becomes clear. Therefore, a screen as shown in FIG. 6 can be provided to the HMD wearer. FIG. 6 is a diagram for explaining a display function of the image processing apparatus in communication corresponding to the first embodiment of the invention.

  In FIG. 6, a screen as shown on a screen 602 is displayed on the display unit 109 of the HMD 201 for the HMD wearer 601. As shown on the screen 602, when the HMD wearer 601 observes the marker 604 attached to the image processing apparatus 603, the marker detection unit 102 extracts information on the marker 604 and extracts corresponding connection information from the marker list 115. The Then, as shown in an area 605 of the screen 602, the image processing apparatus 603 to which the HMD 201 is currently connected can be clearly shown, and information related to the image processing apparatus 603 can be presented to the HMD wearer 601.

  Similarly, when a registered marker is observed with the HMD 201, an identifier can be attached and displayed. Further, the position / orientation information registered in the marker list can be appropriately updated during the system operation.

  For example, when it is detected that the installation position of the image processing apparatus 502b is different from the information registered in the marker list, the current position / orientation information is calculated and the marker list is updated. The detection of the installation position change is realized by using a relative positional relationship with a plurality of surrounding markers and sensor information. By making the information of the marker list updatable, it is possible to adjust the installation position of the image processing apparatus while confirming the communication state between the HMD and the image processing apparatus after the area has been configured once. Further, as described above, the marker list is centrally managed by the system management server 504, and the HMD only transmits the position and orientation information of the marker acquired individually. It is possible to use the area configuration simultaneously.

  Next, a process when a new marker is added in the space corresponding to the present embodiment or when the marker installation position is changed will be described with reference to FIG. FIG. 7 is a flowchart showing an example of the flow of processing when registering marker information corresponding to the first embodiment of the invention. The processing corresponding to FIG. 7 is realized by executing a processing program corresponding to each functional block.

  In step S <b> 701, the HMD 10 broadcasts a connection request to the surrounding image processing apparatuses 11 via the wireless I / F 105 and attempts to establish a wireless connection. If the connection with any of the surrounding image processing apparatuses 11 is successful (“YES” in step S702), the process proceeds to step S703. On the other hand, if the connection fails ("NO" in step S702), the process returns to step 701 and repeats the broadcast until the connection is successful.

  In the process of step S703, imaging is performed by the imaging unit 101 of the HMD 10, and a captured image is acquired. Subsequently, when the marker detection unit 102 detects a marker from the captured image and detects the marker (“YES” in step S704), the process proceeds to step S705. On the other hand, if the marker cannot be detected (“NO” in step S704), the process returns to step S703, and imaging is repeated until the marker can be detected from within the captured image.

  When a marker is detected in step S704, in step S705, an ID is determined from the marker image detected by the position and orientation information generation unit 103. In step S <b> 706, the position / orientation information generation unit 103 acquires the marker list 115 from the system management server 12, and acquires marker information in the captured image in which the position / orientation information is already registered in the marker list 115. In subsequent step S707, the position / orientation information generation unit 103 calculates the position / orientation information of the marker detected in step S704 using the acquired marker information or the sensing information acquired by the sensor.

  If the position / orientation information of the marker in the captured image has been successfully calculated (“YES” in step S708), the process proceeds to step S709. On the other hand, if it cannot be calculated (“NO” in step S708), the process returns to step S703, and the captured image is acquired again. In step S <b> 709, it is determined whether the marker position / orientation information calculated by the position / orientation information generation unit 103 is already registered in the marker list 115. If the calculated marker position / orientation information is not registered in the marker list (“NO” in step S709), the list management unit 114 registers the position / orientation information in the marker list 115 in step S710, and the process ends. To do.

  If the calculated marker position / orientation information has already been registered in the marker list (“YES” in step S709), the process proceeds to step S711. In step S <b> 711, the position / orientation information generation unit 103 determines whether or not there is a change as compared with the position / orientation information registered in the marker list 115. If it is determined that the registered position / orientation information has changed ("YES" in step S711), the process proceeds to step S710 to register the latest position / orientation information. On the other hand, if it is determined that there is no change from the registered position and orientation information (“NO” in step S711), the process returns to step S703, and the captured image is acquired again.

  As described above, the registration contents of the position / orientation information 513 in the marker list 115 can be updated for markers for which position / orientation information is not registered or markers whose position / orientation information has changed.

  Next, an area configuration and communication switching processing in the wireless MR system corresponding to the present embodiment will be described with reference to FIG. FIG. 4 is a diagram for explaining the area configuration and the communication switching process of the wireless MR system in the first embodiment of the invention.

  In the wireless MR system in this embodiment, the HMD 401 performs wireless communication with any one of the image processing apparatuses 402a, b, c, d, and e on a one-to-one basis. Further, by displaying an MR image in which a CG image generated by the image processing device 402 and a captured image captured by the HMD are combined, the HMD wearer can experience the MR space.

  Here, the image processing apparatuses 402a, b, c, d, and e are assumed to perform communication using the frequency channels indicated by 405a, b, c, d, and e, respectively.

  The image processing apparatuses 402 a, b, c, d, and e are connected to the system management server 404 via a hub 403 by wire. The system management server 404 has functions for managing 3D position and orientation information and connection information, and for controlling communication switching. In this embodiment, the system management server 404 is separate hardware, but the function of the system management server 404 can be implemented in the image processing apparatus 402. By mounting the function of the system management server 404 in the image processing apparatus 402, one piece of hardware can be reduced, and the cost of the entire system is expected to be reduced.

  When the HMD 401 is present at the position in the figure, wireless communication is performed with the image processing apparatus 402d having the closest distance. At this time, the HMD communicates with the image processing apparatus 402d using the Ch5 frequency channel set in the image processing apparatus 402d. If the HMD wearer moves and moves to the position of the HMD 401 ′ in the figure, the distance from the image processing apparatus 402d increases, and the radio wave intensity deteriorates, making it difficult to maintain a stable communication state.

  In such a case, the HMD 401 ′ performs switching control so as to connect to the image processing apparatus 402a using the Ch1 frequency channel set in the image processing apparatus 402a that is the closest to the current position. For this switching control, first, the type 512 of the marker position and orientation information included in the image data captured by the imaging unit 101 and the position and orientation information included in the marker list 115 is “connection information”. Compare with certain position and orientation information. As a result of the comparison, the “connection information” marker having the closest distance is selected. In the example of FIG. 4, the marker corresponds to the marker of the image processing apparatus 402a that is closest to the HMD 401 ′.

  By performing the above switching control, it is possible to experience the MR space in a wide area that cannot be compensated for in an area where communication can be performed with one image processing apparatus. In order to realize such switching processing, it is necessary to manage the frequency channel information and installation position information set in the image processing apparatus 402 and the position information of the HMD 401 by the system management server 404.

  As described above, by using the described system configuration, the user can perform the area configuration of the wireless MR system simply by wearing the HMD and walking around the space where the marker and the image processing apparatus are arranged. Can do. Therefore, it is possible to greatly reduce the user load related to registration of marker position and orientation information, connection information of the image processing apparatus, and the like. In addition, since the change of the installation position and orientation of the marker and the image processing apparatus can be detected and the marker list can be updated, even if the arrangement of the image processing apparatus is changed while checking the communication status, the position and orientation information is quickly changed. It is possible. Furthermore, since the position and orientation information of the image processing apparatus and the connection information are simultaneously managed by the system management server, it is possible to perform control for switching the image processing apparatus to be connected according to the position of the HMD.

(Second Embodiment)
Next, a second embodiment of the invention will be described with reference to the drawings. In the first embodiment described above, marker list management and switching control are performed on the system management server and the image processing apparatus side. In contrast, the present embodiment is characterized in that marker list management and switching control are performed on the HMD side. The contents will be described focusing on this point.

  FIG. 8 is a diagram showing an example of functional blocks of the entire wireless MR system corresponding to the second embodiment of the invention. Also in the present embodiment, as in the first embodiment, first, the operation during area configuration will be described.

  The HMD 10 acquires a real space image with the imaging unit 101, detects a marker from the captured image with the marker detection unit 102, and extracts ID information. The position / orientation information generation unit 103 refers to the marker list 802 via the list management unit 801. When the list management unit 801 determines that there is a marker whose position and orientation information is not registered in the marker list 802 among the markers detected from the captured image by the marker detection unit 102, the position and orientation information generation unit 103 Position and orientation information is calculated. The position / orientation information generation unit 103 generates position / orientation information using a known marker in which the position / orientation information has already been registered detected in the captured image and the sensing information acquired by the three-dimensional position / orientation sensor 104. I do. The generated position / orientation information is registered in the marker list 802 by the list management unit 801.

  The above is the operation of the wireless MR system when registering marker information in the present embodiment. The processing contents when performing the area configuration differ greatly from the first embodiment in whether or not wireless communication is performed with the image processing apparatus in order to refer to the marker list 802.

  In this embodiment, since the marker list 802 is managed in the HMD 10, it is not necessary to perform wireless communication at the time of area configuration, and power saving of the HMD 10 can be realized. However, when an area is configured using a plurality of HMDs 10, it is necessary to perform wireless communication in order to exchange information acquired by the respective HMDs. Therefore, an MR experience can be experienced as the processing load of the HMDs increases. Time is shortened.

  Next, the operation of the user during the MR experience in this embodiment will be described. The flow of processing for displaying the MR image is the same as that in the first embodiment. The content of processing at the time of MR experience is greatly different from that of the first embodiment in that the main body of switching control is on the HMD 10 side.

  In the present embodiment, after detecting the movement of the HMD 10 based on the position and orientation information generated by the position and orientation information generation unit 103 in the HMD 10, the marker information in the marker list 802 is immediately referred to, and the switching control unit 803 Switching control can be started. Therefore, the system management server 12 is accessed via the image processing apparatus 11 that has occurred during the switching control of the first embodiment, and the waiting time until the switching instruction from the system management server 12 is received may be omitted. it can.

  In this embodiment, the waiting time for switching is shortened, so that the frequency of transmission errors and frame dropping at the time of switching can be reduced.

  As described above, by using the system configuration of this embodiment that manages the marker list 802 on the HMD 10 side and performs switching control, the system management server can be excluded from the overall system configuration, and the system configuration is simplified. Can be realized. However, as the function of the HMD increases, the size increases and the processing load of the HMD itself increases. However, it is not necessary to perform wireless communication that occupies most of the power consumption when configuring the area. Can be expected to be longer.

  Furthermore, since the communication time with the system management server at the time of switching control can be saved, the processing time required for switching can be shortened, and the frequency of errors and frame dropping can be reduced. However, as described above, the processing load in the HMD increases due to the complexity of the functions. Therefore, in order to realize the system, the processing capability of the HMD is required to realize the system configuration of the first embodiment. It is required to have a higher performance than the processing capability.

[Other Embodiments]
Note that the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, and a printer), and a device (for example, a copying machine and a facsimile device) including a single device. You may apply to.

  The object of the present invention can also be achieved by supplying, to a system, a storage medium that records the code of a computer program that realizes the functions described above, and the system reads and executes the code of the computer program. In this case, the computer program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the computer program code constitutes the present invention. In addition, the operating system (OS) running on the computer performs part or all of the actual processing based on the code instruction of the program, and the above-described functions are realized by the processing. .

  Furthermore, you may implement | achieve with the following forms. That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Then, based on the instruction of the code of the computer program, the above-described functions are realized by the CPU or the like provided in the function expansion card or function expansion unit performing part or all of the actual processing.

  When the present invention is applied to the above storage medium, the computer program code corresponding to the flowchart described above is stored in the storage medium.

It is a figure which shows an example of the functional block of the whole wireless MR system corresponding to the 1st Embodiment of invention. It is a figure which shows an example of a structure of the wireless MR system corresponding to 1st Embodiment. It is a figure for demonstrating the three-dimensional position and orientation information using the captured image of HMD in the 1st Embodiment of invention. It is a figure for demonstrating the area structure and communication switching process of the wireless MR system in the 1st Embodiment of invention. It is a figure for demonstrating the area structure method of the wireless MR system in the 1st Embodiment of invention. It is a figure which shows the creation example of the marker list | wrist 115 in the 1st Embodiment of invention. It is a figure for demonstrating the area structure method of the wireless MR system in the 1st Embodiment of invention. It is a figure which shows the creation example of the marker list | wrist 115 in the 1st Embodiment of invention. It is a figure which shows an example of the marker list | wrist 115 in the 1st Embodiment of invention. It is a figure for demonstrating the display function of the image processing apparatus in communication in the 1st Embodiment of invention. It is a flowchart which shows an example of the flow of the process at the time of the marker information registration corresponding to the 1st Embodiment of invention. It is a functional block diagram which shows the whole system configuration | structure in 2nd Embodiment. It is a figure which shows the structure of a wireless MR system.

Explanation of symbols

10 ... HMD
DESCRIPTION OF SYMBOLS 11 ... Image processing apparatus 12 ... System management server 101 ... Imaging unit 102 ... Marker detection part 103 ... Position and orientation information generation part 104 ... Three-dimensional position and orientation sensor 105 ... Wireless I / F (HMD)
106: Wireless I / F (image processing apparatus)
107 CG image rendering unit 108 Image compositing unit 109 Display unit 110 Connection management unit (image processing apparatus)
111... Wired I / F (image processing device)
112 ... Hub 113 ... Wired I / F (system management server)
114 ... List management unit 115 ... Marker list 116 ... Switching control unit 117 ... Connection management unit (HMD)

Claims (7)

An imaging unit that captures a real space and generates captured image data, a display device that synthesizes the captured image data with a CG image and displays it on the display unit, and generates the CG image and wirelessly communicates with the display device An image processing system comprising a plurality of image processing devices provided by
The first position and orientation information of the first index is registered for the first index arranged in the real space, and the second index of the second index is attached to the second index attached to the image processing apparatus. Registration means for registering a list for registering the position and orientation information of 2 and the connection information for the display device to wirelessly communicate with the image processing device;
When the communication status with any of the plurality of image processing devices communicating with the display device deteriorates, the first index included in the captured image data generated by the imaging unit is registered in the list. Selection means for selecting an image processing apparatus closest to the display device based on first position and orientation information and the second position and orientation information registered in the list;
The display apparatus switches an image processing apparatus to be communicated to the image processing apparatus having the shortest distance selected by the selection unit. The connection information is given to the second indicator,
The display device
Index detecting means for detecting the first index or the second index from the captured image data;
Position and orientation information generating means for generating the position and orientation information of the index detected by the index detection means by using the position and orientation information of other indices registered in the list;
Determination means for determining whether or not the detected index is registered in the list,
When it is determined that the detected index is not registered in the list, and the detected index is the second index, the list includes the second index for the detected second index. The image processing system according to claim 1, wherein the second position and orientation information generated for the second index and the connection information extracted from the second index are added.     When it is determined that the detected index is not registered in the list and the detected index is the first index, the list includes the first index for the detected first index. The image processing system according to claim 2, wherein the first position and orientation information generated for one index is added. When the detected index is already registered in the list, the determining unit matches the position and orientation information of the registered index with the position and orientation information generated by the position and orientation information generating unit. Determine whether or not to
The position / orientation information of the registered index is updated with the position / orientation information generated by the position / orientation information generation unit when the determination unit determines that they do not match. The image processing system according to 2 or 3.   The image processing system according to claim 1, wherein the display device further includes the registration unit and the selection unit. The image processing system further includes a system management server that communicates with the image processing apparatus,
The system management server includes the registration unit and the selection unit, and notifies the display device of an instruction to switch to the closest image processing device via the image processing device. The image processing system according to claim 1. An imaging unit that captures a real space and generates captured image data, a display device that synthesizes the captured image data with a CG image and displays it on the display unit, and generates the CG image and wirelessly communicates with the display device A method for controlling an image processing system comprising a plurality of image processing devices provided by
The registration means of the display device registers the first position / orientation information of the first index for the first index arranged in the real space, and for the second index attached to the image processing apparatus. A registration step of registering a list for associating and registering the second position and orientation information of the second index and connection information for the display device to wirelessly communicate with the image processing device;
The first indicator included in the captured image data generated by the imaging unit when the selection unit of the display device deteriorates the communication status with any of the plurality of image processing devices communicating with the display device Selection for selecting an image processing device closest to the display device based on the first position and orientation information registered in the list and the second position and orientation information registered in the list Process,
With
The display apparatus switches an image processing apparatus to be communicated to the image processing apparatus with the shortest distance selected in the selection step.

Images