JP2019066292A - Video display system - Google Patents

Abstract

To provide a video display system with which it is possible to correct the display arrangement of a virtual object with high accuracy in accordance with a user's position.SOLUTION: A video display system 100 comprises: a first positional relationship computing unit 153 for detecting a first submarker that corresponds to a reference marker in a real space and computing the positional relationship between a first virtual position closest to the first submarker among a plurality of first virtual positions previously correlated to coordinate axes and the first submarker; a second positional relationship computing unit 154 for detecting a second submarker arranged at a predetermined position relative to the reference marker in a real space and computing the positional relationship between a plurality of second virtual positions previously correlated to coordinate axes at narrower intervals than the intervals of the plurality of first virtual positions and the second submarker; and a correction unit 155 for correcting the relationship between the coordinate axes and the position and attitude of an imaging device.SELECTED DRAWING: Figure 1

Description

  BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to a video display system that displays video for supporting a user's work.

  A technique related to Mixed Reality (hereinafter referred to as "MR") is known which displays a virtual object in a virtual space by computer graphics or the like corresponding to a real object in real space (see, for example, Patent Document 1) ). Patent Document 1 describes a technique for correcting positional deviation between a real image and a virtual image when combining and displaying a real image obtained by capturing an architectural model that is a real object and a virtual image.

Unexamined-Japanese-Patent No. 2006-18444

  As a method of using the MR technology, it is conceivable to provide the user with virtual images corresponding to the real object to support the user's work through an HMD (Head Mount Display) or the like (hereinafter referred to as "MR device"). . For example, in the welding operation of steel materials, it is necessary to read drawings, scribe on steel materials, and confirm the scribes. In complicated drawings, even skilled technicians take time to understand, and if there are many welded parts, reworking due to simple errors occurs. For this reason, the welding operation of the conventional steel materials required skilled techniques. If the virtual object indicating the mounting member is superimposed on the actual steel material using the MR device, the welding operation of the steel material is simplified.

  However, when an MR device is used, the position recognition error in internal processing may increase the displacement of the virtual object with respect to the real space as the moving distance of the user increases. For this reason, when the user moves and performs detailed work at the movement destination like the welding operation of steel materials described above, the virtual object is displayed largely shifted with respect to the real space at the movement destination, that is, the work place There is a fear. When the arrangement for displaying virtual objects (hereinafter referred to as "display arrangement") is largely deviated, it is difficult to perform detailed work at the movement destination. For example, even in the case of a welding operation for a steel material of 10 m or more, an accuracy of mm is required, but with the current MR device, an error of several cm occurs every several m of walking.

  The correction technique of the positional deviation described in Patent Document 1 is premised on the user being stationary. Therefore, the related art does not consider that the display arrangement of the virtual object is shifted when the user moves after determining the display arrangement of the virtual object. Therefore, there is a possibility that the display arrangement of the virtual object may be shifted at the moving destination.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide an image display system capable of correcting the display arrangement of a virtual object with high accuracy according to the position of a user, for example.

  In order to achieve the above object, a video display system according to the present invention is a video display system for displaying a virtual object associated with a real space, and an imaging device for imaging the real space, and movement of the imaging device Storage that stores in advance the position information of the virtual object on the coordinate axis in the virtual space, a movement detection unit that detects the position, a position and orientation calculation unit that calculates the position and orientation of the imaging device from the movement detected by the movement detection unit And a reference arrangement setting unit for detecting a reference marker arranged in the real space from the image captured by the imaging device and correlating the coordinate axis in the virtual space to the real space from the detection result of the reference marker; The first sub marker located at a predetermined position relative to the reference marker in real space is detected from the captured image, and the detection result of the first sub marker and the position pattern A position closest to the first sub-marker among the plurality of first virtual positions previously associated with the coordinate axis from the position and orientation of the image-capturing device when the imaging device captures the first sub-marker calculated by the calculation unit (1) A first positional relation operation unit for calculating the positional relation between the first virtual marker and the first submarker, and a second image arranged at a predetermined position with respect to the reference marker in the real space from the image captured by the imaging device A plurality of first virtual positions are detected from the detection results of the second submarker and the position and orientation of the imaging device when the imaging device captures the second submarker, the detection result of the second submarker, and the position and orientation calculation unit. The second positional relationship operation unit that calculates the positional relationship between the plurality of second virtual positions and the second submarker that are associated in advance with the coordinate axes at an interval smaller than the interval, and the first positional relationship operation unit A correction unit that corrects the relationship between the coordinate axis and the position and orientation of the imaging device calculated by the position and orientation calculation unit from the positional relationship and the positional relationship calculated by the second positional relationship calculation unit And a display unit for displaying a virtual object based on the positional information being corrected and the relationship between the coordinate axes and the position and orientation of the imaging device calculated by the position and posture calculation unit corrected by the correction unit.

  In the image display system according to the present invention, the first positional relation operation unit is configured to include the first sub marker disposed at a predetermined position with respect to the reference marker, the first sub marker among the plurality of first virtual positions, and the first sub marker. The positional relationship with the near first virtual position is calculated. For this reason, when the position and orientation of the imaging device calculated from the detection result by the movement detection unit include an error, the coordinate axes and the coordinate positions are determined according to the arrangement of the plurality of first virtual positions even at a position distant from the reference marker. The relationship between the position and orientation of the imaging device may be corrected, and the display arrangement of the virtual object may be corrected. Furthermore, the second positional relation operation unit calculates the positional relation between the second sub marker and the second virtual position. By using the positional relationship between the second submarker and the second virtual position while utilizing the positional relationship between the first submarker and the first virtual position, the deviation of the virtual object with respect to the real space is further increased at the position of the imaging device. It can be corrected to high precision. That is, the display arrangement of the virtual object can be corrected with high accuracy at the position of the user on which the imaging apparatus is mounted.

  The present invention has been made in view of the above problems, and can provide a video display system capable of correcting the display arrangement of a virtual object with high accuracy in accordance with the position of the user.

It is a figure for demonstrating the function of the video display system which concerns on one Embodiment. It is a figure for demonstrating the various members used by the operation | work using a video display system. It is a figure which shows the display of a video display system, and a surrounding environment before a user moves to a work position. FIG. 6 is a diagram showing the display of the video display system and the surrounding environment after the user moves and before the display arrangement of the virtual object is corrected. It is a figure which shows the display of a video display system, after a user moves to an action | operation position and arrange | positions a 2nd submarker, and before correcting the display arrangement of a virtual object. It is a flowchart which shows the process of correcting the display arrangement of a virtual object. It is a flowchart which shows the control processing performed by a display control part. It is a flow chart which shows gap amendment processing performed by a display control part. It is a figure which shows the hardware constitutions of the video display system which concerns on one Embodiment.

  Hereinafter, an embodiment of a video display system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference symbols, without redundant description.

  First, a video display system 100 according to the present embodiment will be described with reference to FIGS. 1 to 5. FIG. 1 shows a block diagram of a video display system 100. The image display system 100 is realized by, for example, an HMD or the like worn by a user, and displays a virtual object by CG (computer graphics) associated with a real space (world coordinate system). The image display system 100 employs an optical see-through method that allows a user to visually recognize a real object and a virtual object in real space using a half mirror. The video display system 100 may adopt a video see-through method in which a virtual object is fused to a video in real space and the user visually recognizes it. In the following, it will be described by way of example that the video display system 100 is an HMD used in MR technology.

  FIG. 2 shows various members used in work using the image display system 100. The video display system 100 supports the user's work by displaying a virtual object on the display 140 corresponding to the arrangement of the work object 121 which is a real object arranged in the real space. "Arrangement" means a state of six degrees of freedom defined by coordinates on a predetermined three-dimensional coordinate axis and rotation about each of the three-dimensional coordinate axes (direction on three-dimensional coordinate). Specifically, the video display system 100 performs imaging in the same imaging direction as the user's gaze. The image display system 100 superimposes and displays a virtual object on an image captured. By viewing this display, the user can view a virtual object that is not realistic according to the work object 121 placed in the real space. The virtual object may include one representing the work object 121 in the real space.

  The image display system 100 displays a virtual object using a virtual space (screen coordinate system) which is a three-dimensional coordinate. The video display system 100 arranges a virtual object at a preset position in the virtual space, and calculates the correspondence between the virtual space and the real space. The arrangement of virtual objects in the virtual space is called display arrangement. In the following description, the coordinate axis indicating the position of the virtual object in the virtual space is called virtual coordinate axis α. The image display system 100 displays an image viewed from the position and direction in the virtual space corresponding to the imaging position and the imaging direction in the real space as the display of the virtual object. That is, an image in which the virtual object in the virtual space is viewed from the virtual user's line of sight is displayed. The display of the virtual object described above can be implemented using conventional MR techniques.

  In the present embodiment, the work object 121 is a steel material having a long shape of 10 m or more. The image display system 100 displays a virtual object 201 indicating the steel material and a virtual object 202 indicating an iron plate to be welded to the steel material on the steel material (work object 121) disposed in the real space. Specifically, the video display system 100 displays the virtual object 201 so that the virtual object 201 is visually recognized as being superimposed on the work object 121 disposed in the real space, and the position where the virtual object 202 should be welded Display to be visible when placed in Thereby, the image display system 100 can support the operation of welding the iron plate to the steel material by the user. FIG. 3 shows the display of the image display system 100 and the surrounding environment before moving to a place where the user moves and works (hereinafter referred to as “work position”). FIG. 4 shows the display of the image display system 100 and the surrounding environment after the user moves and before the display arrangement of the virtual object is corrected.

  The image display system 100 detects the movement of the user, and changes the display according to the movement so as to match the change in the line of sight according to the movement of the user. In the image display system 100, errors in movement detection accumulate according to the amount of movement of the user, and the display arrangement of the virtual objects 201 and 202 is shifted corresponding to the work object 121 (that is, correspondence between real space and virtual space) Misplaced). The image display system 100 displays and arranges the virtual objects 201 and 202 with respect to the work object 121 by the reference marker 122, the index member 123, and the second sub marker 127, which are arranged in the real space with respect to the work object 121. Correct the FIG. 5 shows the display of the image display system 100 after the user moves to the operation position and arranges the second sub-marker 127 and before the display arrangement of the virtual object is corrected.

  The reference marker 122 can be attached to the work object 121, and is disposed at a predetermined position relative to the work object 121 before the work is performed. In the present embodiment, the reference marker 122 is provided on the surface 121 a of the work object 121. The fiducial marker 122 is imaged by the imaging device 101 described later, so that the image display system 100 has a feature or the like capable of identifying the three-dimensional arrangement of the fiducial marker 122 in the real space by image recognition (for example, two-dimensional Code). The reference marker 122 is not limited to this form as long as the image display system 100 can identify the three-dimensional arrangement of the reference marker 122 in real space.

  The index member 123 is disposed at a predetermined position with respect to the work object 121 and the reference marker 122. In the present embodiment, the index member 123 has an elongated shape, and is disposed along the longitudinal direction of the work object 121 on the surface 121 a of the work object 121. The index member 123 includes a plurality of indexes 124 disposed at a predetermined position with respect to the reference marker 122 in real space. The plurality of indicators 124 include a plurality of first sub markers 125 and a plurality of sub indicators 126.

  In the present embodiment, the index member 123 is a tape measure. The plurality of indexes 124 have a rectangular shape, and are attached every 10 cm on a straight line extending from the reference point M indicated by the reference marker 122. The plurality of first sub markers 125 are attached every 1 m on a straight line extending from the reference point M indicated by the reference marker 122. Thus, the first sub markers 125 are provided on the index member 123 at predetermined intervals (for example, at equal intervals). The sub index 126 is an index 124 other than the first sub marker 125. Therefore, the plurality of sub-indexes 126 are attached every 10 cm between adjacent first sub-markers 125. Thus, the sub-indexes 126 are provided on the index member 123 at predetermined intervals (for example, equal intervals) narrower than the interval of the first sub-markers 125.

  The first sub marker 125 has a graphic or the like having a feature capable of identifying the three-dimensional arrangement of the first sub marker 125 in the real space by image recognition. Also, all the first sub markers 125 may be the same figure or the like. In the present embodiment, the first sub marker 125 is shown in red, and the sub index 126 is shown in black. The index member 123 may be a string attached with the first submarker 125 and the subindex 126 at the same interval. As the first sub marker 125 and the sub index 126, a memory display provided on a normal tape measure can be used. That is, a normal tape measure can be used as the index member 123. The first sub-marker 125 and the sub-index 126 may be marked on the work object 121 with a laser, a choke or the like. The first sub-marker 125 and the sub-index 126 may be displayed on the work object 121 by projection mapping.

  The second sub marker 127 has a graphic or the like (for example, a two-dimensional code) having a feature capable of identifying the three-dimensional arrangement of the second sub marker 127 in the real space by image recognition. In addition, the second sub marker 127 has a feature that image recognition is easier than the first sub marker 125 (there are more feature points than the second sub marker 127), and the second sub marker 127 can move relative to the work object 121 and the reference marker 122 It is. The second sub marker 127 is not limited to this form as long as the video display system 100 can identify the three-dimensional arrangement of the second sub marker 127 in the real space. When the video display system 100 identifies the three-dimensional arrangement of the second sub-marker 127, the image display system 100 moves from the information stored in advance to a specific position (hereinafter referred to as "designated point A") according to the three-dimensional arrangement of the second sub-marker 127. , The second virtual position described later is associated.

  In the present embodiment, a user who is a worker carries the second sub-marker 127 to the work position, and places the second sub marker 127 at the position of the index 124 closest to the work position. At this time, the designated point A of the second sub-marker 127 is disposed at the position of the index 124 closest to the user (the imaging device 101). The second sub-marker 127 may have a display description (for example, a display of an arrow) for alignment with the sub-index 126 so that positioning with respect to the sub-index 126 can be easily performed. When the second sub marker 127 is disposed on the sub index 126, the second sub marker 127 is closer to the imaging device 101 than any of the first sub markers 125.

  As shown in FIG. 1, the video display system 100 includes an imaging device 101, a movement detection unit 102 that detects movement of the imaging device 101, a data storage unit 103, a display unit 104, an operation unit 105, and a display control unit. It has 150. At least the imaging device 101, the movement detection unit 102, and the display unit 104 in the video display system 100 can be worn by the user.

  The imaging device 101 captures an image of the real space, and sequentially transmits the captured image to the display control unit 150. In the present embodiment, the user's head is fixed to the user's head, and the position and orientation of the imaging apparatus 101 change according to the movement of the user's head. The position of the imaging device 101 is the coordinate position of the imaging device 101. The orientation of the imaging device 101 is rotation (direction on three-dimensional coordinates) around each axis of three-dimensional coordinates. The imaging device 101 includes a plurality of cameras such as a depth camera and an RGB camera.

  The data storage unit 103 stores various information (CG data) necessary for displaying the virtual objects 201 and 202 in advance. For example, the data storage unit 103 includes information on the display arrangement of the virtual objects 201 and 202, the plurality of first virtual positions 131, and the plurality of second virtual positions 132 previously associated with the virtual coordinate axis α in the virtual space. Are stored in advance. Specifically, the data storage unit 103 stores the display arrangement of the virtual objects 201 and 202, the plurality of first virtual positions 131, and the coordinates of the plurality of second virtual positions 132 on the virtual coordinate axis α. By associating the virtual coordinate axis α with the real space, the display arrangement of the virtual objects 201 and 202 corresponding to the position and posture of the imaging device 101 in the real space, the plurality of first virtual positions 131, and the plurality of second virtual positions 132 Is determined. The plurality of first virtual positions 131 and the plurality of second virtual positions 132 are used for correcting the display arrangement of the virtual objects 201 and 202 (that is, correcting the association between the real space and the virtual space).

  The data storage unit 103 also stores, in advance, an arrangement relationship between a predetermined reference marker 122 disposed in the real space and the virtual coordinate axis α. In the present embodiment, the position of the reference point M in the real space of the reference marker 122 and the origin N of the virtual coordinate axis α are associated with each other. That is, the arrangement of the reference marker 122 in the real space is associated with the origin N of the virtual coordinate axis α.

  The relative positional relationship between the origin N of the virtual coordinate axis α and the plurality of first virtual positions 131 corresponds to the relative positional relationship between the reference point M of the reference marker 122 and the plurality of first sub markers 125. For example, their relative positional relationship is the same. In the present embodiment, the plurality of first virtual positions 131 are positioned on a straight line from the origin point N of the virtual coordinate axis α at a distance corresponding to 1 m of the real space. The relative positional relationship between the origin N of the virtual coordinate axis α and the plurality of second virtual positions 132 corresponds to the relative positional relationship between the reference point M of the reference marker 122 and the plurality of sub-indexes 126. For example, their relative positional relationship is the same.

  The plurality of second virtual positions 132 are associated with the virtual coordinate axis α at an interval narrower than the intervals of the plurality of first virtual positions 131. In the present embodiment, the plurality of second virtual positions 132 are positioned between adjacent first virtual positions 131 at a distance corresponding to 10 cm of the real space. The number of second virtual positions 132 is greater than the number of first virtual positions 131. In addition, the format of the information concerning the above-mentioned virtual space can use what is used by the conventional MR technique.

  The display unit 104 displays a virtual image including the virtual objects 201 and 202 on the display 140 according to the position and orientation of the imaging device 101 as illustrated in FIGS. 3 to 5. The display unit 104 uses the conventional method from the arrangement information such as the display arrangement of the virtual objects 201 and 202 stored in the data storage unit 103 and the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101. The position at which the virtual objects 201 and 202 are displayed on the display 140 is calculated. The display unit 104 acquires various types of information necessary for displaying the virtual objects 201 and 202 from the data storage unit 103. The display by the display unit 104 is to display an image viewed from the position and direction in the virtual space corresponding to the imaging position and the imaging direction in the real space as a display of a virtual object.

  In the present embodiment, the display unit 104 includes, as virtual images, a virtual object 201 indicating the steel material, a virtual object 202 indicating an iron plate to be welded to the steel material, a first virtual position 131, and a second virtual position 132. Display The arrangement of the virtual coordinate axis α with respect to the real space, and the position and orientation of the imaging device 101 are determined by the display control unit 150. The display unit 104 is fixed to the user's head in a state in which the user can appropriately view the virtual objects 201 and 202.

  When operated by the user, the operation unit 105 sequentially transmits a signal corresponding to the operation to the display control unit 150. For example, the operation unit 105 transmits a signal instructing correction of the display arrangement of the virtual objects 201 and 202 to the display control unit 150 according to the user's operation.

  The movement detection unit 102 detects the movement of the imaging apparatus 101 and sequentially transmits the detection result to the display control unit 150. The video display system 100 displays a video following the movement of the user based on the detection result. For example, the movement detection unit 102 detects a movement vector indicating the direction of movement of the imaging device 101 and the amount of movement. In the present embodiment, the movement detection unit 102 is configured by an inertial measurement unit fixed to the head of the user and having an acceleration sensor, a gyro sensor, an orientation sensor, and the like. In the present embodiment, the movement detection unit 102 itself detects movement. The movement detection unit 102 may transmit an output signal of an inertial measurement unit or the like to the display control unit 150, and the display control unit 150 may calculate movement. Movement detection can be performed using methods used in conventional MR techniques.

  The display control unit 150 is operated by the virtual coordinate axis α defining the display arrangement of the virtual objects 201 and 202 based on the image captured by the imaging device 101 (hereinafter referred to as “captured image”), and the position and orientation operation unit 152 The relationship between the detected position and orientation of the imaging apparatus 101 is determined. Thus, the display arrangement of the virtual objects 201 and 202 on the display unit 104 is determined. The display control unit 150 includes a reference layout setting unit 151, a position and orientation calculation unit 152, a first positional relationship calculation unit 153, a second positional relationship calculation unit 154, and a correction unit 155.

  The reference layout setting unit 151 associates the virtual coordinate axis α that defines the display layout of the virtual objects 201 and 202 with the real space based on the captured image of the imaging device 101. Specifically, the reference arrangement setting unit 151 detects the reference marker 122 from the captured image of the imaging device 101. The reference arrangement setting unit 151 associates the virtual coordinate axis α with the arrangement of the reference marker 122 in the real space from the detection result of the reference marker 122 and the information stored in the data storage unit 103. In the present embodiment, in accordance with the information stored in the data storage unit 103, the position of the preset end (reference point M) of the detected reference marker 122 is associated with the origin N of the virtual coordinate axis α. . Further, the direction in the real space is associated with the direction of the virtual coordinate axis α based on the shape of the detected reference marker 122 and the like. As a result, the display arrangement of the virtual objects 201 and 202, the first virtual position 131, and the second virtual position 132 are associated with the arrangement of the reference marker 122 in the real space.

  The position and orientation calculation unit 152 calculates the position and orientation of the imaging device 101. When the power of the image display system 100 is switched on, the position and orientation calculation unit 152 sets an initial state of the position and orientation of the imaging device 101. For example, the position and orientation calculation unit 152 detects a feature point from the captured image of the imaging device 101, and sets the feature point as an origin in the real space. The position and orientation calculation unit 152 calculates the initial state of the position and orientation of the imaging device 101 from the mutual arrangement of the origin and the imaging device 101 and the outputs of the acceleration sensor, the gyro sensor, and the like. In the present embodiment, the reference point M indicated by the reference marker 122 will be described as the origin in real space for calculating the initial state of the position and orientation of the imaging device 101. Further, the position and orientation calculation unit 152 calculates, from the image, the position and orientation of the imaging device 101 with respect to the set origin.

  The position and orientation calculation unit 152 sequentially calculates the position and orientation of the imaging device 101 from the movement detected by the movement detection unit 102 and the transition vector of the feature point detected from the captured image of the imaging device 101. Specifically, the position and orientation calculation unit 152 calculates the position and orientation of the imaging device 101 after movement (that is, the current position and orientation of the imaging device 101). When the reference marker 122 is not detected from the captured image, the display control unit 150 determines the relationship between the virtual coordinate axis α in the initial state and the position and orientation of the imaging device 101, and the position of the imaging device 101 calculated by the position and orientation calculation unit 152. The arrangement of the virtual coordinate axis α in the real space is sequentially determined from the attitude and the posture.

  In the present embodiment, the position and orientation calculation unit 152 sequentially calculates the position and orientation of the imaging device 101 with respect to the reference point M which is an end of the reference marker 122 detected by the reference arrangement setting unit 151. The position and orientation calculation unit 152 may calculate the position and orientation of the imaging device 101 only from the detection result of the movement detection unit 102. The position and orientation calculation unit 152 stores the calculated data of the position and orientation of the imaging device 101 in the data storage unit 103. The position and orientation calculation unit 152 may transmit the calculated data directly to the display unit 104. The error of the calculation result in the position and orientation calculation unit 152 is accumulated according to the increase of the movement amount of the user. As the error of the calculation result in the position and posture calculation unit 152 is accumulated, the deviation of the position and orientation of the imaging device 101 calculated by the position and posture calculation unit 152 with respect to the real space, and the deviation of the arrangement of the virtual coordinate axis α with respect to the real space Will increase. As a result, in the video display system 100, the display arrangement of the virtual objects 201 and 202 is shifted. The calculation of the position and orientation of the imaging device 101 can be performed using a method used in conventional MR techniques.

  The display control unit 150 causes the first positional relationship calculation unit 153, the second positional relationship calculation unit 154, and the correction unit 155 to perform a shift correction process described later. The display control unit 150 corrects the deviation of the display arrangement of the virtual objects 201 and 202 due to the movement detection error accumulated according to the movement amount of the user by the deviation correction processing. In the present embodiment, the display control unit 150 performs the correction according to the operation of the operation unit 105. For example, the display control unit 150 performs the correction at the timing when the operation unit 105 such as pressing of a button is operated. FIGS. 4 and 5 show a state in which the virtual objects 201 and 202 are displayed shifted with respect to the work object 121.

  The first positional relation operation unit 153 derives the relation between the first sub marker 125 in the real space and the virtual coordinate axis α. First, the first positional relation operation unit 153 detects the first sub-marker 125 disposed in the real space from the captured image of the imaging device 101. In the present embodiment, the first positional relationship calculation unit 153 obtains the coordinate position of the first submarker 125 on the virtual coordinate axis α from the position and orientation of the imaging device 101 calculated by the position and posture calculation unit 152 and the captured image. .

  The captured image in which the first sub marker 125 is detected is different from the captured image in which the reference marker 122 is detected. For example, the captured image is an image captured at a position at which the user has moved from the position where the reference marker 122 is installed to perform work. For example, the first positional relation operation unit 153 searches the image captured by the imaging device 101 for a pixel having a red component equal to or greater than the threshold value according to the conventional method. The first positional relationship calculation unit 153 may search for a pixel whose red component is relatively high with respect to surrounding pixels from the captured image. The first positional relation operation unit 153 may notify the user of an error through the display 140 or the like when the number of detected first sub markers 125 is larger than the threshold.

  The first positional relationship calculation unit 153 generates a first virtual position 131 and a first virtual position 131 from the detection result of the first submarker 125 and the position and orientation of the imaging device 101 when the imaging device 101 captures the first submarker 125. The positional relationship with the 1 sub marker 125 is calculated. The position and orientation calculation unit 152 calculates the position and orientation of the imaging device 101 when the first sub marker 125 is imaged. First, the first positional relation operation unit 153 obtains the coordinate position in the real space of the first sub marker 125 searched from the captured image from the position and attitude of the imaging device 101, and further, from the coordinates in the real space, The coordinate position on the virtual coordinate axis α is determined. The calculation of the coordinate position on the real space and virtual coordinate axis α of the first submarker 125 from the searched first submarker 125 can be performed using a method used in the conventional MR technique.

  Subsequently, the first positional relationship computing unit 153 computes the positional relationship between the coordinate position of the first submarker 125 on the virtual coordinate axis α and the first virtual position 131 closest to the first submarker 125. The first positional relation operation unit 153 calculates the amount and direction of deviation of the first virtual position 131 with respect to the first sub-marker 125 on the virtual coordinate axis α as the positional relation between the first virtual position 131 and the first submarker 125. Here, the calculated displacement amount is the displacement amount in the longitudinal direction of the work object 121. That is, the correction using the first sub marker 125 is performed in the longitudinal direction of the work object 121 (the direction other than that is not corrected). However, the correction using the first sub marker 125 may be performed in all directions. The amount of displacement and the direction of displacement are the amount and direction from the coordinate position of the first submarker 125 on the virtual coordinate axis α in the longitudinal direction of the work object 121 to the first virtual position 131 closest to the first submarker 125 . The deviation direction (correction direction) is a line connecting the origin (coordinate position of the reference marker 122) (P0) on the virtual coordinate axis α and the coordinate position (P1 n) of the first sub marker 125, and the coordinate position of the first sub marker 125 It may be indicated by a cos value of an angle θ formed by a line connecting (P 1 n) and the first virtual position 131 (P 2 n) closest to the first sub marker 125. For example, when the formed angle θ satisfies 90 ° <θ <180 °, the sign of the value of cosθ is minus. In this case, it is determined that the first virtual position 131 is deviated in the negative direction relative to the first sub marker 125. The determination of the shift direction based on the sign of the value of cos θ described above is performed for each of the first sub markers 125 detected from the captured image. When the determination result of the shift direction is divided with respect to the first sub marker 125 detected from the captured image, the result in which the determined number is large is adopted.

  The first positional relation operation unit 153 may filter the plurality of first sub markers 125 searched from the image. Among the plurality of first sub-markers 125 searched from the image, the first sub-marker 125 that is considered not to correspond to the actual first sub-marker 125 may be excluded. As described above, the virtual coordinate axis α corresponds to the real space using the reference marker 122 or the like. Also, the moving direction of the user moves in the direction in which the first sub markers 125 are aligned. Therefore, even if errors occur in the position and orientation of the imaging device 101, the direction in which the position coordinates of the first submarker 125 on the virtual coordinate axis α are aligned with the direction in which the first virtual positions 131 are aligned are largely different. It is thought that it is impossible. Therefore, for example, the first positional relation operation unit 153 can connect the line connecting the first sub-marker 125 searched for in the image and the first sub-marker 125 closest to the first sub-marker 125 to each of the first sub-markers 125 The value of cos θ is calculated, where θ is an angle formed by a line connecting the close first virtual positions 131. The first positional relation operation unit 153 extracts only the first sub marker 125 in which the value of cos θ satisfies 1.0 to 0.8 or −1.0 to −0.8. Thereby, the erroneously detected data can be excluded.

  The second positional relation operation unit 154 derives the relation between the second sub marker 127 in the real space and the virtual coordinate axis α. First, the second positional relation operation unit 154 detects the second sub-marker 127 disposed in the real space from the captured image of the imaging device 101 by the conventional method. The captured image in which the second sub marker 127 is detected is different from the captured image in which the first sub marker 125 is detected and the captured image in which the reference marker 122 is detected. For example, the captured image is an image captured at a position at which the user moves to perform work and the second sub marker 127 is installed. In the present embodiment, the second positional relation operation unit 154 operates in real space of the second sub-marker 127 detected from the captured image from the position and orientation of the imaging device 101 computed by the position and orientation calculation unit 152 and the captured image. The coordinate position is determined, and the coordinate position on the virtual coordinate axis α is further determined from the coordinates in the real space. From the detected second sub-marker 127, calculation of coordinate positions on the real space and virtual coordinate axis α of the second sub-marker 127 can be performed using a method used in conventional MR technology.

  The second positional relation operation unit 154 detects a plurality of second virtual positions 132 from the detection result of the second sub marker 127 and the position and orientation of the imaging device 101 when the imaging device 101 captures an image of the second sub marker 127. The positional relationship between one of them and the second sub marker 127 is calculated. The position and orientation calculation unit 152 calculates the position and orientation of the imaging device 101 when the second sub marker 127 is imaged. When the second positional relation operation unit 154 determines that the second sub marker 127 is disposed at the position of the first sub marker 125, the second positional relation operation unit 154 does not calculate the positional relation between the second virtual position 132 and the second sub marker 127.

  Specifically, the second positional relation operation unit 154 calculates the second virtual position closest to the second sub marker 127 in the second sub marker 127 and the deviation amount and deviation direction calculated by the first positional relation operation unit 153. Calculate the positional relationship with 132. When a plurality of deviation amounts are calculated by the first positional relationship calculation unit 153, values such as their average value or median value may be used. Specifically, first, the second positional relation operation unit 154 moves the coordinate position of the second sub-marker 127 on the virtual coordinate axis α so as to cancel the above-mentioned deviation amount and deviation direction. The second positional relation operation unit 154 sets the actual position of the second virtual position 132 closest to the second sub marker 127 after movement relative to the second sub marker 127 as the positional relation between the second virtual position 132 and the second sub marker 127 after movement. The amount of displacement and the direction of displacement in space are calculated. The second virtual position 132 closest to the moved second sub-marker 127 at this time may be the closest second virtual position 132 in the displacement direction calculated by the first positional relation calculation unit 153. The second positional relation operation unit 154 calculates the coordinate position of the second sub marker 127 after movement and the second virtual position calculated from the captured image of the imaging device 101 in the amount and direction of displacement of the second virtual position 132 in the real space. The difference from the coordinate position of 132 is used.

  The correction unit 155 corrects the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 calculated by the position and orientation calculation unit 152. In the present embodiment, the correction unit 155 determines the arrangement of the virtual coordinate axis α with respect to the real space based on the positional relationship calculated by the first positional relationship calculation unit 153 and the positional relationship calculated by the second positional relationship calculation unit 154. Fix it. Specifically, the correction unit 155 determines the position of the imaging device 101 based on the shift direction and the shift amount calculated by the first positional relationship calculation unit 153 and the shift direction and the shift amount calculated by the second positional relationship calculation unit 154. The display arrangement of the first virtual position 131, the second virtual position 132, and the virtual objects 201 and 202 is moved without moving the posture. As a result, the position and orientation of the imaging device 101 on the virtual coordinate axis α with respect to the position and orientation of the imaging device 101 on the virtual coordinate axis α are corrected.

  When the second positional relation operation unit 154 determines that the second sub marker 127 is disposed at the position of the first sub marker 125, the correction unit 155 calculates the positional relation calculated by the second positional relation operation unit 154. Correct the arrangement of the virtual coordinate axis α with respect to the real space without using

  When the correction is performed by the correction unit 155, the display unit 104 calculates the virtual coordinate axis α and the position and orientation calculation unit 152 corrected by the arrangement information stored in the data storage unit 103 and the correction unit 155. The virtual objects 201 and 202 are displayed on the display 140 in the conventional manner based on the relationship between the position and orientation of the imaging device 101. Therefore, when the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 is corrected, the arrangement of the virtual objects 201 and 202 displayed on the display 140 is also corrected.

  The correction unit 155 may correct the position and orientation of the imaging device 101 calculated by the position and orientation calculation unit 152 instead of the arrangement of the virtual coordinate axis α with respect to the real space. Specifically, the correction unit 155 is virtual so as to cancel the shift direction and the shift amount calculated by the first positional relationship calculation unit 153 and the shift direction and the shift amount calculated by the second positional relationship calculation unit 154. The position and orientation of the imaging device 101 in space may be moved. The correction unit 155 may correct both of the virtual coordinate axis α with respect to the real space and the position and orientation of the imaging device 101 calculated by the position and orientation calculation unit 152.

  In the present embodiment, the correction unit 155 corrects the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 from the positional relationship calculated by the first positional relationship calculation unit 153, and then the second positional relationship calculation unit From the positional relationship calculated by 154, the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 is corrected. Specifically, when the correction unit 155 corrects the above-described relationship from the positional relationship calculated by the first positional relationship calculation unit 153, the virtual unit with respect to the real space is in the displacement direction determined by the first positional relationship calculation unit 153. Correct the arrangement of the coordinate axis α. The first positional relation operation unit 153 makes the total of the distances between each of the first sub markers 125 detected from the captured image and each of the first virtual positions 131 corresponding to the first sub marker 125 to be minimum. , Correct the arrangement of the virtual coordinate axis α with respect to the real space. In this case, the total is minimized by shifting the arrangement of the virtual coordinate axis α with respect to the real space in the shift direction determined by the first positional relationship calculation unit 153 by a predetermined distance corresponding to the shift amount.

  If the deviation between each of the first sub markers 125 and each of the first virtual positions 131 corresponding to the first sub markers 125 is less than half the interval of the first sub markers 125, the second positional relation operation unit 154 From the calculated positional relationship, the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 calculated by the position and orientation calculation unit 152 can be appropriately corrected. That is, if the deviation of the position and orientation of the imaging device 101 is less than half (less than 50 cm in this embodiment) of the interval of the first submarker 125 (the period of the second submarker candidate), the position and orientation of the imaging device 101 The relationship with can be properly corrected. If the deviation of the position and orientation of the imaging device 101 is more than half the interval of the first submarker 125 (the period of the second submarker candidate), the video display system 100 transmits an error to the user through the display 140 or the like. The image pickup apparatus 101 prompts the image pickup apparatus 101 to image in the vicinity of the one sub marker 125 in detail.

  The correction unit 155 corrects the deviation of the display arrangement of the virtual objects 201 and 202 in the longitudinal direction, height direction, and width direction of the work object 121 from the three-dimensional arrangement information of the second sub-marker 127. Can. The correction unit 155 may correct the rotation around each direction based on the second sub marker 127.

  The correction unit 155 calculates the imaging device calculated by the virtual coordinate axis α and the position and orientation calculation unit 152 such that the second virtual position 132 closest to the second sub marker 127 is the position of the sub index 126 corresponding to the second sub marker 127. Correct the relationship between the position 101 and the posture. The position of the sub-index 126 where the second sub-marker 127 is arranged is the position of the sub-index 126 closest to the work position. Even when the correction unit 155 corrects the relationship based on the positional relationship calculated by the second positional relationship calculation unit 154, the arrangement of the virtual coordinate axis α with respect to the real space in the shift direction determined by the first positional relationship calculation unit 153. Correct the

  In the present embodiment, the indicators 124 are provided every 10 cm from the origin indicated by the reference marker 122 in the longitudinal direction of the work object 121. Therefore, the second sub marker 127 is disposed at a position corresponding to a multiple of 10 cm from the origin. Therefore, for example, when it is determined by the second positional relation operation unit 154 that the second sub marker 127 is disposed at a position of 1242 cm from the origin, the second positional relation operation unit 154 selects the second sub marker 127. Is determined to be deviated from the second virtual position 132 by 2 cm. In this case, the correction unit 155 sets the position of the virtual coordinate axis α in the real space to 2 cm so that the second positional relation operation unit 154 determines that the second sub marker 127 is arranged at a position of 1240 cm from the reference marker 122. Fix it. As a result, the second virtual position 132 closest to the second sub marker 127 is corrected to be the position of the sub index 126 corresponding to the second sub marker 127, and the virtual objects 201 and 202 are also displayed on the display 140 in an appropriate arrangement. Ru.

  Next, with reference to FIG. 6, the process of correcting the display arrangement of the virtual objects 201 and 202 in the video display system 100 will be described. FIG. 6 is a flowchart showing the process of correcting the display arrangement of the virtual objects 201 and 202. In the following processing, the above-described processing by the imaging apparatus 101 and the movement detection unit 102 is sequentially performed.

  In step S101, the user arranges the reference marker 122 and the index member 123 (the index 124 including the first sub marker 125 and the sub index 126) on the work object 121. Subsequently, the process proceeds to step S102.

  In step S102, the display control unit 150 associates the virtual coordinate axis α with the real space according to the reference marker 122. Subsequently, the process proceeds to step S103.

  In step S103, the user moves to the work position, and places the second sub-marker 127 at the position of the index 124 closest to the work position. Subsequently, the process proceeds to step S104.

  In step S104, the user operates the operation unit 105 to instruct correction of the display arrangement of the virtual objects 201 and 202. Subsequently, the process proceeds to step S105.

  In step S105, the display control unit 150 causes the first positional relationship calculation unit 153, the second positional relationship calculation unit 154, and the correction unit 155 to set the virtual object 201, based on the arrangement of the first sub marker 125 and the second sub marker 127. A correction process of the display arrangement 202 is performed.

  Next, control processing of display arrangement of the virtual objects 201 and 202 by the display control unit 150 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing control processing performed by the display control unit 150.

  In step S111, the display control unit 150 causes the reference arrangement setting unit 151 to detect the reference marker 122 from the captured image of the imaging device 101. Subsequently, the display control unit 150 advances the process to step S112.

  In step S112, the display control unit 150 causes the reference arrangement setting unit 151 to arrange the reference marker 122 in real space based on the detection result of the reference marker 122 and the information stored in the data storage unit 103. Correspond to α. Subsequently, the display control unit 150 advances the process to step S113.

  In step S113, the display control unit 150 causes the position and orientation calculation unit 152 to calculate the position and orientation of the imaging device 101. The calculation result is stored in the data storage unit 103. Subsequently, the display control unit 150 causes the process to proceed to step S114.

  In step S114, the display control unit 150 causes the display unit 104 to display a virtual image on the display 140. At this time, the display unit 104 calculates the position at which the virtual objects 201 and 202 are displayed on the display 140 with reference to the data storage unit 103. Subsequently, the display control unit 150 causes the process to proceed to step S115.

  In step S115, the display control unit 150 determines whether an instruction to correct the display arrangement of the virtual objects 201 and 202 has been received from the operation unit 105. If a correction instruction has been received, the display control unit 150 advances the process to step S116. If the correction instruction has not been received, the display control unit 150 advances the process to step S117.

  In step S116, the display control unit 150 causes the first positional relationship calculation unit 153, the second positional relationship calculation unit 154, and the correction unit 155 to perform a shift correction process. The displacement correction process corrects the displacement of the display arrangement of the virtual objects 201 and 202 due to the error of the movement detection accumulated according to the movement amount of the user. When the shift correction process is completed, the display control unit 150 advances the process to step S113.

  In step S117, the display control unit 150 determines whether to end the control processing. If the control process is not ended, the process proceeds to step S113. If the control process is to be ended, the process is ended.

  Next, the shift correction processing by the display control unit 150 will be described in detail with reference to FIG. FIG. 8 is a flowchart showing the shift correction process performed by the display control unit 150.

  In step S121, the display control unit 150 causes the first positional relation operation unit 153 to detect the first sub-marker 125 from the captured image of the imaging device 101. Subsequently, the display control unit 150 advances the process to step S122.

  In step S122, the display control unit 150 causes the first positional relationship calculation unit 153 to calculate the positional relationship between the first virtual position 131 and the first sub marker 125. The first positional relation operation unit 153 determines the plurality of first virtual positions 131 from the detection result of the first submarker 125 and the position and orientation of the imaging device 101 when the imaging device 101 captures an image of the first submarker 125. The positional relationship between one of them and the first sub marker 125 is calculated. Subsequently, the display control unit 150 proceeds with the process to step S123.

  In step S123, the display control unit 150 causes the correction unit 155 to correct the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 calculated by the position and orientation calculation unit 152. In step S123, the correction unit 155 determines the position of the imaging device 101 from the virtual coordinate axis α based on the positional relationship calculated by the first positional relationship calculation unit 153, that is, the positional relationship between the first virtual position 131 and the first submarker 125. And correct the relationship with attitude. Subsequently, the display control unit 150 advances the process to step S124.

  In step S124, the display control unit 150 causes the second positional relation operation unit 154 to detect the second sub marker 127 from the captured image of the imaging device 101. Subsequently, the display control unit 150 proceeds with the process to step S125.

  In step S125, the display control unit 150 causes the second positional relation operation unit 154 to determine whether the second sub marker 127 is disposed at the position of the first sub marker 125 or not. If it is determined that the second sub marker 127 is disposed at the position of the first sub marker 125, the display control unit 150 ends the deviation correction processing. If it is determined that the second sub marker 127 is not arranged at the position of the first sub marker 125, the display control unit 150 advances the process to step S126.

  In step S 126, the display control unit 150 causes the second positional relation operation unit 154 to calculate the positional relation between the second virtual position 132 and the second sub marker 127. The second positional relation operation unit 154 detects a plurality of second virtual positions 132 from the detection result of the second sub marker 127 and the position and orientation of the imaging device 101 when the imaging device 101 captures an image of the second sub marker 127. The positional relationship between one of them and the second sub marker 127 is calculated. Subsequently, the display control unit 150 proceeds with the process to step S127.

  In step S127, the display control unit 150 causes the correction unit 155 to correct the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 calculated by the position and orientation calculation unit 152. In step S 126, the correction unit 155 determines the position of the imaging device 101 from the virtual coordinate axis α based on the positional relationship calculated by the second positional relationship calculation unit 154, that is, the positional relationship between the second virtual position 132 and the second sub marker 127. And correct the relationship with attitude. When the process of step S127 ends, the display control unit 150 ends the deviation correction process.

  In the present embodiment, as shown in FIG. 7, the display control unit 150 receives the correction instruction according to the operation of the operation unit 105 and starts the deviation correction process. However, the display control unit 150 does not have to use the correction instruction from the operation unit 105 as a trigger for performing the deviation correction process. For example, the display control unit 150 may be set to perform the shift correction process every predetermined time.

  The display control unit 150 may set the process started in response to the correction instruction to any one of steps S121 to S126 in the deviation correction process. For example, the display control unit 150 corrects the correction instruction from the operation unit 105 after correcting the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 based on the positional relationship between the first virtual position 131 and the first submarker 125. Based on the positional relationship between the second virtual position 132 and the second sub marker 127, the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 may be corrected.

  Next, main actions and effects of the video display system 100 according to the present embodiment will be described.

  In the image display system 100, the first positional relation operation unit 153 includes a first sub marker 125 disposed at a predetermined position with respect to the reference marker 122 and a plurality of first virtuals previously associated with the virtual coordinate axis α. The positional relationship between the first sub marker 125 and the closest first virtual position 131 among the positions 131 is calculated. Therefore, when the position and orientation of the imaging device 101 calculated from the detection result by the movement detection unit 102 include an error, the position away from the reference marker 122 also corresponds to the arrangement of the plurality of first virtual positions 131. Thus, the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 may be corrected, and the display arrangement of the virtual objects 201 and 202 may be corrected. Specifically, as described above, the display arrangement of the virtual objects 201 and 202 can be corrected even if there is an error in the position and orientation of the imaging device 101, as long as the distance between the first virtual positions 131 is about half of that. . Since the first sub-marker 125 only performs rough alignment, it is not always necessary to detect the exact position. For example, as the first sub marker 125, it may be difficult to perform image recognition as compared to the reference marker 122 and the second sub marker 127. For example, the first sub marker 125 may have fewer feature points. Therefore, a memory of a tape measure as in this embodiment can be used as the first sub marker 125, and alignment can be performed simply and appropriately.

  The second positional relation operation unit 154 includes a plurality of second sub markers 127 disposed at positions closer to the imaging device 101 than the first sub marker 125 and a plurality of elements corresponding to the virtual coordinate axis α in advance. The positional relationship with the second virtual position 132 is calculated. By using the positional relationship between the second sub-marker 127 closer to the imaging device 101 than the first sub-marker 125 and the second virtual position 132 while using the positional relationship between the first sub-marker 125 and the first virtual position 131, At the position of the imaging device 101, the deviation of the virtual objects 201 and 202 from the real space can be corrected with higher accuracy. That is, the display arrangement of the virtual objects 201 and 202 can be corrected with high accuracy at the position of the user.

  The correction unit 155 corrects the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 calculated by the position and posture calculation unit 152 from the positional relationship calculated by the first positional relationship calculation unit 153, Based on the positional relationship calculated by the positional relationship calculation unit 154, the above relationship is corrected. As described above, before performing the detailed correction using the positional relationship calculated by the second positional relationship calculating unit 154, the general correction is performed based on the positional relationship calculated by the first positional relationship calculating unit 153. Thus, the correction accuracy is improved while the processing load is reduced. In addition, by performing general correction while the user is moving, it is possible to reduce visual discomfort of the user.

  The positional relationship calculated by the first positional relationship calculation unit 153 includes the shift direction between the first sub marker 125 and the first virtual position 131 in the virtual space. One of the plurality of second virtual positions 132 is the second virtual position 132 closest to the second sub-marker 127 in the displacement direction calculated by the first positional relation operation unit 153. For this reason, the range of the amount of deviation which can be appropriately corrected can be improved.

  The operation unit 105 further includes an operation unit 105 operated by the user. At least one of the first position relation operation unit 153, the second position relation operation unit 154, and the correction unit 155 performs processing in accordance with the operation of the operation unit 105. Depending on the situation, the processing load may be reduced since only necessary processing can be performed.

  Note that the block diagram used in the description of the above embodiment shows blocks in units of functions. These functional blocks (components) are realized by any combination of hardware and / or software. Moreover, the implementation means of each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly two or more physically and / or logically separated devices. It may be connected by (for example, wired and / or wireless) and realized by the plurality of devices.

  For example, the video display system in one embodiment of the present invention may function as a computer that performs the processing in this embodiment. FIG. 9 is a diagram showing an example of a hardware configuration of a video display system according to an embodiment of the present invention. The above-described video display system 100 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007 and the like. In addition to the above, the video display system 100 also includes hardware such as an imaging device 101 and a sensor used for movement detection.

  In the following description, the term "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the video display system 100 may be configured to include one or more of the devices illustrated in the figure, or may be configured without including some devices.

  Each function in the image display system 100 causes the processor 1001 to perform an operation by reading predetermined software (program) on hardware such as the processor 1001, the memory 1002, etc., communication by the communication device 1004, a memory 1002 and a storage This is realized by controlling reading and / or writing of data in 1003.

  The processor 1001 operates, for example, an operating system to control the entire computer. The processor 1001 may be configured by a central processing unit (CPU: Central Processing Unit) including an interface with a peripheral device, a control device, an arithmetic device, a register, and the like. For example, the display unit 104, the reference layout setting unit 151, the position and orientation calculation unit 152, the first positional relationship calculation unit 153, the second positional relationship calculation unit 154, and the correction unit 155 may be realized by the processor 1001.

  Also, the processor 1001 reads a program (program code), a software module or data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processing according to these. As a program, a program that causes a computer to execute at least a part of the operations described in the above embodiments is used. For example, the display unit 104, the reference layout setting unit 151, the position and orientation calculation unit 152, the first positional relationship calculation unit 153, the second positional relationship calculation unit 154, and the correction unit 155 are stored in the memory 1002 and operated by the processor 1001. It may be realized by the control program which carries out, and may be realized similarly about other functional blocks. The various processes described above have been described to be executed by one processor 1001, but may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from the network via a telecommunication line.

  The memory 1002 is a computer readable recording medium, and includes, for example, at least one of a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). It may be done. The memory 1002 may be called a register, a cache, a main memory (main storage device) or the like. The memory 1002 can store a program (program code), a software module, etc. that can be executed to implement the method according to one embodiment of the present invention.

  The storage 1003 is a computer readable recording medium, and for example, an optical disc such as a CD-ROM (Compact Disc ROM), a hard disc drive, a flexible disc, a magneto-optical disc (for example, a compact disc, a digital versatile disc, Blu-ray A (registered trademark) disk, a smart card, a flash memory (for example, a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like may be used. The storage 1003 may be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database including the memory 1002 and / or the storage 1003, a server or any other suitable medium.

  The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, or the like. The video display system 100 may rewrite data in the data storage unit 103 by communication with the outside by the communication device 1004.

  The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, and the like) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated (for example, a touch panel).

  In addition, devices such as the processor 1001 and the memory 1002 are connected by a bus 1007 for communicating information. The bus 1007 may be configured by a single bus or may be configured by different buses among the devices.

  In addition, the video display system 100 includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). And part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented in at least one of these hardware.

  Although the present invention has been described above in detail, it is apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be embodied as modifications and alterations without departing from the spirit and scope of the present invention defined by the description of the claims. Accordingly, the description in the present specification is for the purpose of illustration and does not have any limiting meaning on the present invention.

  For example, in the embodiment described above, even when the correction unit 155 corrects the above-described relationship from the positional relationship calculated by the second positional relationship calculation unit 154, in the shift direction determined by the first positional relationship calculation unit 153. , Correct the arrangement of the virtual coordinate axis α with respect to the real space. However, the second positional relation operation unit 154 also determines the shift direction, and the second positional relation operation unit 154 corrects the relation based on the positional relation calculated by the second positional relation operation unit 154. The arrangement of the virtual coordinate axes α with respect to the real space may be corrected in the determined deviation direction.

  The correction unit 155 calculates the positional relationship between the first virtual position 131 and the first submarker 125 by the first positional relationship calculation unit 153, and the second virtual position 132 and the second submarker 127 by the second positional relationship calculation unit 154. The relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 may be corrected after computing the positional relationship of That is, the correction unit 155 does not perform the process of step S123 in FIG. 8 and after the process of step S126 in FIG. 8, the positional relationship between the first virtual position 131 and the first sub marker 125 and the second virtual position 132. Based on the positional relationship with the second sub marker 127, the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 may be corrected.

  In this case, the first positional relation calculation unit 153 may calculate only the displacement direction of the first virtual position 131 with respect to the first sub marker 125 as the positional relation between the first virtual position 131 and the first sub marker 125. . Further, in this case, the display control unit 150 may perform the shift correction process in step S117 in FIG. 7 using the arrangement of the second sub-marker 127 in step S103 as a trigger, without performing the process in step S104 in FIG.

  The second sub-marker 127 may indicate which index 124 is to be selected among the plurality of indices 124 without having information indicating the three-dimensional arrangement of the second sub-marker 127. In this case, the second positional relation operation unit 154 detects the selected indicator 124, and calculates the positional relationship between the indicator 124 and the second virtual position 132.

  The first positional relation operation unit 153 may not calculate the coordinate position of the first sub marker 125. In this case, the correction unit 155 corrects the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 so that the first sub-marker 125 and the first virtual position 131 coincide in the captured image of the imaging device 101. May be

  The second positional relation operation unit 154 may not calculate the coordinate position of the second sub marker 127. In this case, the correction unit 155 corrects the relationship between the virtual coordinate axis α and the position and orientation of the imaging device 101 so that the second sub marker 127 and the second virtual position 132 coincide with each other in the captured image of the imaging device 101. May be

  Each aspect / embodiment described in the present specification is LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (Registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide Band), The present invention may be applied to a system utilizing Bluetooth (registered trademark), other appropriate systems, and / or an advanced next-generation system based on these.

  As long as there is no contradiction, the processing procedure, sequence, flow chart, etc. of each aspect / embodiment described in this specification may be reversed. For example, for the methods described herein, elements of the various steps are presented in an exemplary order and are not limited to the particular order presented.

  The input / output information or the like may be stored in a specific place (for example, a memory) or may be managed by a management table. Information to be input or output may be overwritten, updated or added. The output information etc. may be deleted. The input information or the like may be transmitted to another device.

  The determination may be performed by a value (0 or 1) represented by one bit, may be performed by a boolean value (Boolean: true or false), or may be compared with a numerical value (for example, a predetermined value). Comparison with the value).

  The data described herein may be represented using any of a variety of different techniques. For example, data that may be mentioned throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or photons, or any combination thereof.

  Also, the data, parameters, etc. described in the present specification may be represented by an absolute value, may be represented by a relative value from a predetermined value, or may be represented by corresponding other information. . The names used for the parameters described above are in no way limiting.

  The terms "determining", "determining" as used herein may encompass a wide variety of operations. "Judgment", "decision" are, for example, judging, calculating, calculating, processing, processing, deriving, investigating, looking up (for example, a table) (Searching in a database or another data structure), ascertaining may be regarded as “decision”, “decision”, etc. Also, "determination" and "determination" are receiving (e.g. receiving information), transmitting (e.g. transmitting information), input (input), output (output), access (accessing) (for example, accessing data in a memory) may be regarded as “judged” or “decided”. Also, "judgement" and "decision" are to be considered as "judgement" and "decision" that they have resolved (resolving), selecting (selecting), choosing (choosing), establishing (establishing), etc. May be included. That is, "judgment" "decision" may include considering that some action is "judged" "decision".

  As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

  As long as “include”, “including”, and variations thereof are used in the present specification or claims, these terms as well as the term “comprising” are inclusive. Intended to be Further, it is intended that the term "or" as used in the present specification or in the claims is not an exclusive OR.

  Throughout this disclosure, unless the context clearly indicates otherwise, it is intended to include the plural.

DESCRIPTION OF SYMBOLS 100 ... Video display system, 101 ... Imaging apparatus, 102 ... Movement detection part, 103 ... Data storage part, 104 ... Display part, 105 ... Operation part, 122 ... Reference marker, 125 ... 1st submarker, 127 ... 2nd submarker, 131: first virtual position, 132: second virtual position, 151: reference arrangement setting unit, 152: position and attitude calculation unit, 153: first position relation operation unit, 154: second position relation operation unit, 155: correction unit , 201, 202 ... virtual object, α ... virtual coordinate axis.

Claims (3)

A video display system for displaying a virtual object associated with a real space, comprising:
An imaging device for imaging the real space;
A movement detection unit that detects movement of the imaging device;
A position and orientation calculation unit that calculates the position and orientation of the imaging device from the movement detected by the movement detection unit;
A storage unit which stores in advance arrangement information of virtual objects on coordinate axes in a virtual space;
A reference arrangement setting unit for detecting a reference marker disposed in the real space from an image captured by the imaging device, and associating the coordinate axis in the virtual space with the real space from a detection result of the reference marker;
The first sub-marker disposed at a predetermined position with respect to the reference marker in the real space is detected from the image captured by the imaging device, and the detection result of the first sub-marker, and the position / posture operation unit From the position and orientation of the imaging device when the imaging device captures the first submarker calculated in the above, the first submarker among the plurality of first virtual positions previously associated with the coordinate axis is the first A first positional relation operation unit that calculates a positional relation between a near first virtual position and the first sub marker;
The second sub-marker disposed at a predetermined position with respect to the reference marker in the real space is detected from the image captured by the imaging device, and the detection result of the second sub-marker and the position / posture operation unit And the position and orientation of the imaging device when the imaging device captures the second sub-marker, which is calculated in step (c), is previously associated with the coordinate axis at an interval narrower than the intervals of the plurality of first virtual positions. A second positional relation operation unit that calculates a positional relation between a plurality of second virtual positions and the second sub marker;
From the positional relationship calculated by the first positional relationship calculating unit and the positional relationship calculated by the second positional relationship calculating unit, the coordinate axis, the position of the imaging device calculated by the position and posture calculating unit, and A correction unit that corrects the relationship with the posture;
The virtual on the basis of the arrangement information stored in the storage unit and the relationship between the coordinate axis corrected by the correction unit and the position and orientation of the imaging device calculated by the position and posture calculation unit. And a display unit for displaying an object.   The correction unit corrects the relationship between the coordinate axis and the position and orientation of the image pickup device calculated by the position and posture calculation unit from the position relationship calculated by the first position relationship calculation unit. The image display system according to claim 1, wherein the relationship between the coordinate axis and the position and orientation of the image pickup device calculated by the position and orientation calculation unit is corrected from the position relationship calculated by the position relationship calculation unit. The positional relation calculated by the first positional relation calculation unit includes a deviation direction between the first sub-marker and the first virtual position in the virtual space,
The image display system according to claim 1, wherein the second virtual position is a second virtual position closest to the second sub-marker in the shift direction.

Images