JP2023077071A - Marker setting method - Google Patents

Abstract

To efficiently set a plurality of markers in aligning a virtual space with a real space in an AR device using the markers.SOLUTION: A marker setting method includes the steps of: setting a plurality of markers in a real space at predetermined intervals; storing, in a storage unit, virtual data including a plurality of virtual markers corresponding to the markers in the real space; measuring misalignment between the markers in image data obtained by capturing the markers in the real space through an imaging unit of an AR device and the virtual markers in the virtual data superimposed on the image data; and adjusting the intervals of the markers according to the misalignment.SELECTED DRAWING: Figure 9

Description

The present disclosure relates to a marker placement method.

Conventionally, virtual images (for example, 3D-CAD data, etc.) are superimposed on real space (real world) landscapes and captured images using AR devices that display AR (augmented reality) such as AR glasses, smartphones, and tablet terminals. It is known to display them together (see, for example, Patent Document 1).

JP 2014-115957 A

In the AR device as described above, it is necessary to acquire information such as the position, posture, and movement of the AR device in the real world in order to match the position of the virtual image with the landscape of the real world. In particular, it is often difficult to use GPS in the fields of civil engineering and construction, so it is conceivable to estimate the self-position and create an environment map at the same time by a technique called VSLAM, for example. In addition, in order to match the position of the virtual image with the scenery of the real world, a marker such as an AR marker is installed in the real space, and the image of the marker is captured by the imaging unit of the AR glasses. A method of referencing is conceivable.

For example, no consideration has been given to how to efficiently place the markers when aligning the position of the virtual image with the scenery of the real world using a plurality of markers.

Therefore, the present disclosure has been made in view of the above technical problems, and an object of the present disclosure is to efficiently set a plurality of markers when aligning a virtual space and a real space in an AR device using markers. It is an object of the present invention to provide a marker installation method capable of

A marker installation method according to the present disclosure includes the steps of installing a plurality of markers at predetermined intervals in a physical space;
storing in a storage unit virtual data including a plurality of virtual markers corresponding to the plurality of markers in the physical space;
measuring a deviation between the marker in image data obtained by imaging the marker in the real space with an imaging unit of an AR device and the virtual marker in virtual data displayed superimposed on the image data;
and adjusting the intervals of the plurality of markers according to the deviation.

According to the present disclosure, it is possible to provide a marker installation method that can efficiently install a plurality of markers when aligning a virtual space and a real space in an AR device using markers. .

1 is a diagram illustrating a configuration example of an AR system according to an embodiment of the present disclosure; FIG. It is a figure which shows the structural example of the AR device which concerns on this embodiment. It is a figure which shows the structural example of the server which concerns on this embodiment. It is a figure which shows the installation example of the AR marker which concerns on this embodiment. FIG. 3 is a flowchart of a method for estimating the self-position of an AR device according to the present embodiment; It is a figure which shows the other example of installation of the AR marker which concerns on this embodiment. FIG. 4 is a flow chart diagram relating to a method for aligning a physical space and a virtual space according to the present embodiment; It is a top view showing an example of setting of a marker concerning this embodiment. FIG. 4 is a flow chart diagram relating to a method of aligning a virtual space with a real space according to the present embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

FIG. 1 shows an example of an AR system according to one embodiment of the invention. As shown in FIG. 1 , the AR system of this example includes an AR device 10 and a server 20 . The AR device 10 and the server 20 are connected via a network 30 and can communicate with each other.

<AR device>
The AR device 10 of this example is AR glasses, which is a glasses-type information processing terminal. Note that the AR device 10 is not limited to AR glasses, and may be an information processing device such as a smart phone or a tablet terminal.

FIG. 2 is a block diagram outlining the configuration of the AR device 10. As shown in FIG. As illustrated, the AR device 10 includes, for example, a control unit 11 , a storage unit 12 , a display unit 13 , a communication unit 14 , an imaging unit 15 and a sensor unit 16 , which are connected to each other via a bus 17 .

The control unit 11 exchanges data between units and controls the entire AR device 10 . For example, it is implemented by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The control unit 11 can access the storage unit 12 and can store information in the storage unit 12 and read information from the storage unit 12 .

The control unit 11 acquires, generates, and outputs various data by executing programs stored in the storage unit 12 . The control unit 11 acquires information acquired from other devices such as the imaging unit 15, the sensor unit 16, and the server 20, and based on the information and the information in the storage unit 12, etc., determines the position and orientation (orientation) of the AR glasses. can be estimated.

The storage unit 12 stores a program, code, etc. for performing one or more steps, and the control unit 11 controls the operation of each unit according to the program or the like. The storage unit 12 includes a non-volatile storage device that is a read-only storage area in which a system program is stored, and a volatile storage device that is a rewritable storage area used as a work area for arithmetic processing by the control unit 11. have. Non-volatile storage devices are implemented by, for example, ROM (Read Only Memory), flash memory, hard disks, etc., and volatile storage devices are implemented by RAM (Random Access Memory), VRAM (Video Random Access Memory), and the like.

Storage unit 12 may include, for example, a separable medium such as an SD card or random access memory (RAM), or an external storage device. The storage unit 12 can store each data acquired from the imaging unit 15 and the sensor unit 16 . For example, image data such as still images and moving images captured by a camera as the imaging unit 15 may be stored in the built-in memory or the external memory.

The display unit 13 includes a display device such as a liquid crystal display device. The display device may be, for example, a wave guide display. An image controlled by the control unit 11 is displayed on the display device. That is, the display unit 13 can display the image generated by the control unit 11 . In addition, the display device has a transparent part that is partially or wholly transparent or semi-transparent, and through the transparent part, the scenery in the real space can be visually recognized, and the virtual image can be displayed superimposed on the scenery. can. The display unit 13 may include one or more lenses in addition to the display device.

The display unit 13 displays a virtual image (3D-CAD model, CG model, etc.) is superimposed and displayed on the landscape of the real space seen through the lens of the display device. Note that the display unit 13 may generate an AR image in which a virtual image is superimposed on the image captured by the imaging unit 15 using the control unit 11 or the like, and display the AR image.

The communication unit 14 connects the AR device 10 to a network 30 including the Internet. The communication unit 14 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

The imaging unit 15 includes one or more imaging devices (cameras). The control unit 11 can send image data of still images or moving images captured by the imaging device to the storage unit 12, the server 20, and the like. The imaging device can include, for example, an RGB camera, a pair of left and right fisheye cameras for VSLAM, a ToF camera (Time-of-Flight Camera), and the like. Such an imaging device can also function as the sensor section 16 . For example, a ToF camera (Time-of-Flight Camera) measures the flight time of light to determine the distance to an object, and is also called a ToF sensor.

The image capturing unit 15 can capture images continuously over a predetermined period of time according to a user's operation or based on preset instruction information. The imaging unit 15 can also capture non-continuous images at one point in time based on the user's operation or preset instruction information. That is, the imaging unit 15 can capture still images and moving images.

The sensor unit 16 can be, for example, a 3-axis acceleration sensor, a 3-axis gyro sensor, an orientation sensor, GPS, or the like. Each sensor can transmit (input) to the control unit 11 information (detection signals) such as the position, posture, movement speed, and acceleration detected by each sensor.

The AR glasses may include an input unit that receives input of a request (instruction) from a user or the like. The input unit can be, for example, a touch panel or a mechanical switch. Note that the input unit may receive line-of-sight input using eye tracking technology or voice input using a microphone or the like. The input unit may receive input from another information processing device such as a smart phone. The AR glasses may include an output unit such as a speaker for outputting voice or the like.

<server>
The server 20 is an information processing device used when a system administrator or the like operates and manages various services. may be logically realized by

As shown in FIG. 3, the server 20 includes, for example, a control unit 21, a storage unit 22, an output unit 23, a communication unit 24, and an input unit 25, which are connected to each other via a bus 26. The server 20 can communicate with AR glasses via the communication unit 24 . When receiving various request signals from the AR glasses or the like, the server 20 executes processing by a program in the control unit 21, and appropriately transmits the processing results (for example, generated images, sounds, etc.) to the AR glasses or the like, It is stored in the storage unit 22 . Note that part of the above program may be transmitted to AR glasses and executed on the AR glasses.

The control unit 21 exchanges data between units and controls the entire server 20. A CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) are stored in a predetermined memory (storage unit). It is realized by executing the program

The storage unit 22 includes a non-volatile storage device that is a read-only storage area in which the system program is stored, and a volatile storage device that is a rewritable storage area used as a work area for arithmetic processing by the control unit 21. have. Non-volatile storage devices are implemented by, for example, ROM (Read Only Memory), flash memory, hard disks, etc., and volatile storage devices are implemented by RAM (Random Access Memory), VRAM (Video Random Access Memory), and the like.

The storage unit 22 can store information on virtual space, information on virtual images, information on markers, information on each coordinate system (real space coordinate system, AR device coordinate system, marker coordinate system, virtual space coordinate system), and the like. can.

The storage unit 22 can store information received from an AR camera or the like. Also, the storage unit 22 may store the information stored in the storage unit 12 of the AR camera.

The output unit 23 includes a speaker for outputting sound, a display for displaying images, and the like.

The communication unit 24 connects the server 20 to a network 30 including the Internet. The communication unit 24 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

The input unit 25 can be a keyboard, mouse, touch panel, or the like.

FIG. 4 is a diagram showing a method of increasing the accuracy of self-position estimation of the AR device 10 using AR glasses as the AR device 10 and the markers 40 installed on the wall surface of the structure 50 on site.

<Marker>
The marker 40 can be, for example, AprilTag, but is not limited to it, and can be other AR markers (AR tags) such as ARTag, ARToolkit, ARToolkitPlus, RUN-Tag, reactTIVison, and the like. The marker 40 is, for example, generally square. The marker 40 has a matrix structure in which a plurality of square pixels are arranged two-dimensionally, and each pixel is displayed in white or black. The shape of the marker 40 and the drawn figure are not limited to this.

When AprilTag is adopted as the marker 40, the 36h11 mark series, for example, consists of 587 markers. An ID (identification information) unique to each marker 40 can be acquired simply by capturing an image of an arbitrary marker 40 with the camera of the AR device 10 . Note that when an arbitrary marker 40 is imaged by the camera of the AR device 10, the marker 40 may be imaged from the front or obliquely. identification information, position, and posture information can be obtained. The image of each marker 40 and the ID information corresponding to each image are pre-associated and stored in storage unit 22 (or storage unit 12).

Specifically, for example, when image data captured by a camera of AR glasses is transmitted to the server 20, the control unit 21 of the server 20 refers to the data stored in the storage unit 22, and ID information corresponding to the marker 40 is obtained. to the AR glasses. Thereby, the control unit 11 of the AR glasses can acquire the ID corresponding to the marker 40 . As this ID information, for example, information such as the position and orientation in the real space can be associated and stored in the storage unit 22 . According to this, by simply capturing an image of the marker 40 with a camera, information such as the position and orientation of the marker in the real space can be accurately grasped in addition to the ID of the marker 40 .

Information about such markers 40 can be displayed on the display unit 13 of the AR glasses. The user can confirm information such as the ID, position, and orientation of the marker 40 from the display unit 13 by simply capturing an image of the marker 40 with the camera of the AR glasses. Note that the process of acquiring the ID of the marker 40 based on the image of the marker 40 captured by the camera may be performed only by the AR camera (without communicating with the server 20). In that case, information about the marker 40 is stored in advance in the storage unit 12 of the AR camera.

<Coordinate system>
Here, as shown in FIG. 4, a coordinate system unique to AR glasses (AR device coordinate system) is set for the AR glasses. A coordinate system unique to the marker 40 (marker coordinate system) is set for the marker 40 . Furthermore, a physical space coordinate system (world coordinate system) is set in the physical space, and a virtual space coordinate system (virtual space coordinate system) is set in the virtual model.

Each coordinate system is, for example, a three-dimensional orthogonal coordinate system, and has three orthogonal coordinate axes (x-axis, y-axis, and z-axis) and an origin where the three coordinate axes intersect. Information on each coordinate system can be stored in the storage units 12 and 22 . That is, the origin position of each coordinate system, the direction of the coordinate axis, and the relative positions and attitudes (orientations) of the coordinate systems can be stored in the storage units 12 and 22 . The position can be represented by three-dimensional coordinates, and the orientation can be represented by three-dimensional directions (rotation angles around the x-, y-, and z-axes).

<Virtual data>
The storage unit 22 can store information about a three-dimensional virtual space (virtual space) and information about a virtual model (three-dimensional or two-dimensional CG model) positioned at a predetermined position and posture in the virtual space. can. Virtual data has information on a virtual space coordinate system. The virtual space data preferably corresponds to space data of a predetermined real space (for example, a building site). The position, direction, and coordinate system in the virtual space are associated with the position, direction, and coordinate system in the real space. The origin position of the virtual space coordinate system can match (or correspond to) a predetermined position in the real space. Also, the three coordinate axes of the virtual space coordinate system can be made to match (or correspond to) the three coordinate axes set in the real space. Also, the shape, position and orientation of the virtual model in the virtual space correspond to, for example, the shape, position and orientation of a building or the like provided at an actual construction site in the real space.

A virtual model can be created, for example, from 3D-CAD data or BIM (Building Information Modeling) data of an object. Note that the method of creating the virtual image and the type of image data are not particularly limited. The storage unit 12 has position information of the virtual model with respect to the origin of the coordinate system of the virtual space, and orientation information indicating which direction the virtual model faces in the coordinate system of the virtual space. Also, setting of the virtual space and placement of the virtual model are performed in advance based on design data such as BIM data, for example. The virtual model may also be based on a plan view (eg, a 2D construction plan or plan detail view). The generation of the virtual model may be performed by the server 20, or may be performed by another device. The AR glasses may acquire data on the virtual model from the server 20 in advance, or may repeatedly acquire data on the virtual model continuously (or intermittently) while using the AR glasses. .

The origin and each coordinate axis of the AR device coordinate system are set at predetermined positions and orientations with respect to the AR glasses. The control unit 11 (or the control unit 21) can estimate the position and orientation of the AR glasses based on the image captured by the imaging unit 15 of the AR glasses, and associate the virtual data with the real space coordinate system.

The AR glass is a device that superimposes a virtual image of an object on a real space landscape (or a real world image) based on a real space image (video) captured by the imaging unit 15 and displays the superimposed image. AR glasses are used in a portable manner, for example, by being worn on the head by users such as workers and supervisors at sites of civil engineering, construction, railroads, and transportation. Display a virtual image of the components and buildings to be installed on the scenery of the real space of the site at the planned installation position and orientation (orientation), superimposed on the scenery of the real space (or image of the real world). Let A virtual image is generated in advance by 3D-CAD or the like and stored in the storage units 12 and 22 of the AR device 10 or the server 20 or the like.

The AR glasses acquire information on the position and orientation (angle) of the AR glasses, and the control unit 11 acquires and refers to the data in the storage unit 12 to generate image data corresponding to the position and orientation. Then, the generated image data (virtual image data) is displayed on the display unit 13 . As a result, a virtual image corresponding to the position and orientation of the AR glasses is displayed on the AR glasses so as to overlap the scenery of the real world. By checking the virtual image at the site, the user can work while having an image of the work process or the final image, and can proceed with work reliably while suppressing work mistakes. Also, even when giving instructions from a remote location, it is possible to give instructions efficiently by conversing while viewing a three-dimensional virtual image.

The position information and orientation information (direction information) of the AR glasses can be obtained by, for example, self-position estimation technology. Self-position estimation technology includes, for example, Visual SLAM (Simultaneous Localization And Mapping), VIO (Visual Inertial Odometer) and other technologies that utilize image information, and AR glasses have cameras, sensors, etc. to realize the above technologies. can be loaded. Such self-position estimation technology is particularly effective in non-GPS environments. The AR glasses of this example can simultaneously perform self-position estimation and creation of an environment map (virtual space) from the image captured by the imaging unit 15 by so-called “Visual SLAM”.

The AR glasses of this embodiment are spectacle-type display devices worn by the user on the head, and include one or more lenses (integrated with or separate from the display unit 13), a frame (frame body) for supporting the lenses and the like, It can include a control unit 11, an imaging unit 15, a sensor unit 16, a display unit 13, a communication unit, a storage unit 12, an input unit, and the like. A part of the configuration of the AR glasses (control unit 11, etc.) may be separated as a separate unit and connected by wire or wirelessly. 20 may be executed.

The position and orientation of the AR glasses are determined by the self-position estimator in the controller 11 . The AR glasses have a communication unit for communication connection to an information processing device such as the server 20 or the like via a network such as the Internet or directly. As a result, various data can be transmitted/received to/from another information processing apparatus having a communication unit. For example, image data captured by AR glasses, image data generated by the control unit 11, sensor data acquired by the sensor unit 16, and the like are transmitted to the server 20 and stored and processed on the server 20 side, or transmitted from the server 20. It is also possible to receive data such as images that have been captured.

An image captured by the imaging unit 15 of the AR device 10 may be transmitted to the server 20 so that the control unit 21 of the server 20 generates an image to be displayed on the AR device 10 .

The control unit 11 acquires information on the position (imaging position) and orientation (imaging direction) of the imaging unit 15 (camera), and creates a virtual model corresponding to the position and orientation of the imaging unit 15 when the image was captured. image is generated (based on the information in the storage unit 12 ) and output to the display unit 13 .

The control unit 11 captures an image of the virtual model set in the virtual space with a virtual camera assuming that the image capturing unit 15 is in the virtual space under the same conditions as the position and orientation when the image was captured by the image capturing unit 15. An image of a virtual model of the case is generated, and the display unit 13 displays it so as to overlap the scenery of the real space (or the image of the real space).

When displaying the virtual model on the display unit 13 of the AR glasses, other parameters such as the brightness of the real space may be considered in addition to the position and orientation of the camera. Also, the virtual model may be translucent so that the portion where the virtual model image overlaps (hidden portion) can also be visually recognized. Note that the display unit 13 of the AR glasses sets and modifies the virtual image to be displayed according to the position and orientation of the camera.

Based on the information from the imaging unit 15 or the sensor unit 16, the control unit 11 can estimate the moving state (presence or absence of movement, moving direction, moving speed, etc.) of the AR glasses imaging device. The control unit 11 calculates the position and orientation of the camera in the virtual space (the position and orientation when the camera is in the virtual space) according to the movement state information. The display unit 13 superimposes the virtual model on the image based on the corrected position and orientation. By continuously performing such processing, it is possible to display on the display unit 13 a virtual image that follows the movement of the AR glasses according to the movement of the user.

The display unit 13 generates an image to be displayed and outputs it to the display each time an image to be captured is input (continuously or intermittently) or each time information indicating the movement state is received from the sensor unit or the like. conduct. That is, the display unit 13 superimposes and displays the virtual model on the landscape or image in the real space so as to follow the moving state of the camera. The function of the display unit 13 can be realized by a known AR method (eg, ARToolKit tracking algorithm, SLAM function).

A method of calibrating (correcting) the self-position and orientation using the marker 40 will be described below with reference to FIGS. 4 and 5. FIG.

First, as shown in FIG. 4, a marker 40 is previously positioned and installed at a predetermined position such as a construction site (step S1). The position and orientation (orientation) of the marker 40 in the physical space coordinate system are stored in advance in storage units (12, 22) of at least one of the AR glasses and the server 20. FIG. In the example of FIG. 4, the marker 40 is installed on the vertical wall surface of the structure 50, but it may be installed on a horizontal surface or the like. Also, the two sides of the marker 40 are parallel to the upper edge (horizontal edge located at the upper end) and the side edge (vertical edge located at the side edge) of the wall surface of the structure 50 . , but not limited to.

The user adjusts the orientation of the camera so that the marker 40 is positioned within the imaging range of the camera of the AR glasses, and images the marker 40 with the camera (step S2).

The control unit (11, 21) calculates the three-dimensional position and orientation of the marker 40 on the AR device coordinate system (camera coordinate system) based on the captured image data of the marker 40. FIG. At that time, the position and posture of the marker 40 are determined using a point that forms one of the corners of the square marker 40 and two sides (edges) perpendicular to the corner of the square marker 40 as reference elements (reference points, reference lines). to decide. Control units (11, 21) calculate a three-dimensional relative positional relationship (coordinate transformation matrix (R, T)) between the AR glass coordinate system and the marker coordinate system.

In addition, the control unit (11, 21) performs back calculation using the coordinate transformation matrix (R, T) based on the position and orientation of the marker 40 on the AR device coordinate system to obtain the AR on the marker coordinate system. Calculate the position and orientation of the glass. Thereby, the control unit (11, 21) calculates the relative position and orientation of the AR glasses with respect to the marker 40 (step S3).

Based on the captured image data of the marker 40, the control unit (11, 21) acquires the identification information, the position, and the orientation information of the marker 40 in the physical space coordinate system from the storage unit (12, 22). In other words, the control unit (11, 21) stores the three-dimensional relative positional relationship (coordinate transformation matrix (R0, T0)) between the marker coordinate system and the real space coordinate system from the storage unit (12, 22). get. The position and orientation of the marker 40 on the physical space coordinate system and the relative three-dimensional positional relationship (position and orientation) between the physical space coordinate system and the marker coordinate system are stored in advance in storage units (12, 22). . The control unit (11, 21) solves a so-called PnP problem (Perspective n-Point) to obtain an image of an object whose position and shape are known and the position and orientation of the camera that captured the image, thereby Position coordinates (imaging position coordinates) and orientation of the AR glasses included can be estimated.

The control units (11, 21) are based on the relative position and orientation information of the AR glasses with respect to the marker 40 and the relative three-dimensional positional relationship between the real space coordinate system and the marker coordinate system (that is, , and the coordinate transformation matrix (R0, T0) to calculate the position and orientation of the AR glasses on the physical space coordinate system (step S4).

With such a method, the AR glasses can estimate the self position and orientation on the real space coordinate system with high accuracy.

A method of aligning the virtual space with the real space will be described below with reference to FIGS. More specifically, the initial setting method for using the AR device in the field will be described.

In the method of aligning the virtual space with the real space according to the present embodiment, the steps of placing a marker along a corner or edge in the real space; a step of measuring the position and orientation; a step of calculating the relative position and orientation of the AR device with respect to the marker based on image information of the marker captured by the imaging unit of the AR device; and information obtained in the measuring step. and aligning the coordinate system of the virtual space stored in advance in the storage unit with the coordinate system of the real space based on the information obtained in the calculating step. With such a configuration, it is possible to align the virtual space with the real space with high accuracy in any environment, that is, in various environments.

Specifically, for example, when using the AR glasses for the first time in the field, the user performs initial settings (setup). First, the application stored in the AR glasses is started by turning on the power switch of the AR glasses. Then, as an initial setting, initial alignment is performed between a virtual space coordinate system (virtual world) in which a three-dimensional virtual image is set and a real space coordinate system (real world). Information on the virtual space coordinate system in which the virtual image is set is stored in storage units 12 and 22 . The virtual space coordinate system information includes the virtual coordinate system, virtual model information, and marker information in the virtual space.

At the site, for example, near the entrance of a space where buildings, structures, etc. are installed, a marker 40 is installed near a position that can be easily referred to in a three-dimensional design drawing (3D-CAD, etc.) (step S11). The easy-to-reference locations are, for example, corners (outer corners or inner corners) and edges (eg, straight edges) of the building. When the control unit performs self-position estimation and mapping (map formation) using VSLAM or the like, the edge (edge) where two adjacent planes intersect and the corner where three planes intersect are used as reference points P. can be done. When the marker is installed, the marker can be detachably fixed to the object by sticking it using an adhesive means or the like.

In the example shown in FIG. 6, the reference point P is a corner (inner corner) where three mutually orthogonal surfaces (a horizontal floor surface and two vertical wall surfaces) in the building intersect. A first marker 40a is installed on the floor surface, and a second marker 40b is installed on the wall surface (side wall surface).

When installing the markers 40, it is preferable to install the plurality of markers 40 on different planes (wall surface, floor surface). According to this, more information can be referred to than when performing alignment using only one marker 40, so the accuracy of alignment can be improved. The position where the marker 40 is installed is preferably a vertical wall surface or a horizontal floor surface, but is not limited to this. Moreover, from the viewpoint of increasing the accuracy of alignment, it is preferable to install the markers 40 on planes orthogonal to each other. According to this, the accuracy of alignment can be improved compared to the case where the markers are placed on planes extending obliquely with respect to each other.

Further, it is preferable to install the markers 40 on each of the plurality of wall surfaces forming the corners, which are the reference points P. As shown in FIG. According to this, it is possible to set a plurality of markers at different angles around the corner, which is the reference point P, and to further improve the accuracy of alignment.

In addition, it is preferable to arrange a plurality of markers 40 (that is, at relatively close positions) within an area that can be imaged simultaneously by the imaging unit 15 of the AR device 10 . In the example of FIG. 6, the first marker 40a and the second marker 40b are installed on planes (floor surface and wall surface) perpendicular to each other. As in this example, it is preferable to install the two markers 40 such that the two markers 40 sandwich an edge (a linear edge where two planes intersect) from both sides.

The markers 40 are preferably installed along the corners and edges of the building. More specifically, the marker 40 is installed so that the two orthogonal sides of the wall surface forming the corners of the inner wall surface of the building and the two sides forming the corners of the square marker are parallel to each other. is preferred. Alternatively, it is preferable to arrange so that one edge of a structure such as a building and one side of the square marker 40 are parallel to each other. By doing so, it is easy to set the marker 40 at an appropriate position and attitude, and positioning of the virtual space with respect to the real space is also facilitated. Note that the marker may be positioned on the ground at a predetermined position (latitude and longitude) and in a predetermined orientation (for example, one side of the marker is parallel to any of the north, south, east, and west directions). . The markers 40 can be offset along the plane from the reference corners and edges by a method such as marking.

Using a measuring device such as a laser positioning meter, the relative deviation of each marker 40 from the reference point P and the coordinate axis (orientation axis) of the 3D-CAD design drawing (that is, the relative position and orientation of the marker 40) is indicated. Data (R0, T0) are measured (step S12). Specifically, in the real space, the position and orientation of the marker 40 are measured (actually measured) with reference to the corner (reference point P) and edge of the building. Thereby, the relationship (position, orientation) between the physical space coordinate system and each marker coordinate system can be calculated. Information on the position and orientation of the marker 40 in the physical space is stored in the storage unit 22 of the server 20 . The information is transmitted from the server 20 to the AR glasses. Note that data indicating the shift (coordinate conversion matrix (R0, T0)) may be stored in the storage unit 12 of the AR glasses. The measuring device is not limited to the laser positioning device, and may be any device that can measure the distance, posture, etc. between two specific points in the real space. For example, it may be a level or a telescope for measuring angles and distances from a long distance. In addition, for example, a measuring method by marking including an automatic marking device, a laser marking device, and the like may be used. The automatic marking device can determine the current position by measuring the relative position of the device with respect to the reference point, or draw a line on the floor at the position of the reference line. The measuring device may, for example, calculate the shape and dimensions of the floor, walls, and reference points from images captured by a camera. Alternatively, the measuring device may use a laser range finder to obtain the dimensions of the floor.

Then, at the site, the placed marker 40 (40a, 40b) is imaged by the imaging unit 15 of the AR glasses (step S13). The AR glasses controller automatically detects the marker 40 . The control unit calculates the relative position and orientation of the AR glasses with respect to each marker 40 (40a, 40b) (step S14). Then, using the information of the relative position and orientation of the AR glasses with respect to each marker 40 (40a, 40b) and the data (R0, T0) indicating the deviation, the initial position of the virtual space with respect to the real space Alignment is realized (step S15). That is, the physical space coordinate system and the virtual space coordinate system are associated. Preferably, the origin position and the directions of the three coordinate axes of the virtual space coordinate system are aligned with the position of the origin and the directions of the three coordinate axes of the real space coordinate system.

It should be noted that it is preferable to install the markers 40 on two orthogonal planes, as in this example, whereby the accuracy of alignment can be further improved. In addition, by setting the markers 40 on each of the three planes orthogonal to each other, it is possible to further improve the accuracy of alignment.

The technology of the present disclosure is particularly effective in, for example, places with few feature points such as tunnel construction, places with many similar landscapes, or situations where the environment is constantly changing. Fields having such problems include, for example, fields such as civil engineering, construction, railroads, and transportation.

A method of calibrating (correcting) the self-position and orientation using a plurality of markers 40 will be described below with reference to FIGS. 8 and 9. FIG. FIG. 8 is a plan view showing a case where a plurality of markers are installed at predetermined regular intervals from the entrance inside a building in real space. The position of the marker to be installed may be on the floor surface, the wall surface, or other positions. Although six markers are installed in the example of FIG. 8, the number of markers is not limited to this. In addition, as described above, each marker 40 is preferably installed along the corners and edges of the current space-time that serve as reference points.

A marker placement method according to the present embodiment includes the steps of placing a plurality of markers at predetermined intervals in a physical space, and storing in a storage unit virtual data including a plurality of virtual markers corresponding to the plurality of markers in the physical space. measuring a deviation between the marker in image data obtained by imaging the marker in the physical space with an imaging unit of an AR device and the virtual marker in virtual data superimposed on the image data; and adjusting the spacing of the plurality of markers according to the deviation. With such a configuration, it is possible to efficiently set a plurality of markers when aligning the virtual space and the real space in the AR device using the markers.

As shown in FIGS. 8 and 9, a plurality of markers 40 are placed at regular intervals in the physical space (step S21). For example, the first marker 40 for alignment is installed near the entrance of a space such as a building or structure. The plurality of markers 40 may be the same as each other or different types of markers 40 .

Next, the second marker 40 is installed in accordance with the space so that the distance from the first marker 40 is a specific distance (for example, 10 m). Similarly, the third marker 40 is placed so that the distance from the second marker 40 is the specified distance, and the fourth marker 40 is placed so that the distance from the third marker 40 is the specified distance. Install. In this way, the markers 40 are installed at regular intervals according to the space (situation) of the site. Note that the specific distance is not particularly limited, and is a numerical value that can be changed as appropriate.

Here, a virtual marker is set in the virtual space (step S22). Note that the setting of the virtual marker may be performed before or after step S21 of setting the marker in the physical space. For example, in the virtual model of the virtual space stored in the storage unit, each marker 40 is drawn (3D modeling) and stored so as to match the position and orientation of the marker in the real space. As a result, a virtual marker corresponding to the marker 40 in the physical space is displayed as a virtual image displayed on the display unit 13 of the AR glasses. A virtual model including walls and floors (of the same shape) corresponding to structures including walls and floors installed on site in the real space is stored in advance in the storage unit as virtual data positioned in the virtual space. ing.

Then, while confirming the virtual image displayed on the AR glasses, move in the space from the entrance of the site in the real space, and shoot from the first marker to the second, third, etc. with the imaging unit of the AR glasses. move while At that time, the position and orientation of each virtual marker (virtual marker displayed on the AR glass) assumed to be superimposed and displayed on each marker 40 in the real space, and the position and orientation of the marker 40 in the real space. compare. Note that the AR glasses continue to estimate their own position and orientation using VSLAM or the like, and display a corresponding virtual image on the display unit. Then, it is checked whether or not the amount of deviation between the position and orientation of the marker in the physical space and the position and orientation of the virtual marker exceeds a predetermined threshold (step S23). For the threshold value, a value determined in advance by the administrator may be stored in the storage unit, or the user may input a numerical value via the input unit, or select from a plurality of preset options. You may enable a user to set by doing. The positional deviation threshold is indicated by a distance, such as 5 mm, and the posture deviation threshold is indicated by an angle, such as 5°.

The deviation amount can be measured by, for example, the AR glasses or the control unit of the server. When the user captures an image of the marker with the imaging unit of the AR glasses, the control unit superimposes the image captured with the imaging unit of the AR glasses and the virtual image to be displayed on the display unit of the AR glasses to capture the real space. The deviation between the position and orientation of the marker in the captured image and the position and orientation of the virtual image is measured. More specifically, it is possible to compare the marker coordinate system of the marker in the image obtained by capturing the physical space and the virtual marker coordinate system of the virtual marker in the virtual space. Measure the distance between the origin of the real marker coordinate system and the origin of the virtual marker coordinate system, compare the orientation (orientation) of the coordinate axes of the real marker coordinate system and the virtual marker coordinate system, and measure the difference between them ( angle). Also, for example, reference points can be compared and measured. For example, by measuring the distance between specific corners of the quadrangle marker 40, or by measuring the distance between sides or the relative angle, the deviation between the marker in the real image and the marker in the virtual image is measured. . Note that the method of measuring the deviation is not limited to this.

Then, when the deviation amount exceeds the threshold, the distance between the markers 40 is adjusted (step S24). For example, if the positional deviation amount between the actual second marker 40 and the second virtual marker exceeds 10 mm, the distance between the markers 40 is shortened. Specifically, as a result of confirming the deviation amount with the distance between the markers 40 set to 10 m as described above, if the deviation amount exceeds the threshold value, for example, the distance between the markers is reduced by 2 m to 8 m, or the like. , to a distance shorter than the initial distance of 10 m. In this example, first, the position of the second marker is adjusted, for example, the second marker is moved to a position closer to the first marker (installed closer). How short the distance is to be adjusted may be determined in advance and stored in the storage unit, or may be determined by the user. Alternatively, the adjustment distance may be determined according to the amount of deviation. In this case, if the deviation amount is large, the adjustment distance can be increased, and conversely, if the deviation amount is small, the adjustment distance can be shortened. In addition, in the above example, the adjustment distance is set to 2 m. If there is little deviation, the distance between markers may be increased. Also in that case, how long the distance should be increased may be determined in advance and stored in the storage unit, or may be determined by the user. Alternatively, the adjustment distance may be determined according to the amount of deviation.

After adjusting the position and orientation of the marker in the real space, the position of the virtual marker in the virtual space is also changed accordingly and stored in the storage unit. At that time, for example, a measurement device such as a laser positioning device may be used to measure the position and orientation of the marker in the current spatio-temporal space. According to this, the virtual marker can be modeled with a more accurate position and orientation.

As described above, it is possible to check the accuracy of detecting the movement of the AR glasses by checking the position and posture deviation between the markers in the real space and the virtual markers in the virtual image. It can then check if the distance between the markers 40 is appropriate (depending on the accuracy of the AR glasses). As a result, since the marker 40 can be installed with an appropriate feeling, the usage efficiency of the marker 40 can be improved. That is, it is possible to prevent unnecessary use of many markers 40 .

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that those who have ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. is naturally within the technical scope of the present disclosure.

The devices described herein may be implemented as a single device, or may be implemented by a plurality of devices (eg, cloud servers) partially or wholly connected via a network, or the like. For example, the control unit 21 and the storage unit 22 of the server 20 may be realized by different servers connected to each other via a network.

The sequence of operations performed by the apparatus described herein may be implemented using software, hardware, or a combination of software and hardware. It is possible to prepare a computer program for realizing each function of the AR device 10 and the server 20 according to the present embodiment and implement it in a PC or the like. A computer-readable recording medium storing such a computer program can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Also, the above computer program may be distributed, for example, via a network without using a recording medium.

Also, the processes described with reference to flowcharts in this specification do not necessarily have to be executed in the illustrated order. Some processing steps may be performed in parallel. Also, additional processing steps may be employed, and some processing steps may be omitted.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification in addition to or instead of the above effects.

Note that the following configuration also belongs to the technical scope of the present disclosure.
(Item 1)
placing a plurality of markers at predetermined intervals in the physical space;
storing in a storage unit virtual data including a plurality of virtual markers corresponding to the plurality of markers in the physical space;
measuring a deviation between the marker in image data obtained by imaging the marker in the real space with an imaging unit of an AR device and the virtual marker in virtual data displayed superimposed on the image data;
and adjusting the intervals of the plurality of markers according to the deviation.
(Item 2)
An item characterized in that in the step of measuring the deviation, a difference in at least one of position and orientation between the marker coordinate system of the marker in the imaged image data and the virtual marker coordinate system of the virtual marker is measured. 1. The marker installation method according to 1.
(Item 3)
In the step of adjusting the interval,
3. The marker placement method according to item 1 or 2, wherein the distance is reduced when the deviation is greater than a predetermined first threshold.
(Item 4)
In the step of adjusting the interval,
4. The marker placement method according to any one of items 1 to 3, wherein the interval is increased when the deviation is smaller than a predetermined second threshold.

1 AR System 10 AR Device 20 Server 30 Network 40 Marker

Claims (4)

placing a plurality of markers at predetermined intervals in the physical space;
storing in a storage unit virtual data including a plurality of virtual markers corresponding to the plurality of markers in the physical space;
measuring a deviation between the marker in image data obtained by imaging the marker in the real space with an imaging unit of an AR device and the virtual marker in virtual data displayed superimposed on the image data;
and adjusting the intervals of the plurality of markers according to the deviation. The step of measuring the shift measures a difference in at least one of position and orientation between the marker coordinate system of the marker in the captured image data and the virtual marker coordinate system of the virtual marker. Item 1. The marker installation method according to item 1. In the step of adjusting the interval,
3. The marker placement method according to claim 1, wherein said gap is reduced when said deviation is greater than a predetermined first threshold. In the step of adjusting the interval,
4. The marker placement method according to any one of claims 1 to 3, wherein said gap is increased when said deviation is smaller than a predetermined second threshold.

Images