JP6896688B2 - Position calculation device, position calculation program, position calculation method, and content addition system - Google Patents

Description

An embodiment of the present invention relates to a technique for acquiring a self-position.

Conventionally, AR (Augmented Reality) technology that adds virtual information to things in the real world has been known. According to this AR technology, contents such as 3D models and texts are presented corresponding to a specific pattern in a video showing a real environment, the position of a device in the real environment, and the like.

In such AR technology, the one that presents the content based on the surrounding environment that can be recognized by the device such as the image captured by the camera and the space is called the vision-based AR, and the GNSS (Global Navigation Satellite System) sensor. A device that presents content based on the location information of a device obtained by such means is called a location AR.

Location AR is expected to be applied to work support in the work space due to its characteristic of adding content to the real environment. As work support using AR technology, it is considered to add precautions and instructions regarding work as contents to an arbitrary position in the work space.

In addition, as related technologies, a recognition unit that recognizes an environment recognition matrix that expresses the position and orientation of the environment reflected in the image based on the position and orientation of the terminal that captured the image, and a calculation that calculates the inverse matrix of the environment recognition matrix. An image processing device including a unit and an object arranging unit that arranges a virtual object having a position and an orientation based on an inverse matrix of an environment recognition matrix in the environment is known (Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-177834

When adding content to a specific position in the real world or presenting the added content, it is indispensable to specify the position of the device in the real world as a premise. For specifying the position of the device, it is common to use a GNSS sensor that can obtain the latitude and longitude. However, in an environment where it is difficult to receive a GNSS signal such as underground, there is a problem that the position of the device cannot be obtained accurately, and therefore the content is not properly added and presented.

An embodiment of the present invention has been made to solve the above-mentioned problems, and provides a technique capable of accurately obtaining the position of an apparatus even in an environment where it is difficult to acquire a GNSS signal. With the goal.

In order to solve the above-mentioned problems, an embodiment of the present invention is a position calculation device that calculates its own position, and is an imaging device that images a marker arranged at a predetermined position, and a marker in the captured image. A parameter calculation unit that calculates the initial position and initial posture of the position calculation device with the position of the marker as the origin based on the size and shape, and a three-dimensional feature point of an object around the position calculation device. The displacement of the position and posture of the position calculation device is calculated based on the displacement between the depth sensor that detects the group at predetermined time intervals and the group of three-dimensional feature points that move back and forth in the time series, and the initial position and the initial position are calculated. It is provided with a parameter update unit for updating the position and posture of the position calculation device starting from the posture.

Further, the embodiment of the present invention is a content addition system including a position calculation device for calculating its own position, a position calculation device, and a server device that can be connected by a network, and the position calculation device is arranged at a predetermined position. Based on the image pickup device that captures the image of the marker and the size and shape of the marker in the captured image, the initial position and initial posture of the position calculation device with the position of the marker as the origin are calculated as initial parameters. Based on the parameter calculation unit, the depth sensor that detects the three-dimensional feature point group of the object around the position calculation device at predetermined time intervals, and the displacement between the three-dimensional feature point groups that move back and forth in the time series. A parameter update unit that calculates the displacement of the position and posture of the position calculation device and updates the position and posture of the position calculation device starting from the initial position and the initial posture, a predetermined designated location, and the position calculation device. The distance between the two is acquired by the depth sensor, and the position of the designated location with the position of the marker as the origin is calculated based on the separation distance and the position and orientation of the position calculation device. The server device includes an additional processing unit that transmits the additional content associated with the predetermined content added to the designated location to the server device, and the server device receives the additional content transmitted by the position calculation device. It is provided with a unit and a content information management unit that manages received additional content as content information.

It is the schematic which shows the overall structure of the content addition system which concerns on 1st Embodiment. It is a schematic plan view which shows the work site where the content addition system which concerns on 1st Embodiment is operated. It is a block diagram which shows the hardware configuration of a server device. It is a block diagram which shows the hardware composition of the position calculation apparatus. It is a block diagram which shows the functional structure of the position calculation apparatus. It is a block diagram which shows the functional structure of a server device. It is a flowchart which shows the whole operation of the position calculation apparatus which concerns on the addition of content. It is the schematic for demonstrating the calculation of the initial position and the initial posture. It is a schematic plan view which shows the path from an initial position to the present position. It is a flowchart which shows the operation of the content addition processing. It is a schematic diagram which shows a content addition position and a virtual object. It is a flowchart which shows the whole operation of the server device which concerns on the addition of contents. It is the schematic which shows the additional content superposed on the captured image. It is a schematic diagram which shows the additional content associated with a category. It is a schematic plan view which shows the manufacturing factory which operates the content addition system which concerns on 2nd Embodiment. It is a schematic diagram which shows the output of one additional content in a building. It is the schematic which shows the output of the other additional content in a building. It is a schematic diagram which shows the sightseeing spot where the content addition system which concerns on 3rd Embodiment is operated. It is a schematic diagram which shows the output of the additional content in a monument.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
(overall structure)
The overall configuration of the content addition system 1 according to the first embodiment and the work site where the content addition system is operated will be described. FIG. 1 is a schematic view showing the overall configuration of the content addition system according to the present embodiment. FIG. 2 is a schematic plan view showing a work site where the content addition system according to the present embodiment is operated.

As shown in FIG. 1, the content addition system according to the present embodiment includes a server device 10, an administrator terminal 20, and at least one or more position calculation devices 30. The position calculation device 30 is a portable device, and is used by an operator at a work site where the content addition system is operated in the present embodiment. Further, the server device 10 and the administrator terminal 20 are installed in a management center located at a place different from the work site.

The position calculation device 30 connects to the network via a wireless LAN (Local Area Network) installed at the work site, and communicates with the server device 10 via the network. Further, the administrator terminal 20 communicates with the server device 10 via a wired LAN installed in the management center. The position calculation device 30 does not have to be always connected to the network, and data to be transmitted / received may be temporarily stored in the position calculation device 30 so that data can be transmitted / received when connected to the network. good.

In the present embodiment, the work site is a building in which the content addition system is operated, and is, for example, one section in a nuclear power plant or the like. As shown in FIG. 2, this work site is one section having a plurality of rooms surrounded by walls, and it is assumed that a plurality of facilities are arranged in the section.

At this work site, a worker having each of the position calculation devices 30 is engaged, and by using the content addition system described in detail below, the specific location on the wall or equipment is associated with this location. Add a virtual object that indicates that the content is present. This virtual object is a virtual mark that can be visually recognized by the operator via the touch panel 34 (see FIG. 4) of the position calculation device 30.

Further, a marker, which is a two-dimensional image in which a predetermined pattern is drawn in a square, is attached to the wall portion near the entrance of the work site. This marker is a so-called AR marker, and as will be described in detail later, functions as the origin of coordinates indicating the position of the position calculation device 30 and the position of the virtual object. The marker may be any image as long as it is suitable for estimating the positional relationship between the position calculation device 30 and the marker.

(Hardware configuration)
The hardware configuration of the server device and the position calculation device will be described. FIG. 3 is a block diagram showing a hardware configuration of the server device. FIG. 4 is a block diagram showing a hardware configuration of the position calculation device.

As shown in FIG. 3, the server device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage device 13, an input / output I / F (Interface) 14, and a network I / F 15 as hardware. To be equipped. The CPU 11 and the RAM 12 cooperate to execute various functions, and the storage device 13 stores various data used for processing executed by the various functions. The input / output I / F 14 inputs / outputs data to / from an input device such as a keyboard connected to the server device 10 and an output device such as a display. The network I / F 15 performs wired communication and wireless communication with other devices such as the administrator terminal 20 and the position calculation device 30. In the present embodiment, the administrator terminal 20 has the same hardware configuration as the server device 10, and the description thereof will be omitted.

The position calculation device 30 is a tablet-type terminal, and as shown in FIG. 4, includes a CPU 31, a RAM 32, a storage device 33, a touch panel 34, a network I / F 35, a camera 36, and a depth sensor 37 as hardware. The CPU 31 and the RAM 32 cooperate to execute various functions, and the storage device 33 stores various data used for processing executed by the various functions. The touch panel 34 is an input / output device having a display and a touch sensor and provided on the front side of the position calculation device 30. The network I / F35 performs wireless communication with the server device 10.

The camera 36 is provided on the back side of the position calculation device 30 and is an imaging device that captures a two-dimensional image and a video by visible light. The depth sensor 37 is a sensor provided on the back side of the position calculation device 30 and measures the distance from the position calculation device 30 to surrounding objects by projecting a laser and receiving the reflected light by a light receiving element. is there. In the present embodiment, the depth sensor 37 measures the distance from the position calculation device 30 to a surrounding object by using a ToF (Time of Flight) method that converts the delay time of the received light pulse with respect to the floodlight pulse into a distance. Then, the sensor is made into a three-dimensional point cloud. It is assumed that the imaging direction by the camera 36 and the measurement direction by the depth sensor 37 are the same.

(Functional configuration of position calculation device)
The functional configuration of the position calculation device will be described. FIG. 5 is a block diagram showing a functional configuration of the position calculation device.

As shown in FIG. 5, the position calculation device 30 functions as a parameter calculation unit 301, a parameter update unit 302, an additional processing unit 303, a display unit (presentation unit) 304, a user information transmission unit 305, and a content information reception unit 306. A position transmission unit 307 and a video transmission unit 308 are provided.

The parameter calculation unit 301 recognizes the marker in the image captured by the camera 36, and based on the recognized marker, calculates the initial position and the initial posture of the position calculation device 30 in the marker coordinates with the attached marker as the origin as parameters. To do. The parameter update unit 302 calculates the displacement of the position and orientation of the position calculation device 30 for each frame based on the displacement between the frames before and after the three-dimensional point group acquired by the depth sensor 37, and based on this displacement. , The position and the posture of the position calculation device 30 starting from the initial position and the initial posture calculated by the parameter calculation unit 301 are updated. The position and posture of the position calculation device 30 are associated with the time at the time of calculation and update, respectively.

The addition processing unit 303 is the distance between the current position and the current posture of the position calculation device 30 and the equipment or the designated location on the wall portion as the object to which the content is added from the position calculation device 30 measured by the depth sensor 37. The position of the designated location on the marker coordinates is calculated based on the above, and the content is added by associating the designated location with the content. Further, the additional processing unit 303 transmits the added content to the server device 10, and as the additional content, the server device 10 transmits the content associated with the predetermined coordinates in the marker coordinate system and the orientation of the virtual object pointing to the coordinates. Send to. Here, the content includes any data that can be directly or indirectly presented to the user via an output device such as text, URL, voice, image or the like.

The display unit 304 superimposes a virtual object on the image captured by the camera 36 and displays it on the touch panel 34. When displaying the plurality of virtual objects on the touch panel 34, the display unit 304 calculates the context with the plurality of virtual objects or the actual objects based on the current position and the current posture of the position calculation device 30, and the front side. The virtual object or real object located in is superimposed on the virtual object or real object located in the back and displayed.

The user information transmission unit 305 transmits the user ID uniquely indicating the worker as a user and the authentication information corresponding to the user ID to the server device 10 as an authentication request. The content information receiving unit 306 receives the content information transmitted by the server device 10 as a response to the authentication request by the user information transmitting unit 305. The position transmission unit 307 transmits the initial position calculated by the parameter calculation unit 301 and the subsequent position sequentially calculated by the parameter update unit 302 to the server device 10. The video transmission unit 308 associates the display screen displayed on the touch panel 34 by the display unit 304 with the position of the position calculation device 30 at that time, and transmits the display screen to the server device 10 as a display screen image in which the display screens are continuous in chronological order. ..

(Functional configuration of server device)
The functional configuration of the server device will be described. FIG. 6 is a block diagram showing a functional configuration of the server device.

As shown in FIG. 6, the server device 10 functions as a user information management unit 101, a user authentication unit 102, an additional content reception unit 103, a content information management unit 104, a content information transmission unit 105, a position reception unit 106, and a route. It includes a management unit 107, a route transmission unit 108, a video reception unit 109, a video management unit 110, and a video transmission unit 111.

The user information management unit 101 manages the user ID and the authentication information in association with each other as user information. The user authentication unit 102 authenticates the position calculation device 30 by collating the user information transmitted by the position calculation device 30 with the user information managed by the user information management unit 101. The user ID is also assigned to the user of the administrator terminal 20, and the user authentication unit 102 authenticates the administrator terminal 20 in the same manner as the position calculation device 30.

The additional content receiving unit 103 receives the additional content from the position calculation device 30. The content information management unit 104 manages the content received by the additional content receiving unit 103 as content information with a content ID which is an identifier uniquely indicating the content. Here, the content information is associated with the additional content, whether or not viewing is restricted, and the user ID of the user who is permitted to view. There are two types of content information, one that is not restricted to be viewed by all users and the other that is restricted to be viewed by only some users. Users who are permitted to view only the latter content information It is assumed that the user ID of is set. The content information transmission unit 105 transmits only the content information for which the user is not restricted from viewing to the position calculation device 30 for the user authenticated by the user authentication unit 102.

The position receiving unit 106 receives the position transmitted from the position calculating device 30. Each time the user is authenticated by the user authentication unit 102, the route management unit 107 generates a route ID that uniquely indicates the route of the position calculation device 30 at the work site, and associates the route ID with the user ID and the route information. To manage. Here, the route information includes the initial position and the updated position transmitted from the position calculation device 30 used by the authenticated user, and the calculated times thereof. The route transmission unit 108 transmits the corresponding route information to the administrator terminal 20 in response to the acquisition request of the predetermined route by the administrator terminal 20.

The video receiving unit 109 receives the display screen video transmitted by the position calculation device 30. The video management unit 110 associates the display screen video received by the video reception unit 109 with the user ID, and manages the display screen video by assigning a video ID uniquely indicating the display screen video. The video transmission unit 111 transmits the corresponding display screen image to the administrator terminal 20 in response to the acquisition request of the predetermined display screen image by the administrator terminal 20. By making the display screen image viewable by the administrator terminal 20, the status of the user as a worker can be confirmed from the management center. Since the position of the position calculation device 30 is associated with the series of display screens constituting the display screen image, a part area of the work site is preset as a mask area as shown in FIG. As a result, the display screen based on the captured image in this mask area may be masked such as being painted black. As a result, the equipment in the mask area at the work site can be concealed so that it cannot be viewed from the outside.

(Operation of position calculation device)
The operation of the position calculation device related to the addition of contents will be described. FIG. 7 is a flowchart showing the overall operation of the position calculation device related to the addition of the content. FIG. 8 is a schematic view illustrating the calculation of the initial position and the initial posture. FIG. 9 is a schematic plan view showing a route from the initial position to the current position. In FIG. 7, it is assumed that the image captured by the camera has been acquired in advance by the parameter calculation unit.

As shown in FIG. 7, the parameter calculation unit 301 determines whether or not the marker is recognized in the image captured by the camera 36 (S101).

When the marker is recognized (S101, YES), the parameter calculation unit 301 calculates the initial position and the initial posture of the position calculation device 30 with the marker as the origin as parameters (S102).

Here, the calculation of the initial position and the initial posture will be described. As shown in FIG. 8, when the image pickup range S including the marker M attached to the wall surface perpendicular to the horizontal plane is imaged by the camera 36 with respect to the wall portion W at the work site, the marker M and the position calculation device 30 Depending on the positional relationship with, the marker m on the captured image is scaled or deformed. The parameter calculation unit 301 calculates the relative position and the relative posture of the position calculation device 30 with respect to the marker M from the size and shape of the marker m based on the marker information indicating the shape and size of the reference marker. Further, the parameter calculation unit 301 determines the position of the position calculation device 30 on three axes (x, y, z in the figure) with the marker M as the origin from the relative position and the relative posture, and the rotation axis (x, y, z in the figure) parallel to these three axes. By calculating the rotation of the position calculation device 30 around x1, y1, z1), the initial position and the initial posture of the position calculation device 30 are calculated. In the present embodiment, the center point of the marker M is set as the origin O, and the center of the camera 36, specifically, the center point of the image sensor is set as the origin I of the rotation axis.

After calculating the initial position and the initial posture, the user information transmission unit 305 transmits the user information to the server device 10 as an authentication request (S103). Next, the content information receiving unit 306 determines whether or not the authentication is permitted based on the response of the server device 10 to the authentication request (S104).

When authentication is permitted (S104, YES), the content information receiving unit 306 receives the content information transmitted from the server device 10 (S105), and the position transmitting unit 307 is the current position of the position calculating device 30, that is, the nearest position. The position coordinates calculated in the above are transmitted to the server device 10 together with the calculation or update time (S106), and the position calculation device 30 is based on the displacement of the three-dimensional feature point group detected by the parameter update unit 302 by the depth sensor 37. Update the current position and current posture of (S107).

Here, the update of the current position and the current posture will be described. The parameter update unit 302 calculates the displacement of the position and posture of the position calculation device 30 based on the displacement of the three-dimensional feature point group before and after the time series, and calculates this displacement as the initial position and the initial posture or the position and the previously updated position. The current position and the current posture of the position calculation device 30 are sequentially updated by integrating with the posture. By updating the current position and the current posture in this way, as shown in FIG. 9, the path R of the position calculation device 30 prior to the current position and the current posture C is obtained from the series of positions and postures obtained in the time series. Be done. For the calculation of the displacement of the self-position and the self-posture based on the displacement of the three-dimensional feature point group, a known method such as the SLAM (Simultaneus Localization and Mapping) method or the SfM (Structure from Motion) method shall be used. Further, the displacement of the position and the posture of the position calculation device 30 may be calculated based on the image captured by the camera 36 by using the VSLAM (Visual SLAM) method.

After updating the current position and the current posture, the addition processing unit 303 determines whether or not an addition instruction for adding the content to a predetermined position has been given via the touch panel 34 (S108).

When the addition instruction is given (S108, YES), the addition processing unit 303 executes the content addition process described later (S109). Next, the addition processing unit 303 determines whether or not an end instruction for terminating the processing related to the addition of the content has been given via the touch panel 34 (S110).

When the end instruction is given (S110, YES), the process related to the addition of the content is terminated.

On the other hand, when the end instruction is not given (S110, NO), the position transmission unit 307 transmits the updated position of the position calculation device 30 to the server device 10 (S106).

If no additional instruction is given in step S108 (S108, NO), the position transmission unit 307 transmits the updated position of the position calculation device 30 to the server device 10 (S106).

If the authentication is rejected in step S104 (S104, NO), the process related to the addition of the content is terminated.

If the marker is not recognized in step S101 (S101, NO), the parameter calculation unit 301 again determines whether or not the marker is recognized in the image captured by the camera 36 (S101).

(Content addition processing)
The content addition process will be described. FIG. 10 is a flowchart showing the operation of the content addition process. FIG. 11 is a schematic view showing the content addition position and the virtual object.

As shown in FIG. 10, the addition processing unit 303 determines whether or not the addition position of the content is specified via the touch panel 34 (S201). Here, the additional position is specified by tapping on the image captured by the camera 36 displayed on the touch panel 34.

When the additional position of the content is specified (S201, YES), the additional processing unit 303 is a position calculation device detected by the depth sensor 37 with respect to a predetermined location on the wall or equipment corresponding to the designated location on the captured image. The separation distance from 30 is acquired (S202), and the additional position is calculated based on the separation distance and the current position and current posture of the position calculation device 30 (S203).

Here, the calculation of the coordinates of the additional position will be described. As shown in FIG. 11, regarding the equipment F at the work site, first, when content is added to a predetermined location, a desired position in the imaging range P on the equipment F displayed as an image captured by the camera 36 on the touch panel 34. Is specified by tapping on the touch panel 34. The additional processing unit 303 acquires the separation distance D between the designated location A on the equipment F corresponding to the designated location on the touch panel 34 and the position calculation device 30 detected by the depth sensor 37, and the separation distance D and the separation distance D at this time. Based on the position and orientation of the position calculation device 30, the position coordinates of the designated position A in the coordinate system with the marker as the origin are calculated.

When content is added to the position coordinates of the designated location A, when the camera 36 takes an image so that the designated location A is included in the imaging range P, the virtual object V indicating the designated location A is superimposed on the captured image on the touch panel 34. Is displayed. Here, the virtual object V is represented by a three-dimensional model or image on an arrow pointing to the designated location A, and the three-dimensional model or image is a position calculation device when the pointing direction adds the designated location A and the content. It faces the extending direction of the line segment connecting the position C of 30.

After the coordinates of the additional position A are calculated in this way, the additional processing unit 303 transmits the coordinates of the additional position, the direction of the virtual object V pointing to the coordinates, and the content to the server device 10 as additional content. (S204).

If the content addition position is not specified in step S201 (S201, NO), the addition processing unit 303 again determines whether or not the content addition position is specified via the touch panel 34 (S201). ..

In this way, by calculating the self-position at the coordinates with the marker position as the origin, the self-position can be obtained by the same standard as other devices even in a place where it is difficult to use the GNSS signal such as an underground building. Can be done.

(Operation of server device)
FIG. 12 is a flowchart showing the overall operation of the server device related to the addition of contents. Note that FIG. 12 is intended to show an operation when a predetermined one user is targeted.

As shown in FIG. 12, the user information management unit 101 determines whether or not the user information has been received from the position calculation device 30 (S301).

When the user information is received (S301, YES), the user authentication unit 102 determines whether the user indicated by the user ID included in the user information is a legitimate user (S302). Here, the user authentication unit 102 authenticates in the user information management unit 101 by comparing the authentication information associated with the user ID to be authenticated with the authentication information included in the received user information. The user is judged to be valid based on the match of.

When the user is a legitimate user (S302, YES), the content information transmission unit 105 collects all the content information that can be acquired by the target user among the content information managed by the content information management unit 104. It is transmitted to the position calculation device 30 (S303).

Next, the position receiving unit 106 determines whether or not the current position transmitted from the position calculating device 30 has been received (S304).

When the current position is received (S304, YES), the route management unit 107 records the route ID in association with the current position and the calculation or update time attached to the current position (S305), and the additional content receiving unit 103 records the current position in association with each other (S305). It is determined whether or not the additional content is received from the position calculation device 30 (S306).

When the additional content is received (S306, YES), the content information management unit 104 records the additional content received by the additional content receiving unit 103 with a content ID, manages it as content information (S307), and positions the position. The receiving unit 106 again determines whether or not the current position transmitted from the position calculating device 30 has been received (S304). For the content information, the position calculation device 30 or the administrator terminal 20 used by the user to which the content is added can set whether or not the viewing is restricted and the user ID that permits the viewing.

On the other hand, when the additional content is not received (S306, NO), the position receiving unit 106 determines again whether or not the current position transmitted from the position calculating device 30 has been received (S304).

Further, in step S304, when the current position is not received (S304, NO), it is determined whether or not the position receiving unit 106 has received the current position transmitted from the position calculation device 30 again (S304).

Further, in step S302, if the user is not a legitimate user (S302, NO), the user information management unit 101 again determines whether or not the user information has been received from the position calculation device 30 (S301).

If the user information is not received in step S301 (S301, NO), the user information management unit 101 again determines whether or not the user information has been received from the position calculation device 30 (S301).

(Overlapping display of additional content)
The superimposed display of the additional content will be described. FIG. 13 is a schematic view showing additional content superimposed and displayed on the captured image. FIG. 14 is a schematic view showing additional content associated with a category.

As shown in FIG. 13, the additional contents V1 and V2 are superimposed and displayed on the image captured by the camera 36 on the touch panel 34 of the position calculation device 30. These additional contents V1 and V2 are displayed by tapping the virtual object V displayed on the touch panel 34. Here, the additional contents V1 and V2 are texts indicating work procedures and precautions, respectively.

In this way, by making it possible for a plurality of devices to share the content positioned at the coordinates with the marker position as the origin, the content associated with a specific position can be shared between the plurality of devices without using the GNSS signal. It is possible to achieve a so-called location-based AR that is shared and browsed by.

Further, as shown in FIG. 14, the additional content may be classified by associating the additional content with the categories classified according to the content of the additional content. In FIG. 14, contents are added as additional contents V3 to V5 to each of the three facilities in the work site. Here, the category of the additional content V3 is set to "tradition" including know-how and customs, and includes a text explaining how to open the door. In addition, the additional content V4 has its category set to "immediate response request", which is a request to other workers whose deadline has been set, and explains the reason why the door of the board was not locked and the future. Includes text requesting action. In addition, the category of the additional content V5 is set to "warning" to warn other workers, and since the liquid crystal panel has missing dots, it is necessary to confirm the further deterioration of the state. Includes text

By associating the additional content with the category in this way, for example, when there are a large number of additional contents at the work site, it is possible to display only the additional contents for which a predetermined category is set, and the purpose of the worker. It is possible to easily find additional content according to the above.

<Second embodiment>
The content addition system according to the second embodiment will be described. FIG. 15 is a schematic plan view showing a manufacturing factory in which the content addition system according to the present embodiment is operated. 16 and 17 are schematic views showing the output of one additional content in the building, respectively.

The content addition system 1 according to the present embodiment is presented with a point that the position calculation device 30 further includes a speaker (not shown) that outputs sound, a point that a predetermined object is used as a marker, and an operating environment. The type of content is different from the first embodiment. Hereinafter, only the points different from the first embodiment will be described.

In the present embodiment, the content addition system 1 is operated in the entire site as a manufacturing factory having a plurality of buildings and equipment as shown in FIG. It is assumed that there are three factories and three facilities in this site, and a worker having a position calculation device 30 is engaged as in the work site in the first embodiment. In addition, the inside of the site and the outside of the site are separated by a wall or the like, and workers can enter and exit the inside of the site through a gate.

In this embodiment, a gate is used as a marker instead of the attached two-dimensional image. Specifically, the gate imaged by the camera 36 from the outside of the site is recognized as a marker by the parameter calculation unit 301. Here, the parameter calculation unit 301 is based on the marker information indicating the shape and size of the gate imaged from the front, and the relative position and relative position of the position calculation device 30 with respect to the gate from the size and shape of the gate as a marker. By calculating the posture, the position of the position calculation device 30 on the three axes with the gate as the origin and the rotation of the position calculation device 30 around the rotation axis parallel to these three axes are calculated from the relative position and the relative posture. , The initial position and the initial posture of the position calculation device 30 are calculated. In the present embodiment, the center point of the gate is set as the origin O, and the center point of the image sensor in the camera 36 is set as the origin I of the rotation axis as in the first embodiment.

As described above, according to the content addition system 1 according to the present embodiment, a predetermined object such as a gate can function as the origin of the coordinates indicating the position of the position calculation device 30 and the position of the virtual object. The object used as this marker may be any object as long as it is suitable for estimating the positional relationship with the position calculation device 30.

The parameter update unit 302 differs from the first embodiment only in the initial position calculated based on the imaged gate and the point starting from the initial posture, and changes the position and posture of the position calculation device 30 for each frame. The current position and the current posture C of the position calculation device 30 starting from the initial position and the initial posture are sequentially updated in the same manner as in the first embodiment.

The method of adding the content by the addition processing unit 303 is the same as that of the first embodiment, but is different from the first embodiment in that the text and the voice are added to the designated portion. As shown in FIG. 15, two contents are added to the building B1 in the site, and as shown in FIGS. 16 and 17, the added positions of these contents are virtual objects V6 and V7, and the position calculation device 30 It is displayed on the touch panel 34 of. Here, the content indicated by the virtual object V6 is added to the vicinity of one entrance / exit in the building B1, and the content indicated by the virtual object V7 is added to the vicinity of the other entrance / exit in the building B1.

As shown in FIG. 16, when the virtual object V6 displayed on the touch panel 34 is tapped, the content including the message that it is possible to enter the building B1 from the door in the vicinity of the virtual object V6 is displayed in the position calculation device 30. It is presented to the worker who has. Here, this content includes the text TXT1, the voice SND1, and the image IMG1, and when presented to the operator, the display unit 304 of the position calculation device 30 superimposes the text TXT1 and the image IMG1 on the image captured by the camera 36. Is displayed on the touch panel 34, and the voice SND1 is output to the speaker.

Further, as shown in FIG. 17, when the virtual object V7 displayed on the touch panel 34 is tapped, the content including a message for prohibiting entry into the building B1 from the door in the vicinity of the virtual object V7 is included in the position calculation device 30. It is presented to the worker who has. Here, this content includes the text TXT2, the voice SND2, and the image IMG2, and when presented to the operator, the display unit 304 displays the text TXT2 and the image IMG2 and outputs the voice SND2.

As described above, the content addition system 1 according to the present embodiment is operated outdoors, and audio is added as content. The output voice is not limited to the one added in advance as the content, and may be, for example, the text included in the content read aloud by voice synthesis.

<Third embodiment>
The content addition system according to the third embodiment will be described. FIG. 18 is a schematic plan view showing a tourist spot where the content addition system according to the present embodiment is operated. FIG. 19 is a schematic diagram showing the output of additional content in the monument.

The content addition system 1 according to the present embodiment is different from the second embodiment only in the operating environment, and is operated in a tourist resort having a building or a monument as a tourist resource as shown in FIG. It is assumed that there are three buildings and two monuments in this tourist spot, and tourists who each have a position calculation device 30 are visiting. In addition, there is a sign that guides the tourist spot at a predetermined position in the tourist spot, for example, at a position close to the nearest bus stop or station in the tourist spot, and this sign is used as a marker in the present embodiment. Since the calculation of the origin O based on this sign is the same as that of the second embodiment, the description thereof will be omitted. Further, in the present embodiment, as in the first and second embodiments, the center point of the image sensor in the camera 36 is set as the origin I of the rotation axis.

Two contents are added to the monument M1 in the tourist spot. As shown in FIG. 19, the monument M1 is a Buddha image having historical value, and the additional positions of the two contents are displayed as virtual objects V8 and V9 on the touch panel 34 of the position calculation device 30. Here, the content represented by the virtual object V8 is added to the head in the monument M1, and the content represented by the virtual object V9 is added to the body in the monument M1.

When the virtual object V8 displayed on the touch panel 34 is tapped, content including a message explaining the head of the monument M1 is presented to the tourist having the position calculation device 30. Here, this content includes the text TXT3 and the voice SND3, and when presented to the tourist, the display unit 304 of the position calculation device 30 displays the text TXT3 on the touch panel 34 and causes the voice SND3 to be output to the speaker.

When the virtual object V9 displayed on the touch panel 34 is tapped, content including a message explaining the body of the monument M1 is presented to the tourist having the position calculation device 30. Here, this content includes the text TXT4 and the voice SND4, and when presented to the tourist, the display unit 304 displays the text TXT4 and outputs the voice SND4.

As shown in this embodiment, the content addition system 1 can be operated not only in a place where a worker engages but also in an environment such as a tourist spot.

In the present embodiment, the position calculation program is described as being pre-installed inside the above-mentioned position calculation device, but the position calculation program in the present invention also includes a program stored in a storage medium. Here, the storage medium is a medium that can be attached to and detached from a position calculation device, such as a magnetic tape, a magnetic disk (hard disk drive, etc.), an optical disk (CD-ROM, DVD disk, etc.), an optical magnetic disk (MO, etc.), and a flash memory. Also, all media that can be read and executed by a computer as the above-mentioned position calculation device, such as a medium that can be transmitted via a network.

Although embodiments of the invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

30 Position calculation device 36 Camera (imaging device)
37 Depth sensor 301 Parameter calculation unit 302 Parameter update unit

Claims (2)

It is a position calculation device that can be connected to the server device via a network and calculates its own position.
An imaging device that captures images of markers placed at predetermined locations,
A parameter calculation unit that calculates the initial position and initial posture of the position calculation device with the position of the marker as the origin based on the size and shape of the marker in the captured image.
A depth sensor that detects a group of three-dimensional feature points of an object around the position calculation device at predetermined time intervals, and
Based on the displacement between the three-dimensional feature point groups that move back and forth in the time series, the displacement of the position and posture of the position calculation device is calculated, and the position and posture of the position calculation device starting from the initial position and the initial posture. Parameter update part to update, and
A display unit that displays an image captured by the image pickup device, and a display unit.
The distance between the designated location designated in the displayed captured image and the position calculation device is acquired by the depth sensor, and the position of the marker is determined based on the distance and the position and orientation of the position calculation device. The position of the designated location as the origin is calculated, the predetermined content added to the designated location is associated with the position of the designated location as additional content, and the designated location and the position of the position calculation device are connected. A position calculation device including an additional processing unit that associates a virtual object that faces the extending direction of a line segment and transmits the additional content and the direction of the virtual object associated with a designated portion of the additional content to the server device. .. It is a content addition system including a position calculation device that calculates its own position, the position calculation device, and a server device that can be connected via a network.
The position calculation device is
An imaging device that captures images of markers placed at predetermined locations,
A parameter calculation unit that calculates the initial position and initial posture of the position calculation device with the position of the marker as the origin based on the size and shape of the marker in the captured image.
A depth sensor that detects a group of three-dimensional feature points of an object around the position calculation device at predetermined time intervals, and
Based on the displacement between the three-dimensional feature point groups that move back and forth in the time series, the displacement of the position and posture of the position calculation device is calculated, and the position and posture of the position calculation device starting from the initial position and the initial posture. Parameter update part to update, and
A display unit that displays an image captured by the image pickup device, and a display unit.
The distance between the designated location designated in the displayed captured image and the position calculation device is acquired by the depth sensor, and the position of the marker is determined based on the distance and the position and orientation of the position calculation device. The position of the designated location as the origin is calculated, and additional content in which the position of the designated location is associated with the predetermined content added to the designated location is transmitted to the server device and associated with the designated location. It is provided with an additional processing unit that transmits the extending direction of the line segment connecting the designated position and the position of the position calculation device to the server device as the direction of the virtual object.
The server device
A receiver that receives the orientation of the additional content and virtual object transmitted by the position calculation device, and
A content addition system characterized by including a content information management unit that manages the orientation of received additional content and virtual objects as content information.

Images